| [2024-08-28 14:38:50,674][__main__][INFO] - Using GPU for training. Main device : cuda:2 | |
| [2024-08-28 14:38:50,674][__main__][INFO] - Using multiple GPUs : [2, 3, 1, 0] | |
| [2024-08-28 14:38:57,743][__main__][INFO] - PSF shape : torch.Size([1, 380, 507, 3]) | |
| [2024-08-28 14:38:57,768][__main__][INFO] - PSF min : 0.0 | |
| [2024-08-28 14:38:57,777][__main__][INFO] - PSF max : 0.04469018802046776 | |
| [2024-08-28 14:38:57,778][__main__][INFO] - PSF dtype : torch.float32 | |
| [2024-08-28 14:38:57,802][__main__][INFO] - PSF norm : 1.000011682510376 | |
| [2024-08-28 14:38:57,802][__main__][INFO] - Reconstruction a few images with ADMM... | |
| [2024-08-28 14:38:58,404][__main__][INFO] - Cropped shape : (123, 123, 3) | |
| [2024-08-28 14:38:58,642][__main__][INFO] - Cropped shape : (123, 123, 3) | |
| [2024-08-28 14:39:00,837][__main__][INFO] - Train test size : 21250 | |
| [2024-08-28 14:39:00,837][__main__][INFO] - Test test size : 3750 | |
| [2024-08-28 14:39:00,948][__main__][INFO] - Training model with 8092627 parameters | |
| [2024-08-28 14:39:00,949][__main__][INFO] - -- Pre-process model with 3913696 parameters | |
| [2024-08-28 14:39:00,949][__main__][INFO] - -- Post-process model with 4051168 parameters | |
| [2024-08-28 14:39:00,950][__main__][INFO] - -- Background subtraction model with 127740 parameters | |
| [2024-08-28 14:39:00,950][__main__][INFO] - Setup time : 0.1123654842376709 s | |
| [2024-08-28 14:39:00,950][__main__][INFO] - PSF shape : torch.Size([1, 380, 507, 3]) | |
| [2024-08-28 14:39:00,950][__main__][INFO] - Results saved in /root/FORKS/LenslessPiCamStefan/outputs/2024-08-28/14-38-45 | |
| [2024-08-28 14:48:33,527][__main__][INFO] - Epoch 0 with learning rate [0.0, 0.0] | |
| [2024-08-28 16:54:08,829][__main__][INFO] - loss : 1.0027185649201396 | |
| [2024-08-28 17:03:36,485][__main__][INFO] - Epoch 1 with learning rate [0.0001, 0.0001] | |
| [2024-08-28 19:08:41,481][__main__][INFO] - loss : 0.553566960909936 | |
| [2024-08-28 19:18:06,915][__main__][INFO] - Epoch 2 with learning rate [9.957224306869053e-05, 9.957224306869053e-05] | |
| [2024-08-28 21:23:07,057][__main__][INFO] - loss : 0.43926881171542753 | |
| [2024-08-28 21:32:32,445][__main__][INFO] - Epoch 3 with learning rate [9.829629131445342e-05, 9.829629131445342e-05] | |
| [2024-08-28 23:37:28,221][__main__][INFO] - loss : 0.39886823706588254 | |
| [2024-08-28 23:46:52,234][__main__][INFO] - Epoch 4 with learning rate [9.619397662556435e-05, 9.619397662556435e-05] | |
| [2024-08-29 01:51:37,421][__main__][INFO] - loss : 0.37715716228665935 | |
| [2024-08-29 02:00:58,280][__main__][INFO] - Epoch 5 with learning rate [9.330127018922194e-05, 9.330127018922194e-05] | |
| [2024-08-29 04:05:32,753][__main__][INFO] - loss : 0.3625877354523343 | |
| [2024-08-29 04:14:56,826][__main__][INFO] - Epoch 6 with learning rate [8.966766701456177e-05, 8.966766701456177e-05] | |
| [2024-08-29 06:19:46,792][__main__][INFO] - loss : 0.35109421317184264 | |
| [2024-08-29 06:29:08,995][__main__][INFO] - Epoch 7 with learning rate [8.535533905932738e-05, 8.535533905932738e-05] | |
| [2024-08-29 08:34:07,717][__main__][INFO] - loss : 0.3412525762856277 | |
| [2024-08-29 08:43:29,959][__main__][INFO] - Epoch 8 with learning rate [8.043807145043604e-05, 8.043807145043604e-05] | |
| [2024-08-29 10:48:20,532][__main__][INFO] - loss : 0.3327004362634626 | |
| [2024-08-29 10:57:45,680][__main__][INFO] - Epoch 9 with learning rate [7.500000000000001e-05, 7.500000000000001e-05] | |
| [2024-08-29 13:02:32,673][__main__][INFO] - loss : 0.324895132570735 | |
| [2024-08-29 13:11:56,173][__main__][INFO] - Epoch 10 with learning rate [6.91341716182545e-05, 6.91341716182545e-05] | |
| [2024-08-29 15:09:25,019][__main__][INFO] - loss : 0.31740472280505566 | |
| [2024-08-29 15:18:31,564][__main__][INFO] - Epoch 11 with learning rate [6.294095225512603e-05, 6.294095225512603e-05] | |
| [2024-08-29 17:12:45,659][__main__][INFO] - loss : 0.31045449055477675 | |
| [2024-08-29 17:21:56,979][__main__][INFO] - Epoch 12 with learning rate [5.6526309611002594e-05, 5.6526309611002594e-05] | |
| [2024-08-29 19:16:28,092][__main__][INFO] - loss : 0.30363226063940696 | |
| [2024-08-30 06:30:48,155][__main__][INFO] - Epoch 13 with learning rate [5e-05, 5e-05] | |
| [2024-08-30 08:30:40,022][__main__][INFO] - loss : 0.2970853329781817 | |
| [2024-08-30 08:39:50,419][__main__][INFO] - Epoch 14 with learning rate [4.347369038899744e-05, 4.347369038899744e-05] | |
| [2024-08-30 10:33:14,477][__main__][INFO] - loss : 0.2909040022048936 | |
| [2024-08-30 10:42:16,711][__main__][INFO] - Epoch 15 with learning rate [3.705904774487396e-05, 3.705904774487396e-05] | |
| [2024-08-30 12:35:28,050][__main__][INFO] - loss : 0.2849222946089577 | |
| [2024-08-30 12:44:32,864][__main__][INFO] - Epoch 16 with learning rate [3.086582838174551e-05, 3.086582838174551e-05] | |
| [2024-08-30 14:37:47,692][__main__][INFO] - loss : 0.2792259944981831 | |
| [2024-08-30 14:46:48,464][__main__][INFO] - Epoch 17 with learning rate [2.500000000000001e-05, 2.500000000000001e-05] | |
| [2024-08-30 16:39:47,691][__main__][INFO] - loss : 0.27395995486294084 | |
| [2024-08-30 16:48:53,325][__main__][INFO] - Epoch 18 with learning rate [1.9561928549563968e-05, 1.9561928549563968e-05] | |
| [2024-08-30 18:41:35,471][__main__][INFO] - loss : 0.2692438893324451 | |
| [2024-08-30 18:50:42,467][__main__][INFO] - Epoch 19 with learning rate [1.4644660940672627e-05, 1.4644660940672627e-05] | |
| [2024-08-30 20:43:15,687][__main__][INFO] - loss : 0.26508778246158593 | |
| [2024-08-30 20:52:23,691][__main__][INFO] - Epoch 20 with learning rate [1.0332332985438248e-05, 1.0332332985438248e-05] | |
| [2024-08-30 22:45:08,380][__main__][INFO] - loss : 0.2615289281546766 | |
| [2024-08-30 22:54:08,731][__main__][INFO] - Epoch 21 with learning rate [6.698729810778065e-06, 6.698729810778065e-06] | |
| [2024-08-31 00:47:00,724][__main__][INFO] - loss : 0.2586739508418654 | |
| [2024-08-31 00:56:03,414][__main__][INFO] - Epoch 22 with learning rate [3.8060233744356633e-06, 3.8060233744356633e-06] | |
| [2024-08-31 02:49:01,882][__main__][INFO] - loss : 0.2565490553776421 | |
| [2024-08-31 02:58:07,315][__main__][INFO] - Epoch 23 with learning rate [1.70370868554659e-06, 1.70370868554659e-06] | |
| [2024-08-31 04:51:01,585][__main__][INFO] - loss : 0.2551095724756517 | |
| [2024-08-31 05:00:02,982][__main__][INFO] - Epoch 24 with learning rate [4.277569313094809e-07, 4.277569313094809e-07] | |
| [2024-08-31 06:53:04,193][__main__][INFO] - loss : 0.25428580412100915 | |
| [2024-08-31 07:02:04,474][__main__][INFO] - Train time [hour] : 64.38390766408709 h | |
| [2024-08-31 07:02:04,474][__main__][INFO] - Results saved in /root/FORKS/LenslessPiCamStefan/outputs/2024-08-28/14-38-45 | |