| [2024-09-03 08:13:57,415][__main__][INFO] - Using GPU for training. Main device : cuda:2 | |
| [2024-09-03 08:13:57,416][__main__][INFO] - Using multiple GPUs : [2, 3, 1, 0] | |
| [2024-09-03 08:14:04,559][__main__][INFO] - PSF shape : torch.Size([1, 380, 507, 3]) | |
| [2024-09-03 08:14:04,587][__main__][INFO] - PSF min : 0.0 | |
| [2024-09-03 08:14:04,596][__main__][INFO] - PSF max : 0.04469018802046776 | |
| [2024-09-03 08:14:04,596][__main__][INFO] - PSF dtype : torch.float32 | |
| [2024-09-03 08:14:04,626][__main__][INFO] - PSF norm : 1.000011682510376 | |
| [2024-09-03 08:14:04,627][__main__][INFO] - Reconstruction a few images with ADMM... | |
| [2024-09-03 08:14:05,299][__main__][INFO] - Cropped shape : (123, 123, 3) | |
| [2024-09-03 08:14:05,568][__main__][INFO] - Cropped shape : (123, 123, 3) | |
| [2024-09-03 08:14:07,940][__main__][INFO] - Train test size : 21250 | |
| [2024-09-03 08:14:07,941][__main__][INFO] - Test test size : 3750 | |
| [2024-09-03 08:14:08,064][__main__][INFO] - Training model with 8680318 parameters | |
| [2024-09-03 08:14:08,065][__main__][INFO] - -- Pre-process model with 4051168 parameters | |
| [2024-09-03 08:14:08,065][__main__][INFO] - -- Post-process model with 4051168 parameters | |
| [2024-09-03 08:14:08,066][__main__][INFO] - Setup time : 0.1247551441192627 s | |
| [2024-09-03 08:14:08,066][__main__][INFO] - PSF shape : torch.Size([1, 380, 507, 3]) | |
| [2024-09-03 08:14:08,066][__main__][INFO] - Results saved in /root/FORKS/LenslessPiCamStefan/outputs/2024-09-03/08-13-52 | |
| [2024-09-03 08:21:24,008][__main__][INFO] - Epoch 0 with learning rate [0.0, 0.0] | |
| [2024-09-03 09:49:13,212][__main__][INFO] - loss : 0.8601106564787273 | |
| [2024-09-03 09:56:25,152][__main__][INFO] - Epoch 1 with learning rate [0.0001, 0.0001] | |
| [2024-09-03 11:24:05,059][__main__][INFO] - loss : 0.6646762082026144 | |
| [2024-09-03 11:31:15,918][__main__][INFO] - Epoch 2 with learning rate [9.957224306869053e-05, 9.957224306869053e-05] | |
| [2024-09-03 12:58:50,960][__main__][INFO] - loss : 0.5776314393804931 | |
| [2024-09-03 13:06:01,159][__main__][INFO] - Epoch 3 with learning rate [9.829629131445342e-05, 9.829629131445342e-05] | |
| [2024-09-03 14:33:18,547][__main__][INFO] - loss : 0.5144455876308655 | |
| [2024-09-03 14:40:30,287][__main__][INFO] - Epoch 4 with learning rate [9.619397662556435e-05, 9.619397662556435e-05] | |
| [2024-09-03 16:07:51,483][__main__][INFO] - loss : 0.4862989857778834 | |
| [2024-09-03 16:15:05,979][__main__][INFO] - Epoch 5 with learning rate [9.330127018922194e-05, 9.330127018922194e-05] | |
| [2024-09-03 17:42:12,825][__main__][INFO] - loss : 0.46879550605961884 | |
| [2024-09-03 17:49:23,475][__main__][INFO] - Epoch 6 with learning rate [8.966766701456177e-05, 8.966766701456177e-05] | |
| [2024-09-03 19:16:20,040][__main__][INFO] - loss : 0.45263505421765465 | |
| [2024-09-03 19:23:30,605][__main__][INFO] - Epoch 7 with learning rate [8.535533905932738e-05, 8.535533905932738e-05] | |
| [2024-09-03 20:50:29,649][__main__][INFO] - loss : 0.4395513916765037 | |
| [2024-09-03 20:57:40,418][__main__][INFO] - Epoch 8 with learning rate [8.043807145043604e-05, 8.043807145043604e-05] | |
| [2024-09-03 22:24:43,373][__main__][INFO] - loss : 0.4286410703556744 | |
| [2024-09-03 22:31:53,849][__main__][INFO] - Epoch 9 with learning rate [7.500000000000001e-05, 7.500000000000001e-05] | |
| [2024-09-03 23:58:53,840][__main__][INFO] - loss : 0.4194181306133738 | |
| [2024-09-04 00:06:02,819][__main__][INFO] - Epoch 10 with learning rate [6.91341716182545e-05, 6.91341716182545e-05] | |
| [2024-09-04 01:33:09,751][__main__][INFO] - loss : 0.41061601158860406 | |
| [2024-09-04 01:40:20,901][__main__][INFO] - Epoch 11 with learning rate [6.294095225512603e-05, 6.294095225512603e-05] | |
| [2024-09-04 03:07:24,365][__main__][INFO] - loss : 0.40228850427442153 | |
| [2024-09-04 03:14:36,888][__main__][INFO] - Epoch 12 with learning rate [5.6526309611002594e-05, 5.6526309611002594e-05] | |
| [2024-09-04 04:41:42,016][__main__][INFO] - loss : 0.39414237295192345 | |
| [2024-09-04 04:48:51,884][__main__][INFO] - Epoch 13 with learning rate [5e-05, 5e-05] | |
| [2024-09-04 06:16:06,479][__main__][INFO] - loss : 0.38625356541522676 | |
| [2024-09-04 06:23:19,760][__main__][INFO] - Epoch 14 with learning rate [4.347369038899744e-05, 4.347369038899744e-05] | |
| [2024-09-04 07:50:18,498][__main__][INFO] - loss : 0.37879284768707544 | |
| [2024-09-04 07:57:28,895][__main__][INFO] - Epoch 15 with learning rate [3.705904774487396e-05, 3.705904774487396e-05] | |
| [2024-09-04 09:24:30,574][__main__][INFO] - loss : 0.37188338201283355 | |
| [2024-09-04 09:31:43,358][__main__][INFO] - Epoch 16 with learning rate [3.086582838174551e-05, 3.086582838174551e-05] | |
| [2024-09-04 10:58:49,927][__main__][INFO] - loss : 0.3654300492939109 | |
| [2024-09-04 11:06:01,771][__main__][INFO] - Epoch 17 with learning rate [2.500000000000001e-05, 2.500000000000001e-05] | |
| [2024-09-04 12:33:07,902][__main__][INFO] - loss : 0.3594383925182684 | |
| [2024-09-04 12:40:19,397][__main__][INFO] - Epoch 18 with learning rate [1.9561928549563968e-05, 1.9561928549563968e-05] | |
| [2024-09-04 14:07:19,565][__main__][INFO] - loss : 0.3542062355965835 | |
| [2024-09-04 14:14:33,171][__main__][INFO] - Epoch 19 with learning rate [1.4644660940672627e-05, 1.4644660940672627e-05] | |
| [2024-09-04 15:41:32,793][__main__][INFO] - loss : 0.3493611904977071 | |
| [2024-09-04 15:48:46,705][__main__][INFO] - Epoch 20 with learning rate [1.0332332985438248e-05, 1.0332332985438248e-05] | |
| [2024-09-04 17:15:38,218][__main__][INFO] - loss : 0.3454234600347605 | |
| [2024-09-04 17:22:49,055][__main__][INFO] - Epoch 21 with learning rate [6.698729810778065e-06, 6.698729810778065e-06] | |
| [2024-09-04 18:49:43,957][__main__][INFO] - loss : 0.3421878985557187 | |
| [2024-09-04 18:56:55,228][__main__][INFO] - Epoch 22 with learning rate [3.8060233744356633e-06, 3.8060233744356633e-06] | |
| [2024-09-04 20:23:51,417][__main__][INFO] - loss : 0.3396796187886899 | |
| [2024-09-04 20:31:03,132][__main__][INFO] - Epoch 23 with learning rate [1.70370868554659e-06, 1.70370868554659e-06] | |
| [2024-09-04 21:57:52,499][__main__][INFO] - loss : 0.3381009503859668 | |
| [2024-09-04 22:05:01,818][__main__][INFO] - Epoch 24 with learning rate [4.277569313094809e-07, 4.277569313094809e-07] | |
| [2024-09-04 23:31:52,611][__main__][INFO] - loss : 0.3371191366107456 | |
| [2024-09-04 23:39:04,173][__main__][INFO] - Train time [hour] : 39.415022127694556 h | |
| [2024-09-04 23:39:04,174][__main__][INFO] - Results saved in /root/FORKS/LenslessPiCamStefan/outputs/2024-09-03/08-13-52 | |