SDGPA / snow2 /log.txt
roujin's picture
init
44a8b4b verified
Model pushed to 1 GPU(s), type NVIDIA GeForce RTX 3090.
Loaded pretrained model from /data/shared/luojun/runs/snow1/weights/checkpoint.pth.tar
--- Training ---
Epoch 0 train loss: 0.6085, acc: 0.8926
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.829
mIoU improved from 0.0000 to 0.4715.
mIoU improved from 0.0000 to 0.6457.
--- Training ---
Epoch 1 train loss: 0.5992, acc: 0.8947
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.854
mIoU improved from 0.4715 to 0.4840.
mIoU improved from 0.6457 to 0.6482.
--- Training ---
Epoch 2 train loss: 0.5943, acc: 0.8953
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.839
--- Training ---
Epoch 3 train loss: 0.5837, acc: 0.8977
--- Validation - Source ---
* Acc 0.933
--- Validation - Target ---
* Acc 0.828
--- Training ---
Epoch 4 train loss: 0.5794, acc: 0.8984
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.847
mIoU improved from 0.6482 to 0.6508.
--- Training ---
Epoch 5 train loss: 0.5752, acc: 0.8990
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.816
--- Training ---
Epoch 6 train loss: 0.5719, acc: 0.8999
--- Validation - Source ---
* Acc 0.932
--- Validation - Target ---
* Acc 0.833
--- Training ---
Epoch 7 train loss: 0.5682, acc: 0.9004
--- Validation - Source ---
* Acc 0.933
--- Validation - Target ---
* Acc 0.843
--- Training ---
Epoch 8 train loss: 0.5659, acc: 0.9010
--- Validation - Source ---
* Acc 0.931
--- Validation - Target ---
* Acc 0.847
--- Training ---
Epoch 9 train loss: 0.5675, acc: 0.9005
--- Validation - Source ---
* Acc 0.933
--- Validation - Target ---
* Acc 0.826
--- Training ---
Epoch 10 train loss: 0.5579, acc: 0.9027
--- Validation - Source ---
* Acc 0.932
--- Validation - Target ---
* Acc 0.838
--- Training ---
Epoch 11 train loss: 0.5569, acc: 0.9024
--- Validation - Source ---
* Acc 0.933
--- Validation - Target ---
* Acc 0.836
--- Training ---
Epoch 12 train loss: 0.5530, acc: 0.9033
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.832
--- Training ---
Epoch 13 train loss: 0.5535, acc: 0.9032
--- Validation - Source ---
* Acc 0.933
--- Validation - Target ---
* Acc 0.857
--- Training ---
Epoch 14 train loss: 0.5509, acc: 0.9035
--- Validation - Source ---
* Acc 0.933
--- Validation - Target ---
* Acc 0.825
--- Training ---
Epoch 15 train loss: 0.5475, acc: 0.9043
--- Validation - Source ---
* Acc 0.933
--- Validation - Target ---
* Acc 0.835
--- Training ---
Epoch 16 train loss: 0.5471, acc: 0.9041
--- Validation - Source ---
* Acc 0.932
--- Validation - Target ---
* Acc 0.820
--- Training ---
Epoch 17 train loss: 0.5439, acc: 0.9052
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.833
--- Training ---
Epoch 18 train loss: 0.5415, acc: 0.9053
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.845
--- Training ---
Epoch 19 train loss: 0.5382, acc: 0.9059
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.837
mIoU improved from 0.6508 to 0.6520.
--- Training ---
Epoch 20 train loss: 0.5375, acc: 0.9062
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.831
mIoU improved from 0.6520 to 0.6532.
--- Training ---
Epoch 21 train loss: 0.5339, acc: 0.9067
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.835
mIoU improved from 0.6532 to 0.6549.
--- Training ---
Epoch 22 train loss: 0.5356, acc: 0.9058
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.818
--- Training ---
Epoch 23 train loss: 0.5318, acc: 0.9071
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.841
--- Training ---
Epoch 24 train loss: 0.5310, acc: 0.9072
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.830
--- Training ---
Epoch 25 train loss: 0.5306, acc: 0.9070
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.834
--- Training ---
Epoch 26 train loss: 0.5282, acc: 0.9079
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.836
--- Training ---
Epoch 27 train loss: 0.5270, acc: 0.9080
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.848
--- Training ---
Epoch 28 train loss: 0.5243, acc: 0.9084
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.848
--- Training ---
Epoch 29 train loss: 0.5216, acc: 0.9087
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.841
--- Training ---
Epoch 30 train loss: 0.5198, acc: 0.9094
--- Validation - Source ---
* Acc 0.933
--- Validation - Target ---
* Acc 0.847
--- Training ---
Epoch 31 train loss: 0.5196, acc: 0.9094
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.860
--- Training ---
Epoch 32 train loss: 0.5171, acc: 0.9096
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.840
--- Training ---
Epoch 33 train loss: 0.5165, acc: 0.9098
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.840
--- Training ---
Epoch 34 train loss: 0.5156, acc: 0.9099
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.826
mIoU improved from 0.6549 to 0.6551.
--- Training ---
Epoch 35 train loss: 0.5144, acc: 0.9103
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.838
--- Training ---
Epoch 36 train loss: 0.5123, acc: 0.9107
--- Validation - Source ---
* Acc 0.933
--- Validation - Target ---
* Acc 0.832
--- Training ---
Epoch 37 train loss: 0.5160, acc: 0.9102
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.833
mIoU improved from 0.6551 to 0.6556.
--- Training ---
Epoch 38 train loss: 0.5113, acc: 0.9106
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.824
--- Training ---
Epoch 39 train loss: 0.5128, acc: 0.9100
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.826
--- Training ---
Epoch 40 train loss: 0.5097, acc: 0.9114
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.847
--- Training ---
Epoch 41 train loss: 0.5076, acc: 0.9112
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.839
--- Training ---
Epoch 42 train loss: 0.5074, acc: 0.9114
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.842
mIoU improved from 0.6556 to 0.6569.
--- Training ---
Epoch 43 train loss: 0.5054, acc: 0.9119
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.840
mIoU improved from 0.6569 to 0.6578.
--- Training ---
Epoch 44 train loss: 0.5049, acc: 0.9118
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.838
--- Training ---
Epoch 45 train loss: 0.5026, acc: 0.9125
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.837
--- Training ---
Epoch 46 train loss: 0.5046, acc: 0.9120
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.839
mIoU improved from 0.6578 to 0.6579.
--- Training ---
Epoch 47 train loss: 0.5023, acc: 0.9125
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.822
--- Training ---
Epoch 48 train loss: 0.4998, acc: 0.9128
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.826
mIoU improved from 0.6579 to 0.6581.
--- Training ---
Epoch 49 train loss: 0.4978, acc: 0.9131
--- Validation - Source ---
* Acc 0.934
--- Validation - Target ---
* Acc 0.805
--- Training ---
Epoch 50 train loss: 0.4982, acc: 0.9132
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.793
--- Training ---
Epoch 51 train loss: 0.4959, acc: 0.9136
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.839
--- Training ---
Epoch 52 train loss: 0.4948, acc: 0.9138
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.828
--- Training ---
Epoch 53 train loss: 0.4965, acc: 0.9132
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.824
--- Training ---
Epoch 54 train loss: 0.4954, acc: 0.9134
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.849
mIoU improved from 0.6581 to 0.6589.
--- Training ---
Epoch 55 train loss: 0.4943, acc: 0.9138
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.857
--- Training ---
Epoch 56 train loss: 0.4943, acc: 0.9137
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.819
--- Training ---
Epoch 57 train loss: 0.4944, acc: 0.9137
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.845
--- Training ---
Epoch 58 train loss: 0.4924, acc: 0.9141
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.837
--- Training ---
Epoch 59 train loss: 0.4903, acc: 0.9143
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.823
--- Training ---
Epoch 60 train loss: 0.4906, acc: 0.9139
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.847
--- Training ---
Epoch 61 train loss: 0.4900, acc: 0.9145
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.834
--- Training ---
Epoch 62 train loss: 0.4918, acc: 0.9141
--- Validation - Source ---
* Acc 0.937
--- Validation - Target ---
* Acc 0.847
mIoU improved from 0.6589 to 0.6596.
--- Training ---
Epoch 63 train loss: 0.4899, acc: 0.9145
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.841
--- Training ---
Epoch 64 train loss: 0.4886, acc: 0.9148
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.836
--- Training ---
Epoch 65 train loss: 0.4874, acc: 0.9151
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.840
--- Training ---
Epoch 66 train loss: 0.4842, acc: 0.9153
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.833
--- Training ---
Epoch 67 train loss: 0.4847, acc: 0.9153
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.850
--- Training ---
Epoch 68 train loss: 0.4836, acc: 0.9158
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.849
mIoU improved from 0.6596 to 0.6599.
--- Training ---
Epoch 69 train loss: 0.4842, acc: 0.9156
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.835
mIoU improved from 0.6599 to 0.6629.
--- Training ---
Epoch 70 train loss: 0.4835, acc: 0.9158
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.848
--- Training ---
Epoch 71 train loss: 0.4861, acc: 0.9152
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.845
--- Training ---
Epoch 72 train loss: 0.4815, acc: 0.9159
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.839
--- Training ---
Epoch 73 train loss: 0.4819, acc: 0.9159
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.826
--- Training ---
Epoch 74 train loss: 0.4832, acc: 0.9157
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.844
--- Training ---
Epoch 75 train loss: 0.4829, acc: 0.9160
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.845
--- Training ---
Epoch 76 train loss: 0.4840, acc: 0.9157
--- Validation - Source ---
* Acc 0.935
--- Validation - Target ---
* Acc 0.833
--- Training ---
Epoch 77 train loss: 0.4781, acc: 0.9165
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.825
--- Training ---
Epoch 78 train loss: 0.4819, acc: 0.9158
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.809
--- Training ---
Epoch 79 train loss: 0.4801, acc: 0.9163
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.834
--- Training ---
Epoch 80 train loss: 0.4822, acc: 0.9161
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.851
--- Training ---
Epoch 81 train loss: 0.4770, acc: 0.9166
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.822
--- Training ---
Epoch 82 train loss: 0.4809, acc: 0.9163
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.830
--- Training ---
Epoch 83 train loss: 0.4776, acc: 0.9166
--- Validation - Source ---
* Acc 0.937
--- Validation - Target ---
* Acc 0.815
mIoU improved from 0.6629 to 0.6652.
--- Training ---
Epoch 84 train loss: 0.4795, acc: 0.9163
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.833
--- Training ---
Epoch 85 train loss: 0.4802, acc: 0.9163
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.851
--- Training ---
Epoch 86 train loss: 0.4788, acc: 0.9162
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.847
--- Training ---
Epoch 87 train loss: 0.4777, acc: 0.9167
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.832
--- Training ---
Epoch 88 train loss: 0.4757, acc: 0.9173
--- Validation - Source ---
* Acc 0.937
--- Validation - Target ---
* Acc 0.832
--- Training ---
Epoch 89 train loss: 0.4772, acc: 0.9169
--- Validation - Source ---
* Acc 0.937
--- Validation - Target ---
* Acc 0.838
--- Training ---
Epoch 90 train loss: 0.4767, acc: 0.9169
--- Validation - Source ---
* Acc 0.937
--- Validation - Target ---
* Acc 0.823
--- Training ---
Epoch 91 train loss: 0.4745, acc: 0.9174
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.832
--- Training ---
Epoch 92 train loss: 0.4764, acc: 0.9171
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.844
--- Training ---
Epoch 93 train loss: 0.4765, acc: 0.9168
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.842
--- Training ---
Epoch 94 train loss: 0.4763, acc: 0.9169
--- Validation - Source ---
* Acc 0.937
--- Validation - Target ---
* Acc 0.841
--- Training ---
Epoch 95 train loss: 0.4769, acc: 0.9170
--- Validation - Source ---
* Acc 0.937
--- Validation - Target ---
* Acc 0.821
--- Training ---
Epoch 96 train loss: 0.4754, acc: 0.9170
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.840
--- Training ---
Epoch 97 train loss: 0.4773, acc: 0.9169
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.831
--- Training ---
Epoch 98 train loss: 0.4775, acc: 0.9168
--- Validation - Source ---
* Acc 0.936
--- Validation - Target ---
* Acc 0.840
--- Training ---
Epoch 99 train loss: 0.4746, acc: 0.9171
--- Validation - Source ---
* Acc 0.937
--- Validation - Target ---
* Acc 0.840
Training complete in 333m 51s
Best mIoU target: 0.484036