Training in progress, step 30000

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:50c9225ef897a1c0012c88a93a4f2199738fa3993e8ccc39425c957ad69086e1
+oid sha256:837fabe2f9887cb975f4c577c865c8c31d3ecc4fcc9afcd2a38e8c97918a8409
 size 3219908024

wandb/run-20231122_234140-blvl5h7j/files/output.log

@@ -1206,3 +1206,1071 @@ Reading metadata...: 1it [00:00,  6.65it/s]
 
 
 
+
+
+
+
+
+
+
+
+
+ 48%|██████████▏          | 29080/60000 [9:20:39<92:22:52, 10.76s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 48%|██████████▏          | 29099/60000 [9:23:43<83:24:26,  9.72s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|██████████▏          | 29119/60000 [9:26:58<81:51:00,  9.54s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|██████████▏          | 29139/60000 [9:30:11<82:11:33,  9.59s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|██████████▏          | 29159/60000 [9:33:25<82:17:32,  9.61s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|██████████▏          | 29179/60000 [9:36:49<90:12:46, 10.54s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|██████████▏          | 29200/60000 [9:40:24<87:33:43, 10.23s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|██████████▏          | 29220/60000 [9:43:42<84:20:19,  9.86s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|██████████▏          | 29239/60000 [9:46:50<83:21:09,  9.75s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|██████████▏          | 29260/60000 [9:50:23<86:01:10, 10.07s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|██████████▏          | 29279/60000 [9:53:44<95:31:58, 11.19s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|██████████▎          | 29299/60000 [9:57:12<83:44:55,  9.82s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▊          | 29319/60000 [10:00:28<82:02:13,  9.63s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▊          | 29340/60000 [10:03:52<83:00:25,  9.75s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▊          | 29360/60000 [10:07:10<83:05:27,  9.76s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▊          | 29380/60000 [10:10:46<88:50:51, 10.45s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▊          | 29399/60000 [10:13:50<81:25:21,  9.58s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▊          | 29419/60000 [10:17:02<81:32:06,  9.60s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▊          | 29439/60000 [10:20:20<82:08:03,  9.68s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▊          | 29459/60000 [10:23:49<84:20:46,  9.94s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▊          | 29479/60000 [10:27:05<83:13:29,  9.82s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▊          | 29499/60000 [10:30:17<81:34:06,  9.63s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▊          | 29519/60000 [10:33:33<90:22:14, 10.67s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▊          | 29539/60000 [10:36:46<81:25:44,  9.62s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▊          | 29560/60000 [10:40:04<80:49:14,  9.56s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▊          | 29579/60000 [10:43:05<81:37:34,  9.66s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▊          | 29599/60000 [10:46:20<81:06:14,  9.60s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▊          | 29619/60000 [10:49:50<86:24:20, 10.24s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▉          | 29638/60000 [10:52:51<72:37:00,  8.61s/it]
+
+ 49%|█████████▉          | 29639/60000 [10:52:58<67:10:10,  7.96s/it]
+[2023-11-23 10:34:45,267] [INFO] [loss_scaler.py:183:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 131072, reducing to 65536
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▉          | 29660/60000 [10:56:17<80:12:33,  9.52s/it]
+
+
+
+
+
+
+
+
+
+
+
+Reading metadata...: 1650it [00:00, 9494.14it/s]<81:33:52,  9.68s/it]
+
+
+
+
+
+
+
+
+ 49%|█████████▉          | 29679/60000 [10:59:23<81:41:42,  9.70s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 49%|█████████▉          | 29699/60000 [11:02:42<85:14:16, 10.13s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|█████████▉          | 29719/60000 [11:05:53<80:14:58,  9.54s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|█████████▉          | 29740/60000 [11:09:12<80:22:17,  9.56s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|█████████▉          | 29760/60000 [11:12:27<81:09:55,  9.66s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|█████████▉          | 29779/60000 [11:15:29<80:25:00,  9.58s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|█████████▉          | 29800/60000 [11:19:00<81:34:22,  9.72s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|█████████▉          | 29820/60000 [11:22:11<79:15:11,  9.45s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|█████████▉          | 29840/60000 [11:25:23<79:57:25,  9.54s/it]
+
+
+
+
+
+
+
+Reading metadata...: 2165it [00:00, 3642.98it/s]<80:42:51,  9.64s/it]
+
+
+
+
+
+
+
+
+
+
+
+ 50%|█████████▉          | 29859/60000 [11:28:29<80:50:42,  9.66s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|█████████▉          | 29880/60000 [11:31:58<81:50:14,  9.78s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|█████████▉          | 29899/60000 [11:34:59<79:31:12,  9.51s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|█████████▉          | 29919/60000 [11:38:37<99:04:03, 11.86s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|█████████▉          | 29939/60000 [11:41:54<82:04:31,  9.83s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|█████████▉          | 29960/60000 [11:45:33<93:34:41, 11.21s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|█████████▉          | 29979/60000 [11:48:38<81:32:05,  9.78s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|██████████          | 30000/60000 [11:51:59<78:56:02,  9.47s/it][INFO|trainer.py:3173] 2023-11-23 11:33:47,040 >> ***** Running Evaluation *****
+[INFO|trainer.py:3177] 2023-11-23 11:33:47,041 >>   Num examples: Unknown
+[INFO|trainer.py:3178] 2023-11-23 11:33:47,041 >>   Batch size = 4
+Reading metadata...: 1704it [00:00, 8465.87it/s]
+Reading metadata...: 1it [00:00,  6.49it/s]
+
+[INFO|trainer_utils.py:759] 2023-11-23 11:33:48,041 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: down_votes, up_votes, gender, locale, segment, client_id, accent, input_length, path, age. If down_votes, up_votes, gender, locale, segment, client_id, accent, input_length, path, age are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
+ 50%|██████████          | 30000/60000 [12:02:00<78:56:02,  9.47s/it]
+ 50%|██████████          | 30000/60000 [12:02:00<78:56:02,  9.47s/it][INFO|trainer.py:2896] 2023-11-23 11:44:15,860 >> Saving model checkpoint to ./checkpoint-30000
+[INFO|configuration_utils.py:462] 2023-11-23 11:44:15,870 >> Configuration saved in ./checkpoint-30000/config.json
+[INFO|configuration_utils.py:568] 2023-11-23 11:44:15,875 >> Configuration saved in ./checkpoint-30000/generation_config.json
+[INFO|modeling_utils.py:2194] 2023-11-23 11:45:02,136 >> Model weights saved in ./checkpoint-30000/pytorch_model.bin
+[INFO|feature_extraction_utils.py:425] 2023-11-23 11:45:02,141 >> Feature extractor saved in ./checkpoint-30000/preprocessor_config.json
+[2023-11-23 11:45:02,169] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step30000 is about to be saved!
+[2023-11-23 11:45:02,193] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: ./checkpoint-30000/global_step30000/mp_rank_00_model_states.pt
+[2023-11-23 11:45:02,194] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoint-30000/global_step30000/mp_rank_00_model_states.pt...
+[2023-11-23 11:45:08,737] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoint-30000/global_step30000/mp_rank_00_model_states.pt.
+[2023-11-23 11:45:08,744] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoint-30000/global_step30000/zero_pp_rank_0_mp_rank_00_optim_states.pt...
+[2023-11-23 11:45:36,754] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoint-30000/global_step30000/zero_pp_rank_0_mp_rank_00_optim_states.pt.
+[2023-11-23 11:45:36,765] [INFO] [engine.py:3417:_save_zero_checkpoint] zero checkpoint saved ./checkpoint-30000/global_step30000/zero_pp_rank_0_mp_rank_00_optim_states.pt
+[2023-11-23 11:45:36,767] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step30000 is ready now!
+[INFO|feature_extraction_utils.py:425] 2023-11-23 11:46:33,500 >> Feature extractor saved in ./preprocessor_config.json
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|██████████          | 30020/60000 [12:08:10<85:33:08, 10.27s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|██████████          | 30039/60000 [12:11:36<88:09:32, 10.59s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 50%|██████████          | 30059/60000 [12:14:58<82:44:30,  9.95s/it]
+
+
+
+
+
+
+
+

wandb/run-20231122_234140-blvl5h7j/files/wandb-summary.json

@@ -1 +1 @@
-{"train/loss": 0.0522, "train/learning_rate": 1.575559322033898e-06, "train/epoch": 0.48, "train/global_step": 29060, "_timestamp": 1700722741.684163, "_runtime": 33440.79569220543, "_step": 53, "eval/loss": 0.19091796875, "eval/wer": 8.330192235205427, "eval/runtime": 607.5708, "eval/samples_per_second": 2.805, "eval/steps_per_second": 0.701}
+{"train/loss": 0.0812, "train/learning_rate": 1.5248135593220338e-06, "train/epoch": 0.5, "train/global_step": 30060, "_timestamp": 1700733415.2111008, "_runtime": 44114.32262992859, "_step": 104, "eval/loss": 0.193603515625, "eval/wer": 8.358462118356577, "eval/runtime": 600.6457, "eval/samples_per_second": 2.837, "eval/steps_per_second": 0.709}

wandb/run-20231122_234140-blvl5h7j/run-blvl5h7j.wandb

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:29423430447005f81dbb0bf9147426a558b431006c91a01ce9d0d23957de2e9a
-size 1397461
+oid sha256:2907ff4ad50e187d1d77714ad79bcb13875da01d13afbec0001cade84a998108
+size 1994140