End of training

README.md

@@ -44,32 +44,32 @@ More information needed
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
 | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
 | **teacher eval** |  | 43.25 | 61.25 |  |  |  |  | 11.6875 | 19.125 |
-| 0 | 0 | 2130303778816.0 | 135239930216448.0 | 25.4492 | 25.2159 | 99.144 | 12.413 | 10334765056.0 | 38482906972160.0 |
-| 2500 | 0.0404 | 892.0 | 6624.0 | 6.0041 | 25.2726 | 98.921 | 12.385 | 572.0 | 7200.0 |
-| 5000 | 0.0808 | 372.0 | 1912.0 | 5.0307 | 25.3204 | 98.734 | 12.362 | 294.0 | 298.0 |
-| 7500 | 0.1212 | 228.0 | 856.0 | 4.5171 | 25.3333 | 98.684 | 12.355 | 199.0 | 182.0 |
-| 10000 | 0.1616 | 181.0 | 696.0 | 4.2020 | 25.3363 | 98.673 | 12.354 | 152.0 | 171.0 |
-| 12500 | 0.2020 | 125.0 | 478.0 | 3.8242 | 25.3199 | 98.737 | 12.362 | 98.0 | 141.0 |
-| 15000 | 0.2424 | 114.5 | 414.0 | 3.6365 | 25.3181 | 98.744 | 12.363 | 81.0 | 143.0 |
-| 17500 | 0.2828 | 97.5 | 346.0 | 3.5083 | 25.3564 | 98.595 | 12.344 | 79.0 | 102.0 |
-| 20000 | 0.3232 | 76.0 | 290.0 | 3.3324 | 25.2953 | 98.833 | 12.374 | 66.5 | 106.0 |
-| 22500 | 0.3636 | 66.5 | 215.0 | 3.1472 | 25.3749 | 98.523 | 12.335 | 50.5 | 82.0 |
-| 25000 | 0.4040 | 60.0 | 197.0 | 3.0655 | 25.3266 | 98.711 | 12.359 | 47.25 | 65.0 |
-| 27500 | 0.4444 | 61.75 | 197.0 | 3.0053 | 25.3492 | 98.622 | 12.348 | 43.0 | 80.0 |
-| 30000 | 0.4848 | 60.0 | 182.0 | 2.9949 | 25.3186 | 98.741 | 12.362 | 43.5 | 68.5 |
-| 32500 | 0.5253 | 59.75 | 179.0 | 2.9704 | 25.3443 | 98.642 | 12.35 | 40.25 | 83.5 |
-| 35000 | 0.5657 | 57.5 | 175.0 | 2.9129 | 25.3354 | 98.676 | 12.354 | 38.25 | 64.5 |
-| 37500 | 0.6061 | 58.75 | 183.0 | 2.8906 | 25.3463 | 98.634 | 12.349 | 36.75 | 62.5 |
-| 40000 | 0.6465 | 54.75 | 157.0 | 2.8612 | 25.3397 | 98.659 | 12.352 | 37.25 | 56.5 |
-| 42500 | 0.6869 | 55.25 | 168.0 | 2.8460 | 25.3447 | 98.64 | 12.35 | 36.5 | 54.75 |
-| 45000 | 0.7273 | 50.5 | 135.0 | 2.7442 | 25.34 | 98.658 | 12.352 | 30.125 | 42.5 |
-| 47500 | 0.7677 | 50.25 | 129.0 | 2.7166 | 25.3456 | 98.637 | 12.349 | 30.75 | 39.5 |
-| 50000 | 0.8081 | 49.75 | 132.0 | 2.7034 | 25.2334 | 99.075 | 12.404 | 29.5 | 41.75 |
-| 52500 | 0.8485 | 48.5 | 129.0 | 2.6925 | 25.3023 | 98.805 | 12.37 | 28.875 | 37.25 |
-| 55000 | 0.8889 | 48.5 | 127.5 | 2.6764 | 25.2784 | 98.898 | 12.382 | 28.25 | 34.5 |
-| 57500 | 0.9293 | 48.25 | 122.0 | 2.6696 | 25.3217 | 98.729 | 12.361 | 28.25 | 33.75 |
-| 60000 | 0.9697 | 48.25 | 121.5 | 2.6663 | 25.3402 | 98.657 | 12.352 | 28.0 | 33.5 |
-| 61875 | 1.0 | 48.25 | 121.5 | 2.6658 | 25.2795 | 98.895 | 12.382 | 28.0 | 33.5 |
+| 0 | 0 | 2473901162496.0 | 170424302305280.0 | 45.7764 | 25.2689 | 98.936 | 12.387 | 4060086272.0 | 71468255805440.0 |
+| 2500 | 0.0404 | 2256.0 | 19712.0 | 20.8732 | 25.2001 | 99.206 | 12.421 | 1480.0 | 77312.0 |
+| 5000 | 0.0808 | 486.0 | 3104.0 | 18.4096 | 25.2957 | 98.831 | 12.374 | 338.0 | 1128.0 |
+| 7500 | 0.1212 | 276.0 | 1296.0 | 17.0014 | 25.2794 | 98.895 | 12.382 | 255.0 | 249.0 |
+| 10000 | 0.1616 | 202.0 | 756.0 | 16.2107 | 25.3199 | 98.736 | 12.362 | 187.0 | 296.0 |
+| 12500 | 0.2020 | 145.0 | 540.0 | 15.1004 | 25.2816 | 98.886 | 12.381 | 131.0 | 176.0 |
+| 15000 | 0.2424 | 123.5 | 488.0 | 14.5287 | 25.2154 | 99.146 | 12.413 | 93.5 | 146.0 |
+| 17500 | 0.2828 | 94.5 | 374.0 | 14.1501 | 25.2965 | 98.828 | 12.373 | 76.0 | 134.0 |
+| 20000 | 0.3232 | 78.5 | 306.0 | 13.6428 | 25.2525 | 99.0 | 12.395 | 63.25 | 149.0 |
+| 22500 | 0.3636 | 66.0 | 218.0 | 13.1496 | 25.2593 | 98.973 | 12.391 | 50.0 | 82.0 |
+| 25000 | 0.4040 | 63.5 | 202.0 | 12.9562 | 25.3101 | 98.775 | 12.367 | 44.0 | 74.0 |
+| 27500 | 0.4444 | 59.0 | 200.0 | 12.8429 | 25.2626 | 98.96 | 12.39 | 40.25 | 65.0 |
+| 30000 | 0.4848 | 58.25 | 198.0 | 12.8220 | 25.2928 | 98.843 | 12.375 | 39.75 | 61.25 |
+| 32500 | 0.5253 | 59.25 | 171.0 | 12.7662 | 25.3006 | 98.812 | 12.371 | 39.25 | 55.25 |
+| 35000 | 0.5657 | 58.25 | 171.0 | 12.6525 | 25.3008 | 98.811 | 12.371 | 36.5 | 47.25 |
+| 37500 | 0.6061 | 57.0 | 158.0 | 12.6054 | 25.3065 | 98.789 | 12.368 | 36.5 | 49.5 |
+| 40000 | 0.6465 | 55.5 | 159.0 | 12.5787 | 25.1987 | 99.212 | 12.421 | 33.5 | 64.5 |
+| 42500 | 0.6869 | 55.0 | 151.0 | 12.5195 | 25.2996 | 98.816 | 12.372 | 34.75 | 48.25 |
+| 45000 | 0.7273 | 50.75 | 135.0 | 12.2859 | 25.2872 | 98.864 | 12.378 | 29.25 | 43.5 |
+| 47500 | 0.7677 | 50.75 | 125.5 | 12.2428 | 25.1481 | 99.411 | 12.446 | 28.75 | 38.25 |
+| 50000 | 0.8081 | 50.25 | 123.5 | 12.2168 | 25.2928 | 98.842 | 12.375 | 28.375 | 37.25 |
+| 52500 | 0.8485 | 49.0 | 121.0 | 12.1935 | 25.3029 | 98.803 | 12.37 | 28.125 | 36.0 |
+| 55000 | 0.8889 | 49.0 | 121.0 | 12.1614 | 25.271 | 98.927 | 12.386 | 27.5 | 35.75 |
+| 57500 | 0.9293 | 48.75 | 119.5 | 12.1483 | 25.2276 | 99.098 | 12.407 | 27.5 | 34.25 |
+| 60000 | 0.9697 | 48.75 | 118.5 | 12.1410 | 25.3014 | 98.809 | 12.371 | 27.375 | 33.75 |
+| 61875 | 1.0 | 48.5 | 119.0 | 12.1404 | 25.2551 | 98.99 | 12.394 | 27.25 | 33.75 |
 
 # Resource Usage Comparison
 
@@ -93,7 +93,7 @@ More information needed
 <br/>
 
 # Train Dataset
-Trained on 145,721,245 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
+Trained on 145,744,973 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
 
 - Num Samples: `247,500`
 - Subset: `20231101.en`
@@ -103,7 +103,7 @@ Trained on 145,721,245 tokens from the [wikimedia/wikipedia](https://huggingface
 # Training Objective
 
 ```
-DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=5, loss_fn=cos, layer_mapper=layer-2))
+DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=25.0, loss_fn=cos, layer_mapper=layer-2))
 ```
 
 # Hyperparameters
@@ -120,9 +120,9 @@ The following hyperparameters were used during training:
 - lr_scheduler_type: `linear`
 - lr_scheduler_warmup_ratio: `0.5`
 - num_epochs: `1.0`
-- distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=5, loss_fn=cos, layer_mapper=layer-2))`
+- distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=25.0, loss_fn=cos, layer_mapper=layer-2))`
 - train_embeddings: `True`
-- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7f148c97c340>`
+- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7f148c79de10>`
 - student_model_name_or_path: `None`
 - student_config_name_or_path: `None`
 - student_model_config: `None`

logs/attn_loss_fn=cos, attn_weight=25.0, layer_mapper=all, projector=linear/events.out.tfevents.1724344302.e3f806ea38c9

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:efa919d2018e6a185c123128d04c98b9f9da9763bd3cd45aa4ff1515bb9e2e8e
+size 588