Upload folder using huggingface_hub

README.md

@@ -1,3 +1,169 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+datasets:
+- lmms-lab/ClothoAQA
+- Loie/VGGSound
+language:
+- en
+metrics:
+- accuracy
+pipeline_tag: visual-question-answering
+library_name: transformers
+tags:
+- Audio-visual Question Answering
+- Audio Question Answering
+- multimodal large language model
+---
+
+
+<p align="center">
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/63913b120cf6b11c487ca31d/ROs4bHIp4zJ7g7vzgUycu.png" width="150" style="margin-bottom: 0.2;"/>
+<p>
+
+
+<h3 align="center"><a href="https://arxiv.org/abs/2406.07476">VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs</a></h3>
+<h5 align="center"> If you like our project, please give us a star ⭐ on <a href="https://github.com/DAMO-NLP-SG/VideoLLaMA2">Github</a> for the latest update.  </h2>
+
+<p align="center"><video src="https://cdn-uploads.huggingface.co/production/uploads/63913b120cf6b11c487ca31d/Wj7GuqQ0CB9JRoPo6_GoH.webm" width="800"></p>
+
+## 📰 News
+* **[2024.10.22]**  Release checkpoints of [VideoLLaMA2.1-7B-AV](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2.1-7B-AV)
+* **[2024.10.15]**  Release checkpoints of [VideoLLaMA2.1-7B-16F-Base](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2.1-7B-16F-Base) and [VideoLLaMA2.1-7B-16F](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2.1-7B-16F)
+* **[2024.08.14]**  Release checkpoints of [VideoLLaMA2-72B-Base](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-72B-Base) and [VideoLLaMA2-72B](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-72B)
+* **[2024.07.30]**  Release checkpoints of [VideoLLaMA2-8x7B-Base](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-8x7B-Base) and [VideoLLaMA2-8x7B](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-8x7B).
+* **[2024.06.25]**  🔥🔥 As of Jun 25, our [VideoLLaMA2-7B-16F](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-7B-16F) is the **Top-1** ~7B-sized VideoLLM on the [MLVU Leaderboard](https://github.com/JUNJIE99/MLVU?tab=readme-ov-file#trophy-mini-leaderboard).
+* **[2024.06.18]**  🔥🔥 As of Jun 18, our [VideoLLaMA2-7B-16F](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-7B-16F) is the **Top-1** ~7B-sized VideoLLM on the [VideoMME Leaderboard](https://video-mme.github.io/home_page.html#leaderboard).
+* **[2024.06.17]**  👋👋 Update technical report with the latest results and the missing references. If you have works closely related to VideoLLaMA 2 but not mentioned in the paper, feel free to let us know.  
+* **[2024.06.14]**  🔥🔥 [Online Demo](https://huggingface.co/spaces/lixin4ever/VideoLLaMA2) is available.
+* **[2024.06.03]**  Release training, evaluation, and serving codes of VideoLLaMA 2.
+
+
+## 🌎 Model Zoo
+### Vision-Only Checkpoints
+| Model Name     | Type | Visual Encoder | Language Decoder | # Training Frames |
+|:-------------------|:--------------:|:----------------|:------------------|:----------------------:|
+| [VideoLLaMA2-7B-Base](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-7B-Base)  | Base  | [clip-vit-large-patch14-336](https://huggingface.co/openai/clip-vit-large-patch14-336) | [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)  | 8 |
+| [VideoLLaMA2-7B](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-7B)  | Chat | [clip-vit-large-patch14-336](https://huggingface.co/openai/clip-vit-large-patch14-336) | [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)  | 8 |
+| [VideoLLaMA2-7B-16F-Base](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-7B-16F-Base)  | Base  | [clip-vit-large-patch14-336](https://huggingface.co/openai/clip-vit-large-patch14-336) | [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)  | 16 |
+| [VideoLLaMA2-7B-16F](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-7B-16F)  | Chat | [clip-vit-large-patch14-336](https://huggingface.co/openai/clip-vit-large-patch14-336) | [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)  | 16 |
+| [VideoLLaMA2-8x7B-Base](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-8x7B-Base)  | Base | [clip-vit-large-patch14-336](https://huggingface.co/openai/clip-vit-large-patch14-336) | [Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)  | 8 |
+| [VideoLLaMA2-8x7B](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-8x7B)  | Chat | [clip-vit-large-patch14-336](https://huggingface.co/openai/clip-vit-large-patch14-336) | [Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)  | 8 |
+| [VideoLLaMA2-72B-Base](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-72B-Base)  | Base | [clip-vit-large-patch14-336](https://huggingface.co/openai/clip-vit-large-patch14-336) | [Qwen2-72B-Instruct](https://huggingface.co/Qwen/Qwen2-72B-Instruct)  | 8 |
+| [VideoLLaMA2-72B](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-72B)  | Chat | [clip-vit-large-patch14-336](https://huggingface.co/openai/clip-vit-large-patch14-336) | [Qwen2-72B-Instruct](https://huggingface.co/Qwen/Qwen2-72B-Instruct)  | 8 |
+| [VideoLLaMA2.1-7B-16F-Base](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2.1-7B-16F-Base)  | Base | [siglip-so400m-patch14-384](https://huggingface.co/google/siglip-so400m-patch14-384) | [Qwen2-7B-Instruct](https://huggingface.co/Qwen/Qwen2-7B-Instruct)  | 16 |
+| [VideoLLaMA2.1-7B-16F](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2.1-7B-16F)  | Chat | [siglip-so400m-patch14-384](https://huggingface.co/google/siglip-so400m-patch14-384) | [Qwen2-7B-Instruct](https://huggingface.co/Qwen/Qwen2-7B-Instruct)  | 16 |
+
+### Audio-Visual Checkpoints
+| Model Name     | Type | Audio Encoder | Language Decoder |  
+|:-------------------|:--------------:|:----------------|:----------------------:|  
+| [VideoLLaMA2.1-7B-AV](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2.1-7B-AV) (**This Checkpoint**)  | Chat | [Fine-tuned BEATs_iter3+(AS2M)(cpt2)](https://1drv.ms/u/s!AqeByhGUtINrgcpj8ujXH1YUtxooEg?e=E9Ncea) | [VideoLLaMA2.1-7B-16F](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2.1-7B-16F)  |
+
+
+## 🚀 Main Results
+
+### Multi-Choice Video QA & Video Captioning
+<p><img src="https://cdn-uploads.huggingface.co/production/uploads/63913b120cf6b11c487ca31d/Z81Dl2MeVlg8wLbYOyTvI.png" width="800" "/></p>
+
+###  Open-Ended Video QA
+<p><img src="https://cdn-uploads.huggingface.co/production/uploads/63913b120cf6b11c487ca31d/UoAr7SjbPSPe1z23HBsUh.png" width="800" "/></p>
+
+### Multi-Choice & Open-Ended Audio QA 
+<p><img src="https://huggingface.co/YifeiXin/xin/resolve/main/VideoLLaMA2-audio.png" width="800" "/></p>
+
+### Open-Ended Audio-Visual QA 
+<p><img src="https://huggingface.co/YifeiXin/xin/resolve/main/VideoLLaAM2.1-AV.png" width="800" "/></p>
+
+
+
+## 🤖 Inference with VideoLLaMA2-AV
+```python
+import sys
+sys.path.append('./')
+from videollama2 import model_init, mm_infer
+from videollama2.utils import disable_torch_init
+import argparse
+
+def inference(args):
+
+    model_path = args.model_path
+    model, processor, tokenizer = model_init(model_path)
+
+    if args.modal_type == "a":
+        model.model.vision_tower = None
+    elif args.modal_type == "v":
+        model.model.audio_tower = None
+    elif args.modal_type == "av":
+        pass
+    else:
+        raise NotImplementedError
+    # Audio-visual Inference
+    audio_video_path = "assets/00003491.mp4"
+    preprocess = processor['audio' if args.modal_type == "a" else "video"]
+    if args.modal_type == "a":
+        audio_video_tensor = preprocess(audio_video_path)
+    else:
+        audio_video_tensor = preprocess(audio_video_path, va=True if args.modal_type == "av" else False)
+    question = f"Please describe the video with audio information."
+
+    # Audio Inference
+    audio_video_path = "assets/bird-twitter-car.wav"
+    preprocess = processor['audio' if args.modal_type == "a" else "video"]
+    if args.modal_type == "a":
+        audio_video_tensor = preprocess(audio_video_path)
+    else:
+        audio_video_tensor = preprocess(audio_video_path, va=True if args.modal_type == "av" else False)
+    question = f"Please describe the audio."
+
+    # Video Inference
+    audio_video_path = "assets/output_v_1jgsRbGzCls.mp4"
+    preprocess = processor['audio' if args.modal_type == "a" else "video"]
+    if args.modal_type == "a":
+        audio_video_tensor = preprocess(audio_video_path)
+    else:
+        audio_video_tensor = preprocess(audio_video_path, va=True if args.modal_type == "av" else False)
+    question = f"What activity are the people practicing in the video?"
+
+    output = mm_infer(
+        audio_video_tensor,
+        question,
+        model=model,
+        tokenizer=tokenizer,
+        modal='audio' if args.modal_type == "a" else "video",
+        do_sample=False,
+    )
+
+    print(output)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument('--model-path', help='', , required=False, default='DAMO-NLP-SG/VideoLLaMA2.1-7B-AV')
+    parser.add_argument('--modal-type', choices=["a", "v", "av"], help='', required=True)
+    args = parser.parse_args()
+
+    inference(args)
+
+```
+
+
+## Citation
+
+If you find VideoLLaMA useful for your research and applications, please cite using this BibTeX:
+```bibtex
+@article{damonlpsg2024videollama2,
+  title={VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs},
+  author={Cheng, Zesen and Leng, Sicong and Zhang, Hang and Xin, Yifei and Li, Xin and Chen, Guanzheng and Zhu, Yongxin and Zhang, Wenqi and Luo, Ziyang and Zhao, Deli and Bing, Lidong},
+  journal={arXiv preprint arXiv:2406.07476},
+  year={2024},
+  url = {https://arxiv.org/abs/2406.07476}
+}
+
+@article{damonlpsg2023videollama,
+  title = {Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding},
+  author = {Zhang, Hang and Li, Xin and Bing, Lidong},
+  journal = {arXiv preprint arXiv:2306.02858},
+  year = {2023},
+  url = {https://arxiv.org/abs/2306.02858}
+}
+```

added_tokens.json

@@ -0,0 +1,5 @@
+{
+  "<|endoftext|>": 151643,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644
+}

audio_tower.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7afcafb3a7ba5640cc79e8a6741e215dea015286a2963ef9ff845c115fe445e9
+size 181514532

config.json

@@ -0,0 +1,46 @@
+{
+  "_name_or_path": "pretrained_models/VideoLLaMA2.1-7B-AV",
+  "architectures": [
+    "Videollama2Qwen2ForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "eos_token_id": 151645,
+  "freeze_mm_mlp_adapter": false,
+  "hidden_act": "silu",
+  "hidden_size": 3584,
+  "hidden_size_a": 3584,
+  "image_aspect_ratio": "pad",
+  "initializer_range": 0.02,
+  "intermediate_size": 18944,
+  "max_position_embeddings": 32768,
+  "max_window_layers": 28,
+  "mm_audio_tower": "pretrained_models/VideoLLaMA2.1-7B-AV/audio_tower.bin",
+  "mm_hidden_size": 1152,
+  "mm_hidden_size_a": 768,
+  "mm_projector_a_type": "mlp2x_gelu",
+  "mm_projector_lr": null,
+  "mm_projector_type": "stc_connector_v35",
+  "mm_vision_select_feature": "patch",
+  "mm_vision_select_layer": -2,
+  "mm_vision_tower": "/ailab-train/speech/liangyunming/20250212/VideoLLaMA2/siglip-so400m-patch14-384",
+  "model_type": "videollama2_qwen2",
+  "num_attention_heads": 28,
+  "num_frames": 8,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 4,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 1000000.0,
+  "sliding_window": 131072,
+  "tie_word_embeddings": false,
+  "tokenizer_model_max_length": 2048,
+  "tokenizer_padding_side": "right",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.42.3",
+  "tune_mm_mlp_adapter": false,
+  "tune_mm_mlp_adapter_a": true,
+  "use_cache": true,
+  "use_mm_proj": true,
+  "use_sliding_window": false,
+  "vocab_size": 152064
+}

generation_config.json

@@ -0,0 +1,14 @@
+{
+  "bos_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "repetition_penalty": 1.05,
+  "temperature": 0.7,
+  "top_k": 20,
+  "top_p": 0.8,
+  "transformers_version": "4.42.3"
+}

mm_projector_a.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d38f880ad4ea6a4a55a3dafce476137857edbcdd5e23abb6342bae98a7332f1
+size 31211592

model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a4fe85086019948241aaecd3deb6317182a98dd3be85051bf9ecc4cb2e797208
+size 4877660776

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2360f34ccbefcdf6e6d1b69332f50543fca080a0e84b522e543b06bba3011574
+size 4932751008

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4772cece868ce7cff8898b65ab9e2745196145f5467dced47355be96fb9167c1
+size 4994571904

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:87c9d27142c0f2175928b61253435cf6166662c7676684358c7c1f7c1c887291
+size 2249362518

special_tokens_map.json

@@ -0,0 +1,20 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,44 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "model_max_length": 2048,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "right",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

trainer_state.json

@@ -0,0 +1,1064 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 146,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.00684931506849315,
+      "grad_norm": 19.29674702806549,
+      "learning_rate": 4.000000000000001e-06,
+      "loss": 1.7245,
+      "step": 1
+    },
+    {
+      "epoch": 0.0136986301369863,
+      "grad_norm": 19.21857209627261,
+      "learning_rate": 8.000000000000001e-06,
+      "loss": 1.7229,
+      "step": 2
+    },
+    {
+      "epoch": 0.02054794520547945,
+      "grad_norm": 17.41514384667566,
+      "learning_rate": 1.2e-05,
+      "loss": 1.5905,
+      "step": 3
+    },
+    {
+      "epoch": 0.0273972602739726,
+      "grad_norm": 11.529843794329034,
+      "learning_rate": 1.6000000000000003e-05,
+      "loss": 0.8558,
+      "step": 4
+    },
+    {
+      "epoch": 0.03424657534246575,
+      "grad_norm": 10.40509960722128,
+      "learning_rate": 2e-05,
+      "loss": 0.4032,
+      "step": 5
+    },
+    {
+      "epoch": 0.0410958904109589,
+      "grad_norm": 7.504234388952183,
+      "learning_rate": 1.9997517932672592e-05,
+      "loss": 0.1401,
+      "step": 6
+    },
+    {
+      "epoch": 0.04794520547945205,
+      "grad_norm": 3.122635162419864,
+      "learning_rate": 1.999007296282201e-05,
+      "loss": 0.0589,
+      "step": 7
+    },
+    {
+      "epoch": 0.0547945205479452,
+      "grad_norm": 1.0239540198472452,
+      "learning_rate": 1.9977668786231536e-05,
+      "loss": 0.015,
+      "step": 8
+    },
+    {
+      "epoch": 0.06164383561643835,
+      "grad_norm": 0.7641911750868134,
+      "learning_rate": 1.9960311560501457e-05,
+      "loss": 0.0112,
+      "step": 9
+    },
+    {
+      "epoch": 0.0684931506849315,
+      "grad_norm": 0.7361915045518228,
+      "learning_rate": 1.993800990199235e-05,
+      "loss": 0.0198,
+      "step": 10
+    },
+    {
+      "epoch": 0.07534246575342465,
+      "grad_norm": 0.48133196299135517,
+      "learning_rate": 1.9910774881547803e-05,
+      "loss": 0.0233,
+      "step": 11
+    },
+    {
+      "epoch": 0.0821917808219178,
+      "grad_norm": 0.45830620408895073,
+      "learning_rate": 1.9878620018998696e-05,
+      "loss": 0.0205,
+      "step": 12
+    },
+    {
+      "epoch": 0.08904109589041095,
+      "grad_norm": 0.40251365492026353,
+      "learning_rate": 1.984156127645178e-05,
+      "loss": 0.0135,
+      "step": 13
+    },
+    {
+      "epoch": 0.0958904109589041,
+      "grad_norm": 0.21100873184731916,
+      "learning_rate": 1.979961705036587e-05,
+      "loss": 0.01,
+      "step": 14
+    },
+    {
+      "epoch": 0.10273972602739725,
+      "grad_norm": 0.10663215007709867,
+      "learning_rate": 1.975280816241959e-05,
+      "loss": 0.0059,
+      "step": 15
+    },
+    {
+      "epoch": 0.1095890410958904,
+      "grad_norm": 0.2136985001931379,
+      "learning_rate": 1.970115784917523e-05,
+      "loss": 0.0066,
+      "step": 16
+    },
+    {
+      "epoch": 0.11643835616438356,
+      "grad_norm": 0.19320727681702302,
+      "learning_rate": 1.964469175054377e-05,
+      "loss": 0.0061,
+      "step": 17
+    },
+    {
+      "epoch": 0.1232876712328767,
+      "grad_norm": 0.22488490193570698,
+      "learning_rate": 1.9583437897056915e-05,
+      "loss": 0.0054,
+      "step": 18
+    },
+    {
+      "epoch": 0.13013698630136986,
+      "grad_norm": 0.12949314004112783,
+      "learning_rate": 1.9517426695952358e-05,
+      "loss": 0.0048,
+      "step": 19
+    },
+    {
+      "epoch": 0.136986301369863,
+      "grad_norm": 0.07325589720293409,
+      "learning_rate": 1.944669091607919e-05,
+      "loss": 0.0046,
+      "step": 20
+    },
+    {
+      "epoch": 0.14383561643835616,
+      "grad_norm": 0.2667026911328205,
+      "learning_rate": 1.9371265671631038e-05,
+      "loss": 0.0049,
+      "step": 21
+    },
+    {
+      "epoch": 0.1506849315068493,
+      "grad_norm": 0.23774565539015635,
+      "learning_rate": 1.9291188404714876e-05,
+      "loss": 0.0047,
+      "step": 22
+    },
+    {
+      "epoch": 0.15753424657534246,
+      "grad_norm": 0.06311021183333536,
+      "learning_rate": 1.920649886676429e-05,
+      "loss": 0.0038,
+      "step": 23
+    },
+    {
+      "epoch": 0.1643835616438356,
+      "grad_norm": 0.18265444495687314,
+      "learning_rate": 1.9117239098806296e-05,
+      "loss": 0.0036,
+      "step": 24
+    },
+    {
+      "epoch": 0.17123287671232876,
+      "grad_norm": 0.17871695956695147,
+      "learning_rate": 1.902345341059164e-05,
+      "loss": 0.0034,
+      "step": 25
+    },
+    {
+      "epoch": 0.1780821917808219,
+      "grad_norm": 0.14022615585862463,
+      "learning_rate": 1.8925188358598815e-05,
+      "loss": 0.003,
+      "step": 26
+    },
+    {
+      "epoch": 0.18493150684931506,
+      "grad_norm": 0.4217735239695949,
+      "learning_rate": 1.882249272292282e-05,
+      "loss": 0.0036,
+      "step": 27
+    },
+    {
+      "epoch": 0.1917808219178082,
+      "grad_norm": 0.2745329088554571,
+      "learning_rate": 1.871541748306005e-05,
+      "loss": 0.003,
+      "step": 28
+    },
+    {
+      "epoch": 0.19863013698630136,
+      "grad_norm": 0.23380507547266036,
+      "learning_rate": 1.8604015792601395e-05,
+      "loss": 0.0028,
+      "step": 29
+    },
+    {
+      "epoch": 0.2054794520547945,
+      "grad_norm": 0.1428536424621058,
+      "learning_rate": 1.8488342952846074e-05,
+      "loss": 0.0023,
+      "step": 30
+    },
+    {
+      "epoch": 0.21232876712328766,
+      "grad_norm": 0.16718027431083146,
+      "learning_rate": 1.8368456385349333e-05,
+      "loss": 0.0025,
+      "step": 31
+    },
+    {
+      "epoch": 0.2191780821917808,
+      "grad_norm": 0.18797868021085826,
+      "learning_rate": 1.8244415603417603e-05,
+      "loss": 0.0027,
+      "step": 32
+    },
+    {
+      "epoch": 0.22602739726027396,
+      "grad_norm": 0.12331990781903307,
+      "learning_rate": 1.8116282182565313e-05,
+      "loss": 0.0024,
+      "step": 33
+    },
+    {
+      "epoch": 0.2328767123287671,
+      "grad_norm": 0.16756761059169867,
+      "learning_rate": 1.7984119729947944e-05,
+      "loss": 0.002,
+      "step": 34
+    },
+    {
+      "epoch": 0.23972602739726026,
+      "grad_norm": 0.14512823117742887,
+      "learning_rate": 1.7847993852786612e-05,
+      "loss": 0.0022,
+      "step": 35
+    },
+    {
+      "epoch": 0.2465753424657534,
+      "grad_norm": 0.033276917673730214,
+      "learning_rate": 1.7707972125799738e-05,
+      "loss": 0.0018,
+      "step": 36
+    },
+    {
+      "epoch": 0.2534246575342466,
+      "grad_norm": 0.05328530240886233,
+      "learning_rate": 1.7564124057658057e-05,
+      "loss": 0.0017,
+      "step": 37
+    },
+    {
+      "epoch": 0.2602739726027397,
+      "grad_norm": 0.17408349740736515,
+      "learning_rate": 1.7416521056479577e-05,
+      "loss": 0.0024,
+      "step": 38
+    },
+    {
+      "epoch": 0.2671232876712329,
+      "grad_norm": 0.1289878517079893,
+      "learning_rate": 1.7265236394381634e-05,
+      "loss": 0.002,
+      "step": 39
+    },
+    {
+      "epoch": 0.273972602739726,
+      "grad_norm": 0.07727670281756385,
+      "learning_rate": 1.711034517110761e-05,
+      "loss": 0.0018,
+      "step": 40
+    },
+    {
+      "epoch": 0.2808219178082192,
+      "grad_norm": 0.0935035121293635,
+      "learning_rate": 1.6951924276746425e-05,
+      "loss": 0.002,
+      "step": 41
+    },
+    {
+      "epoch": 0.2876712328767123,
+      "grad_norm": 0.10355890785277629,
+      "learning_rate": 1.6790052353563254e-05,
+      "loss": 0.0022,
+      "step": 42
+    },
+    {
+      "epoch": 0.2945205479452055,
+      "grad_norm": 0.04714349399211733,
+      "learning_rate": 1.6624809756960445e-05,
+      "loss": 0.0013,
+      "step": 43
+    },
+    {
+      "epoch": 0.3013698630136986,
+      "grad_norm": 0.054834818605121315,
+      "learning_rate": 1.6456278515588023e-05,
+      "loss": 0.0015,
+      "step": 44
+    },
+    {
+      "epoch": 0.3082191780821918,
+      "grad_norm": 0.06250829705501665,
+      "learning_rate": 1.6284542290623568e-05,
+      "loss": 0.0013,
+      "step": 45
+    },
+    {
+      "epoch": 0.3150684931506849,
+      "grad_norm": 0.0563640070785858,
+      "learning_rate": 1.6109686334241655e-05,
+      "loss": 0.0016,
+      "step": 46
+    },
+    {
+      "epoch": 0.3219178082191781,
+      "grad_norm": 0.09704124691215613,
+      "learning_rate": 1.5931797447293553e-05,
+      "loss": 0.0017,
+      "step": 47
+    },
+    {
+      "epoch": 0.3287671232876712,
+      "grad_norm": 0.07610196592393692,
+      "learning_rate": 1.5750963936218104e-05,
+      "loss": 0.0014,
+      "step": 48
+    },
+    {
+      "epoch": 0.3356164383561644,
+      "grad_norm": 0.11034470568619353,
+      "learning_rate": 1.5567275569205216e-05,
+      "loss": 0.0018,
+      "step": 49
+    },
+    {
+      "epoch": 0.3424657534246575,
+      "grad_norm": 0.05344287464292232,
+      "learning_rate": 1.5380823531633727e-05,
+      "loss": 0.0019,
+      "step": 50
+    },
+    {
+      "epoch": 0.3493150684931507,
+      "grad_norm": 0.06818765213019927,
+      "learning_rate": 1.5191700380805754e-05,
+      "loss": 0.001,
+      "step": 51
+    },
+    {
+      "epoch": 0.3561643835616438,
+      "grad_norm": 0.07736500670457502,
+      "learning_rate": 1.5000000000000002e-05,
+      "loss": 0.0013,
+      "step": 52
+    },
+    {
+      "epoch": 0.363013698630137,
+      "grad_norm": 0.08416203716950547,
+      "learning_rate": 1.4805817551866839e-05,
+      "loss": 0.0013,
+      "step": 53
+    },
+    {
+      "epoch": 0.3698630136986301,
+      "grad_norm": 0.08000624861117452,
+      "learning_rate": 1.460924943118828e-05,
+      "loss": 0.0017,
+      "step": 54
+    },
+    {
+      "epoch": 0.3767123287671233,
+      "grad_norm": 0.026831863602624892,
+      "learning_rate": 1.4410393217026317e-05,
+      "loss": 0.0014,
+      "step": 55
+    },
+    {
+      "epoch": 0.3835616438356164,
+      "grad_norm": 0.026046969137480933,
+      "learning_rate": 1.4209347624283352e-05,
+      "loss": 0.0012,
+      "step": 56
+    },
+    {
+      "epoch": 0.3904109589041096,
+      "grad_norm": 0.06996591904539229,
+      "learning_rate": 1.4006212454698798e-05,
+      "loss": 0.0015,
+      "step": 57
+    },
+    {
+      "epoch": 0.3972602739726027,
+      "grad_norm": 0.08076573337383759,
+      "learning_rate": 1.3801088547306149e-05,
+      "loss": 0.0015,
+      "step": 58
+    },
+    {
+      "epoch": 0.4041095890410959,
+      "grad_norm": 0.06814214008336002,
+      "learning_rate": 1.3594077728375129e-05,
+      "loss": 0.0013,
+      "step": 59
+    },
+    {
+      "epoch": 0.410958904109589,
+      "grad_norm": 0.045916785375625435,
+      "learning_rate": 1.3385282760863758e-05,
+      "loss": 0.0017,
+      "step": 60
+    },
+    {
+      "epoch": 0.4178082191780822,
+      "grad_norm": 0.03587505653315555,
+      "learning_rate": 1.3174807293405427e-05,
+      "loss": 0.0012,
+      "step": 61
+    },
+    {
+      "epoch": 0.4246575342465753,
+      "grad_norm": 0.04958824946765545,
+      "learning_rate": 1.2962755808856341e-05,
+      "loss": 0.0017,
+      "step": 62
+    },
+    {
+      "epoch": 0.4315068493150685,
+      "grad_norm": 0.055768018261520134,
+      "learning_rate": 1.2749233572428805e-05,
+      "loss": 0.0015,
+      "step": 63
+    },
+    {
+      "epoch": 0.4383561643835616,
+      "grad_norm": 0.022968020713589454,
+      "learning_rate": 1.2534346579436158e-05,
+      "loss": 0.0012,
+      "step": 64
+    },
+    {
+      "epoch": 0.4452054794520548,
+      "grad_norm": 0.029150590324992027,
+      "learning_rate": 1.2318201502675285e-05,
+      "loss": 0.0017,
+      "step": 65
+    },
+    {
+      "epoch": 0.4520547945205479,
+      "grad_norm": 0.0838087444466505,
+      "learning_rate": 1.210090563947278e-05,
+      "loss": 0.0012,
+      "step": 66
+    },
+    {
+      "epoch": 0.4589041095890411,
+      "grad_norm": 0.03256867146120048,
+      "learning_rate": 1.1882566858421137e-05,
+      "loss": 0.0014,
+      "step": 67
+    },
+    {
+      "epoch": 0.4657534246575342,
+      "grad_norm": 0.05522775344747511,
+      "learning_rate": 1.1663293545831302e-05,
+      "loss": 0.0016,
+      "step": 68
+    },
+    {
+      "epoch": 0.4726027397260274,
+      "grad_norm": 0.04414044075585935,
+      "learning_rate": 1.1443194551928267e-05,
+      "loss": 0.0015,
+      "step": 69
+    },
+    {
+      "epoch": 0.4794520547945205,
+      "grad_norm": 0.03049587207603872,
+      "learning_rate": 1.1222379136816347e-05,
+      "loss": 0.0012,
+      "step": 70
+    },
+    {
+      "epoch": 0.4863013698630137,
+      "grad_norm": 0.04682460462951094,
+      "learning_rate": 1.1000956916240985e-05,
+      "loss": 0.0012,
+      "step": 71
+    },
+    {
+      "epoch": 0.4931506849315068,
+      "grad_norm": 0.05892485409761165,
+      "learning_rate": 1.0779037807174032e-05,
+      "loss": 0.0011,
+      "step": 72
+    },
+    {
+      "epoch": 0.5,
+      "grad_norm": 0.03550653877962121,
+      "learning_rate": 1.0556731973249486e-05,
+      "loss": 0.0013,
+      "step": 73
+    },
+    {
+      "epoch": 0.5068493150684932,
+      "grad_norm": 0.0894703655391614,
+      "learning_rate": 1.0334149770076747e-05,
+      "loss": 0.0008,
+      "step": 74
+    },
+    {
+      "epoch": 0.5136986301369864,
+      "grad_norm": 0.030497628260740314,
+      "learning_rate": 1.0111401690458655e-05,
+      "loss": 0.0013,
+      "step": 75
+    },
+    {
+      "epoch": 0.5205479452054794,
+      "grad_norm": 0.06677410842425625,
+      "learning_rate": 9.888598309541347e-06,
+      "loss": 0.0017,
+      "step": 76
+    },
+    {
+      "epoch": 0.5273972602739726,
+      "grad_norm": 0.07484818283470532,
+      "learning_rate": 9.665850229923258e-06,
+      "loss": 0.0011,
+      "step": 77
+    },
+    {
+      "epoch": 0.5342465753424658,
+      "grad_norm": 0.025732343070697165,
+      "learning_rate": 9.443268026750521e-06,
+      "loss": 0.0008,
+      "step": 78
+    },
+    {
+      "epoch": 0.541095890410959,
+      "grad_norm": 0.034210499510211344,
+      "learning_rate": 9.22096219282597e-06,
+      "loss": 0.0007,
+      "step": 79
+    },
+    {
+      "epoch": 0.547945205479452,
+      "grad_norm": 0.10015949753448726,
+      "learning_rate": 8.999043083759016e-06,
+      "loss": 0.0015,
+      "step": 80
+    },
+    {
+      "epoch": 0.5547945205479452,
+      "grad_norm": 0.09275161932686507,
+      "learning_rate": 8.777620863183658e-06,
+      "loss": 0.0011,
+      "step": 81
+    },
+    {
+      "epoch": 0.5616438356164384,
+      "grad_norm": 0.05947458847354757,
+      "learning_rate": 8.556805448071736e-06,
+      "loss": 0.0008,
+      "step": 82
+    },
+    {
+      "epoch": 0.5684931506849316,
+      "grad_norm": 0.0294571121538469,
+      "learning_rate": 8.336706454168701e-06,
+      "loss": 0.001,
+      "step": 83
+    },
+    {
+      "epoch": 0.5753424657534246,
+      "grad_norm": 0.04241767860802742,
+      "learning_rate": 8.117433141578865e-06,
+      "loss": 0.0011,
+      "step": 84
+    },
+    {
+      "epoch": 0.5821917808219178,
+      "grad_norm": 0.06495783919409748,
+      "learning_rate": 7.89909436052722e-06,
+      "loss": 0.0016,
+      "step": 85
+    },
+    {
+      "epoch": 0.589041095890411,
+      "grad_norm": 0.032464640498104755,
+      "learning_rate": 7.681798497324717e-06,
+      "loss": 0.0013,
+      "step": 86
+    },
+    {
+      "epoch": 0.5958904109589042,
+      "grad_norm": 0.05908691816887433,
+      "learning_rate": 7.465653420563846e-06,
+      "loss": 0.0013,
+      "step": 87
+    },
+    {
+      "epoch": 0.6027397260273972,
+      "grad_norm": 0.027918707497596708,
+      "learning_rate": 7.2507664275712e-06,
+      "loss": 0.0009,
+      "step": 88
+    },
+    {
+      "epoch": 0.6095890410958904,
+      "grad_norm": 0.04718877763216423,
+      "learning_rate": 7.037244191143662e-06,
+      "loss": 0.0012,
+      "step": 89
+    },
+    {
+      "epoch": 0.6164383561643836,
+      "grad_norm": 0.029120200990469487,
+      "learning_rate": 6.8251927065945755e-06,
+      "loss": 0.0009,
+      "step": 90
+    },
+    {
+      "epoch": 0.6232876712328768,
+      "grad_norm": 0.04632805174112383,
+      "learning_rate": 6.614717239136246e-06,
+      "loss": 0.0008,
+      "step": 91
+    },
+    {
+      "epoch": 0.6301369863013698,
+      "grad_norm": 0.02233614952979064,
+      "learning_rate": 6.405922271624874e-06,
+      "loss": 0.0012,
+      "step": 92
+    },
+    {
+      "epoch": 0.636986301369863,
+      "grad_norm": 0.03987865814667557,
+      "learning_rate": 6.1989114526938535e-06,
+      "loss": 0.0007,
+      "step": 93
+    },
+    {
+      "epoch": 0.6438356164383562,
+      "grad_norm": 0.04518664140322487,
+      "learning_rate": 5.993787545301204e-06,
+      "loss": 0.0013,
+      "step": 94
+    },
+    {
+      "epoch": 0.6506849315068494,
+      "grad_norm": 0.05916217857082617,
+      "learning_rate": 5.790652375716653e-06,
+      "loss": 0.0012,
+      "step": 95
+    },
+    {
+      "epoch": 0.6575342465753424,
+      "grad_norm": 0.05563218539193664,
+      "learning_rate": 5.589606782973683e-06,
+      "loss": 0.0013,
+      "step": 96
+    },
+    {
+      "epoch": 0.6643835616438356,
+      "grad_norm": 0.06101113089335977,
+      "learning_rate": 5.39075056881172e-06,
+      "loss": 0.0008,
+      "step": 97
+    },
+    {
+      "epoch": 0.6712328767123288,
+      "grad_norm": 0.028524450712115297,
+      "learning_rate": 5.194182448133163e-06,
+      "loss": 0.0008,
+      "step": 98
+    },
+    {
+      "epoch": 0.678082191780822,
+      "grad_norm": 0.04585647335088978,
+      "learning_rate": 5.000000000000003e-06,
+      "loss": 0.001,
+      "step": 99
+    },
+    {
+      "epoch": 0.684931506849315,
+      "grad_norm": 0.03557767533880564,
+      "learning_rate": 4.808299619194251e-06,
+      "loss": 0.001,
+      "step": 100
+    },
+    {
+      "epoch": 0.6917808219178082,
+      "grad_norm": 0.035139350217428346,
+      "learning_rate": 4.619176468366274e-06,
+      "loss": 0.0012,
+      "step": 101
+    },
+    {
+      "epoch": 0.6986301369863014,
+      "grad_norm": 0.03923989663420306,
+      "learning_rate": 4.432724430794786e-06,
+      "loss": 0.0014,
+      "step": 102
+    },
+    {
+      "epoch": 0.7054794520547946,
+      "grad_norm": 0.05465108299814873,
+      "learning_rate": 4.2490360637818965e-06,
+      "loss": 0.001,
+      "step": 103
+    },
+    {
+      "epoch": 0.7123287671232876,
+      "grad_norm": 0.05277740141204182,
+      "learning_rate": 4.0682025527064486e-06,
+      "loss": 0.0012,
+      "step": 104
+    },
+    {
+      "epoch": 0.7191780821917808,
+      "grad_norm": 0.04069062627827433,
+      "learning_rate": 3.890313665758348e-06,
+      "loss": 0.001,
+      "step": 105
+    },
+    {
+      "epoch": 0.726027397260274,
+      "grad_norm": 0.027683781214485625,
+      "learning_rate": 3.7154577093764334e-06,
+      "loss": 0.0012,
+      "step": 106
+    },
+    {
+      "epoch": 0.7328767123287672,
+      "grad_norm": 0.04082961220757915,
+      "learning_rate": 3.543721484411976e-06,
+      "loss": 0.0013,
+      "step": 107
+    },
+    {
+      "epoch": 0.7397260273972602,
+      "grad_norm": 0.057665851827404115,
+      "learning_rate": 3.3751902430395558e-06,
+      "loss": 0.0012,
+      "step": 108
+    },
+    {
+      "epoch": 0.7465753424657534,
+      "grad_norm": 0.03184853057743126,
+      "learning_rate": 3.209947646436752e-06,
+      "loss": 0.0015,
+      "step": 109
+    },
+    {
+      "epoch": 0.7534246575342466,
+      "grad_norm": 0.03058775070190777,
+      "learning_rate": 3.0480757232535773e-06,
+      "loss": 0.001,
+      "step": 110
+    },
+    {
+      "epoch": 0.7602739726027398,
+      "grad_norm": 0.021197542044870794,
+      "learning_rate": 2.889654828892393e-06,
+      "loss": 0.001,
+      "step": 111
+    },
+    {
+      "epoch": 0.7671232876712328,
+      "grad_norm": 0.026836346212118074,
+      "learning_rate": 2.73476360561837e-06,
+      "loss": 0.001,
+      "step": 112
+    },
+    {
+      "epoch": 0.773972602739726,
+      "grad_norm": 0.02567047668470243,
+      "learning_rate": 2.5834789435204245e-06,
+      "loss": 0.0013,
+      "step": 113
+    },
+    {
+      "epoch": 0.7808219178082192,
+      "grad_norm": 0.02644076542732853,
+      "learning_rate": 2.4358759423419476e-06,
+      "loss": 0.001,
+      "step": 114
+    },
+    {
+      "epoch": 0.7876712328767124,
+      "grad_norm": 0.028839912523458685,
+      "learning_rate": 2.2920278742002677e-06,
+      "loss": 0.0011,
+      "step": 115
+    },
+    {
+      "epoch": 0.7945205479452054,
+      "grad_norm": 0.04738085952559155,
+      "learning_rate": 2.1520061472133903e-06,
+      "loss": 0.0017,
+      "step": 116
+    },
+    {
+      "epoch": 0.8013698630136986,
+      "grad_norm": 0.06809650532157646,
+      "learning_rate": 2.0158802700520576e-06,
+      "loss": 0.0011,
+      "step": 117
+    },
+    {
+      "epoch": 0.8082191780821918,
+      "grad_norm": 0.023735350189323164,
+      "learning_rate": 1.8837178174346882e-06,
+      "loss": 0.0012,
+      "step": 118
+    },
+    {
+      "epoch": 0.815068493150685,
+      "grad_norm": 0.04205761142071848,
+      "learning_rate": 1.7555843965823992e-06,
+      "loss": 0.0014,
+      "step": 119
+    },
+    {
+      "epoch": 0.821917808219178,
+      "grad_norm": 0.036552531951304985,
+      "learning_rate": 1.6315436146506702e-06,
+      "loss": 0.0008,
+      "step": 120
+    },
+    {
+      "epoch": 0.8287671232876712,
+      "grad_norm": 0.04975466008883568,
+      "learning_rate": 1.5116570471539294e-06,
+      "loss": 0.0014,
+      "step": 121
+    },
+    {
+      "epoch": 0.8356164383561644,
+      "grad_norm": 0.04244581829003469,
+      "learning_rate": 1.3959842073986085e-06,
+      "loss": 0.0017,
+      "step": 122
+    },
+    {
+      "epoch": 0.8424657534246576,
+      "grad_norm": 0.03061287445805811,
+      "learning_rate": 1.2845825169399506e-06,
+      "loss": 0.0014,
+      "step": 123
+    },
+    {
+      "epoch": 0.8493150684931506,
+      "grad_norm": 0.03212166979592859,
+      "learning_rate": 1.1775072770771833e-06,
+      "loss": 0.001,
+      "step": 124
+    },
+    {
+      "epoch": 0.8561643835616438,
+      "grad_norm": 0.030081381253936994,
+      "learning_rate": 1.074811641401189e-06,
+      "loss": 0.0009,
+      "step": 125
+    },
+    {
+      "epoch": 0.863013698630137,
+      "grad_norm": 0.02162685436714291,
+      "learning_rate": 9.765465894083637e-07,
+      "loss": 0.0011,
+      "step": 126
+    },
+    {
+      "epoch": 0.8698630136986302,
+      "grad_norm": 0.06344389689997086,
+      "learning_rate": 8.827609011937066e-07,
+      "loss": 0.001,
+      "step": 127
+    },
+    {
+      "epoch": 0.8767123287671232,
+      "grad_norm": 0.028695281312298433,
+      "learning_rate": 7.935011332357113e-07,
+      "loss": 0.0013,
+      "step": 128
+    },
+    {
+      "epoch": 0.8835616438356164,
+      "grad_norm": 0.04658193757911157,
+      "learning_rate": 7.088115952851238e-07,
+      "loss": 0.0008,
+      "step": 129
+    },
+    {
+      "epoch": 0.8904109589041096,
+      "grad_norm": 0.06588466626095466,
+      "learning_rate": 6.287343283689662e-07,
+      "loss": 0.0012,
+      "step": 130
+    },
+    {
+      "epoch": 0.8972602739726028,
+      "grad_norm": 0.0596321795178758,
+      "learning_rate": 5.533090839208133e-07,
+      "loss": 0.0009,
+      "step": 131
+    },
+    {
+      "epoch": 0.9041095890410958,
+      "grad_norm": 0.026161831276280006,
+      "learning_rate": 4.825733040476465e-07,
+      "loss": 0.0013,
+      "step": 132
+    },
+    {
+      "epoch": 0.910958904109589,
+      "grad_norm": 0.05665994704513745,
+      "learning_rate": 4.165621029430855e-07,
+      "loss": 0.0012,
+      "step": 133
+    },
+    {
+      "epoch": 0.9178082191780822,
+      "grad_norm": 0.03186905845651481,
+      "learning_rate": 3.553082494562354e-07,
+      "loss": 0.0013,
+      "step": 134
+    },
+    {
+      "epoch": 0.9246575342465754,
+      "grad_norm": 0.026290462399068817,
+      "learning_rate": 2.988421508247741e-07,
+      "loss": 0.001,
+      "step": 135
+    },
+    {
+      "epoch": 0.9315068493150684,
+      "grad_norm": 0.027234972781248173,
+      "learning_rate": 2.4719183758041056e-07,
+      "loss": 0.0014,
+      "step": 136
+    },
+    {
+      "epoch": 0.9383561643835616,
+      "grad_norm": 0.02358313731833951,
+      "learning_rate": 2.0038294963413251e-07,
+      "loss": 0.0012,
+      "step": 137
+    },
+    {
+      "epoch": 0.9452054794520548,
+      "grad_norm": 0.024465700570268718,
+      "learning_rate": 1.5843872354822099e-07,
+      "loss": 0.0012,
+      "step": 138
+    },
+    {
+      "epoch": 0.952054794520548,
+      "grad_norm": 0.026052644199600516,
+      "learning_rate": 1.21379981001305e-07,
+      "loss": 0.0013,
+      "step": 139
+    },
+    {
+      "epoch": 0.958904109589041,
+      "grad_norm": 0.03726493241217445,
+      "learning_rate": 8.922511845219972e-08,
+      "loss": 0.0013,
+      "step": 140
+    },
+    {
+      "epoch": 0.9657534246575342,
+      "grad_norm": 0.02392684919003666,
+      "learning_rate": 6.199009800765265e-08,
+      "loss": 0.0008,
+      "step": 141
+    },
+    {
+      "epoch": 0.9726027397260274,
+      "grad_norm": 0.03581373833161317,
+      "learning_rate": 3.96884394985475e-08,
+      "loss": 0.0009,
+      "step": 142
+    },
+    {
+      "epoch": 0.9794520547945206,
+      "grad_norm": 0.051353061359069495,
+      "learning_rate": 2.2331213768468363e-08,
+      "loss": 0.0011,
+      "step": 143
+    },
+    {
+      "epoch": 0.9863013698630136,
+      "grad_norm": 0.03743681014771785,
+      "learning_rate": 9.927037177993593e-09,
+      "loss": 0.0013,
+      "step": 144
+    },
+    {
+      "epoch": 0.9931506849315068,
+      "grad_norm": 0.040135491167517835,
+      "learning_rate": 2.4820673274095207e-09,
+      "loss": 0.0016,
+      "step": 145
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 0.032445402371938076,
+      "learning_rate": 0.0,
+      "loss": 0.0012,
+      "step": 146
+    },
+    {
+      "epoch": 1.0,
+      "step": 146,
+      "total_flos": 2.8412514950499533e+17,
+      "train_loss": 0.046784546577901105,
+      "train_runtime": 2768.684,
+      "train_samples_per_second": 6.743,
+      "train_steps_per_second": 0.053
+    }
+  ],
+  "logging_steps": 1.0,
+  "max_steps": 146,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 2000,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.8412514950499533e+17,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:00760658f9340604217342ba0009dcc5678cd094d63e1198a060a02187a27cbc
+size 6840