Adaptive Senko AI - 30,000 examples, gpt2 base

README.md

@@ -0,0 +1,202 @@
+---
+base_model: gpt2
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.14.0

adapter_config.json

@@ -0,0 +1,32 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "gpt2",
+  "bias": "none",
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_proj",
+    "c_attn"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1cbeee3dfc268cbdbc22f727c37945ebd4275a3ea49512bd0ee049c583c3112b
+size 3253104

checkpoint-11600/README.md

@@ -0,0 +1,202 @@
+---
+base_model: gpt2
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.14.0

checkpoint-11600/adapter_config.json

@@ -0,0 +1,32 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "gpt2",
+  "bias": "none",
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_proj",
+    "c_attn"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-11600/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8137c1eda8ebedf087f23f1f39ad57fa24e9b35b3fc584acc19896ff81984421
+size 3253104

checkpoint-11600/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c6698b23ae4afc12ca05cd227dc927d2c778fd5abc3ce6f3c2c02a8a7d3b3794
+size 6548858

checkpoint-11600/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:752ae17d06428ff7db5e14fdee45efee8a27d6a0dc1859545aba6773ec56480e
+size 14244

checkpoint-11600/scaler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:684a2d80447af9cc76005199cfec8e7dbd71d967ea01cae183e7fcace028e157
+size 988

checkpoint-11600/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f6a219de3badb6cb2e9c9c7b282785dd34d720128250f541f986799be86e14e5
+size 1064

checkpoint-11600/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-11600/tokenizer_config.json

@@ -0,0 +1,21 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-11600/trainer_state.json

@@ -0,0 +1,2122 @@
+{
+  "best_global_step": 11600,
+  "best_metric": 2.4329476356506348,
+  "best_model_checkpoint": "/kaggle/working/senko_adaptive/checkpoint-11600",
+  "epoch": 2.936886750648857,
+  "eval_steps": 200,
+  "global_step": 11600,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0126606317655251,
+      "grad_norm": 0.4274106025695801,
+      "learning_rate": 2.067510548523207e-06,
+      "loss": 3.4405,
+      "step": 50
+    },
+    {
+      "epoch": 0.0253212635310502,
+      "grad_norm": 0.5292551517486572,
+      "learning_rate": 4.177215189873418e-06,
+      "loss": 3.4567,
+      "step": 100
+    },
+    {
+      "epoch": 0.0379818952965753,
+      "grad_norm": 0.7541739344596863,
+      "learning_rate": 6.28691983122363e-06,
+      "loss": 3.4683,
+      "step": 150
+    },
+    {
+      "epoch": 0.0506425270621004,
+      "grad_norm": 0.8833445906639099,
+      "learning_rate": 8.39662447257384e-06,
+      "loss": 3.5084,
+      "step": 200
+    },
+    {
+      "epoch": 0.0506425270621004,
+      "eval_loss": 3.5248923301696777,
+      "eval_runtime": 39.9384,
+      "eval_samples_per_second": 43.968,
+      "eval_steps_per_second": 43.968,
+      "step": 200
+    },
+    {
+      "epoch": 0.0633031588276255,
+      "grad_norm": 0.9998921155929565,
+      "learning_rate": 1.0506329113924052e-05,
+      "loss": 3.359,
+      "step": 250
+    },
+    {
+      "epoch": 0.0759637905931506,
+      "grad_norm": 0.8041885495185852,
+      "learning_rate": 1.2616033755274262e-05,
+      "loss": 3.351,
+      "step": 300
+    },
+    {
+      "epoch": 0.0886244223586757,
+      "grad_norm": 0.9213416576385498,
+      "learning_rate": 1.4725738396624473e-05,
+      "loss": 3.2244,
+      "step": 350
+    },
+    {
+      "epoch": 0.1012850541242008,
+      "grad_norm": 1.0922213792800903,
+      "learning_rate": 1.6835443037974685e-05,
+      "loss": 3.1565,
+      "step": 400
+    },
+    {
+      "epoch": 0.1012850541242008,
+      "eval_loss": 3.151216983795166,
+      "eval_runtime": 39.7515,
+      "eval_samples_per_second": 44.174,
+      "eval_steps_per_second": 44.174,
+      "step": 400
+    },
+    {
+      "epoch": 0.1139456858897259,
+      "grad_norm": 1.4199283123016357,
+      "learning_rate": 1.8945147679324897e-05,
+      "loss": 3.0154,
+      "step": 450
+    },
+    {
+      "epoch": 0.126606317655251,
+      "grad_norm": 1.077143907546997,
+      "learning_rate": 2.1054852320675106e-05,
+      "loss": 3.0456,
+      "step": 500
+    },
+    {
+      "epoch": 0.1392669494207761,
+      "grad_norm": 1.5466052293777466,
+      "learning_rate": 2.3164556962025318e-05,
+      "loss": 2.9099,
+      "step": 550
+    },
+    {
+      "epoch": 0.1519275811863012,
+      "grad_norm": 1.2139467000961304,
+      "learning_rate": 2.5274261603375527e-05,
+      "loss": 2.8839,
+      "step": 600
+    },
+    {
+      "epoch": 0.1519275811863012,
+      "eval_loss": 2.793567657470703,
+      "eval_runtime": 40.2573,
+      "eval_samples_per_second": 43.619,
+      "eval_steps_per_second": 43.619,
+      "step": 600
+    },
+    {
+      "epoch": 0.1645882129518263,
+      "grad_norm": 1.0270315408706665,
+      "learning_rate": 2.738396624472574e-05,
+      "loss": 2.8389,
+      "step": 650
+    },
+    {
+      "epoch": 0.1772488447173514,
+      "grad_norm": 1.5865377187728882,
+      "learning_rate": 2.949367088607595e-05,
+      "loss": 2.8228,
+      "step": 700
+    },
+    {
+      "epoch": 0.18990947648287648,
+      "grad_norm": 1.076073408126831,
+      "learning_rate": 3.160337552742616e-05,
+      "loss": 2.9255,
+      "step": 750
+    },
+    {
+      "epoch": 0.2025701082484016,
+      "grad_norm": 1.4510694742202759,
+      "learning_rate": 3.3713080168776376e-05,
+      "loss": 2.8165,
+      "step": 800
+    },
+    {
+      "epoch": 0.2025701082484016,
+      "eval_loss": 2.667301893234253,
+      "eval_runtime": 40.0906,
+      "eval_samples_per_second": 43.801,
+      "eval_steps_per_second": 43.801,
+      "step": 800
+    },
+    {
+      "epoch": 0.2152307400139267,
+      "grad_norm": 1.5206592082977295,
+      "learning_rate": 3.5822784810126585e-05,
+      "loss": 2.8022,
+      "step": 850
+    },
+    {
+      "epoch": 0.2278913717794518,
+      "grad_norm": 1.173909068107605,
+      "learning_rate": 3.7932489451476794e-05,
+      "loss": 2.8034,
+      "step": 900
+    },
+    {
+      "epoch": 0.24055200354497688,
+      "grad_norm": 1.4551103115081787,
+      "learning_rate": 4.004219409282701e-05,
+      "loss": 2.774,
+      "step": 950
+    },
+    {
+      "epoch": 0.253212635310502,
+      "grad_norm": 1.509749412536621,
+      "learning_rate": 4.215189873417722e-05,
+      "loss": 2.7978,
+      "step": 1000
+    },
+    {
+      "epoch": 0.253212635310502,
+      "eval_loss": 2.603116273880005,
+      "eval_runtime": 39.9617,
+      "eval_samples_per_second": 43.942,
+      "eval_steps_per_second": 43.942,
+      "step": 1000
+    },
+    {
+      "epoch": 0.2658732670760271,
+      "grad_norm": 1.745764136314392,
+      "learning_rate": 4.426160337552743e-05,
+      "loss": 2.7249,
+      "step": 1050
+    },
+    {
+      "epoch": 0.2785338988415522,
+      "grad_norm": 1.6712589263916016,
+      "learning_rate": 4.637130801687764e-05,
+      "loss": 2.7618,
+      "step": 1100
+    },
+    {
+      "epoch": 0.2911945306070773,
+      "grad_norm": 2.256267786026001,
+      "learning_rate": 4.8481012658227845e-05,
+      "loss": 2.7431,
+      "step": 1150
+    },
+    {
+      "epoch": 0.3038551623726024,
+      "grad_norm": 1.5181586742401123,
+      "learning_rate": 4.993434627649597e-05,
+      "loss": 2.778,
+      "step": 1200
+    },
+    {
+      "epoch": 0.3038551623726024,
+      "eval_loss": 2.5704379081726074,
+      "eval_runtime": 40.1791,
+      "eval_samples_per_second": 43.704,
+      "eval_steps_per_second": 43.704,
+      "step": 1200
+    },
+    {
+      "epoch": 0.31651579413812747,
+      "grad_norm": 1.1885608434677124,
+      "learning_rate": 4.969986869255299e-05,
+      "loss": 2.7224,
+      "step": 1250
+    },
+    {
+      "epoch": 0.3291764259036526,
+      "grad_norm": 1.2136404514312744,
+      "learning_rate": 4.946539110861002e-05,
+      "loss": 2.6823,
+      "step": 1300
+    },
+    {
+      "epoch": 0.3418370576691777,
+      "grad_norm": 0.8780732750892639,
+      "learning_rate": 4.9230913524667046e-05,
+      "loss": 2.6961,
+      "step": 1350
+    },
+    {
+      "epoch": 0.3544976894347028,
+      "grad_norm": 1.0844959020614624,
+      "learning_rate": 4.899643594072407e-05,
+      "loss": 2.7014,
+      "step": 1400
+    },
+    {
+      "epoch": 0.3544976894347028,
+      "eval_loss": 2.54587721824646,
+      "eval_runtime": 39.8803,
+      "eval_samples_per_second": 44.032,
+      "eval_steps_per_second": 44.032,
+      "step": 1400
+    },
+    {
+      "epoch": 0.3671583212002279,
+      "grad_norm": 1.3518335819244385,
+      "learning_rate": 4.8761958356781096e-05,
+      "loss": 2.6393,
+      "step": 1450
+    },
+    {
+      "epoch": 0.37981895296575297,
+      "grad_norm": 1.1389687061309814,
+      "learning_rate": 4.852748077283812e-05,
+      "loss": 2.7002,
+      "step": 1500
+    },
+    {
+      "epoch": 0.3924795847312781,
+      "grad_norm": 1.6295430660247803,
+      "learning_rate": 4.829300318889514e-05,
+      "loss": 2.6754,
+      "step": 1550
+    },
+    {
+      "epoch": 0.4051402164968032,
+      "grad_norm": 1.387499451637268,
+      "learning_rate": 4.8058525604952173e-05,
+      "loss": 2.6853,
+      "step": 1600
+    },
+    {
+      "epoch": 0.4051402164968032,
+      "eval_loss": 2.5297553539276123,
+      "eval_runtime": 39.722,
+      "eval_samples_per_second": 44.207,
+      "eval_steps_per_second": 44.207,
+      "step": 1600
+    },
+    {
+      "epoch": 0.4178008482623283,
+      "grad_norm": 1.014020323753357,
+      "learning_rate": 4.7824048021009195e-05,
+      "loss": 2.7275,
+      "step": 1650
+    },
+    {
+      "epoch": 0.4304614800278534,
+      "grad_norm": 1.1505990028381348,
+      "learning_rate": 4.7589570437066216e-05,
+      "loss": 2.6651,
+      "step": 1700
+    },
+    {
+      "epoch": 0.4431221117933785,
+      "grad_norm": 1.1389458179473877,
+      "learning_rate": 4.7355092853123244e-05,
+      "loss": 2.6993,
+      "step": 1750
+    },
+    {
+      "epoch": 0.4557827435589036,
+      "grad_norm": 1.2159587144851685,
+      "learning_rate": 4.7120615269180266e-05,
+      "loss": 2.7239,
+      "step": 1800
+    },
+    {
+      "epoch": 0.4557827435589036,
+      "eval_loss": 2.5201079845428467,
+      "eval_runtime": 39.8313,
+      "eval_samples_per_second": 44.086,
+      "eval_steps_per_second": 44.086,
+      "step": 1800
+    },
+    {
+      "epoch": 0.4684433753244287,
+      "grad_norm": 1.1873971223831177,
+      "learning_rate": 4.6886137685237294e-05,
+      "loss": 2.6368,
+      "step": 1850
+    },
+    {
+      "epoch": 0.48110400708995377,
+      "grad_norm": 1.5109103918075562,
+      "learning_rate": 4.665166010129432e-05,
+      "loss": 2.6827,
+      "step": 1900
+    },
+    {
+      "epoch": 0.4937646388554789,
+      "grad_norm": 1.9981125593185425,
+      "learning_rate": 4.641718251735134e-05,
+      "loss": 2.6513,
+      "step": 1950
+    },
+    {
+      "epoch": 0.506425270621004,
+      "grad_norm": 1.4879294633865356,
+      "learning_rate": 4.6182704933408365e-05,
+      "loss": 2.6433,
+      "step": 2000
+    },
+    {
+      "epoch": 0.506425270621004,
+      "eval_loss": 2.5107176303863525,
+      "eval_runtime": 40.0643,
+      "eval_samples_per_second": 43.83,
+      "eval_steps_per_second": 43.83,
+      "step": 2000
+    },
+    {
+      "epoch": 0.5190859023865291,
+      "grad_norm": 1.2832767963409424,
+      "learning_rate": 4.5952916901144253e-05,
+      "loss": 2.6225,
+      "step": 2050
+    },
+    {
+      "epoch": 0.5317465341520542,
+      "grad_norm": 1.2915899753570557,
+      "learning_rate": 4.5718439317201275e-05,
+      "loss": 2.6592,
+      "step": 2100
+    },
+    {
+      "epoch": 0.5444071659175793,
+      "grad_norm": 1.229929804801941,
+      "learning_rate": 4.54839617332583e-05,
+      "loss": 2.6411,
+      "step": 2150
+    },
+    {
+      "epoch": 0.5570677976831044,
+      "grad_norm": 1.2569608688354492,
+      "learning_rate": 4.524948414931533e-05,
+      "loss": 2.6436,
+      "step": 2200
+    },
+    {
+      "epoch": 0.5570677976831044,
+      "eval_loss": 2.504101514816284,
+      "eval_runtime": 39.8694,
+      "eval_samples_per_second": 44.044,
+      "eval_steps_per_second": 44.044,
+      "step": 2200
+    },
+    {
+      "epoch": 0.5697284294486294,
+      "grad_norm": 1.3688510656356812,
+      "learning_rate": 4.501500656537235e-05,
+      "loss": 2.6819,
+      "step": 2250
+    },
+    {
+      "epoch": 0.5823890612141546,
+      "grad_norm": 1.1405905485153198,
+      "learning_rate": 4.4780528981429374e-05,
+      "loss": 2.6116,
+      "step": 2300
+    },
+    {
+      "epoch": 0.5950496929796797,
+      "grad_norm": 1.453338861465454,
+      "learning_rate": 4.45460513974864e-05,
+      "loss": 2.6154,
+      "step": 2350
+    },
+    {
+      "epoch": 0.6077103247452048,
+      "grad_norm": 1.0401395559310913,
+      "learning_rate": 4.431157381354343e-05,
+      "loss": 2.6018,
+      "step": 2400
+    },
+    {
+      "epoch": 0.6077103247452048,
+      "eval_loss": 2.498344898223877,
+      "eval_runtime": 39.9496,
+      "eval_samples_per_second": 43.955,
+      "eval_steps_per_second": 43.955,
+      "step": 2400
+    },
+    {
+      "epoch": 0.6203709565107299,
+      "grad_norm": 1.4646718502044678,
+      "learning_rate": 4.407709622960045e-05,
+      "loss": 2.5734,
+      "step": 2450
+    },
+    {
+      "epoch": 0.6330315882762549,
+      "grad_norm": 1.3828164339065552,
+      "learning_rate": 4.384261864565748e-05,
+      "loss": 2.6445,
+      "step": 2500
+    },
+    {
+      "epoch": 0.6456922200417801,
+      "grad_norm": 2.1768596172332764,
+      "learning_rate": 4.36081410617145e-05,
+      "loss": 2.6618,
+      "step": 2550
+    },
+    {
+      "epoch": 0.6583528518073052,
+      "grad_norm": 1.6110296249389648,
+      "learning_rate": 4.337366347777152e-05,
+      "loss": 2.6509,
+      "step": 2600
+    },
+    {
+      "epoch": 0.6583528518073052,
+      "eval_loss": 2.4937028884887695,
+      "eval_runtime": 39.8698,
+      "eval_samples_per_second": 44.043,
+      "eval_steps_per_second": 44.043,
+      "step": 2600
+    },
+    {
+      "epoch": 0.6710134835728303,
+      "grad_norm": 1.2363536357879639,
+      "learning_rate": 4.313918589382856e-05,
+      "loss": 2.6274,
+      "step": 2650
+    },
+    {
+      "epoch": 0.6836741153383554,
+      "grad_norm": 2.192110538482666,
+      "learning_rate": 4.290470830988558e-05,
+      "loss": 2.6932,
+      "step": 2700
+    },
+    {
+      "epoch": 0.6963347471038804,
+      "grad_norm": 1.2024074792861938,
+      "learning_rate": 4.26702307259426e-05,
+      "loss": 2.6221,
+      "step": 2750
+    },
+    {
+      "epoch": 0.7089953788694056,
+      "grad_norm": 1.8665797710418701,
+      "learning_rate": 4.243575314199963e-05,
+      "loss": 2.6313,
+      "step": 2800
+    },
+    {
+      "epoch": 0.7089953788694056,
+      "eval_loss": 2.4876773357391357,
+      "eval_runtime": 40.026,
+      "eval_samples_per_second": 43.871,
+      "eval_steps_per_second": 43.871,
+      "step": 2800
+    },
+    {
+      "epoch": 0.7216560106349307,
+      "grad_norm": 1.4088993072509766,
+      "learning_rate": 4.220127555805665e-05,
+      "loss": 2.5675,
+      "step": 2850
+    },
+    {
+      "epoch": 0.7343166424004558,
+      "grad_norm": 1.3225140571594238,
+      "learning_rate": 4.196679797411368e-05,
+      "loss": 2.56,
+      "step": 2900
+    },
+    {
+      "epoch": 0.7469772741659809,
+      "grad_norm": 1.3416539430618286,
+      "learning_rate": 4.1732320390170706e-05,
+      "loss": 2.6517,
+      "step": 2950
+    },
+    {
+      "epoch": 0.7596379059315059,
+      "grad_norm": 1.079567790031433,
+      "learning_rate": 4.149784280622773e-05,
+      "loss": 2.698,
+      "step": 3000
+    },
+    {
+      "epoch": 0.7596379059315059,
+      "eval_loss": 2.4842560291290283,
+      "eval_runtime": 39.7988,
+      "eval_samples_per_second": 44.122,
+      "eval_steps_per_second": 44.122,
+      "step": 3000
+    },
+    {
+      "epoch": 0.772298537697031,
+      "grad_norm": 1.4532116651535034,
+      "learning_rate": 4.126336522228475e-05,
+      "loss": 2.6232,
+      "step": 3050
+    },
+    {
+      "epoch": 0.7849591694625562,
+      "grad_norm": 1.5380038022994995,
+      "learning_rate": 4.102888763834178e-05,
+      "loss": 2.6212,
+      "step": 3100
+    },
+    {
+      "epoch": 0.7976198012280813,
+      "grad_norm": 1.3965916633605957,
+      "learning_rate": 4.0794410054398805e-05,
+      "loss": 2.5804,
+      "step": 3150
+    },
+    {
+      "epoch": 0.8102804329936064,
+      "grad_norm": 1.4798463582992554,
+      "learning_rate": 4.0559932470455826e-05,
+      "loss": 2.6724,
+      "step": 3200
+    },
+    {
+      "epoch": 0.8102804329936064,
+      "eval_loss": 2.480894088745117,
+      "eval_runtime": 39.9604,
+      "eval_samples_per_second": 43.943,
+      "eval_steps_per_second": 43.943,
+      "step": 3200
+    },
+    {
+      "epoch": 0.8229410647591315,
+      "grad_norm": 1.2598360776901245,
+      "learning_rate": 4.0325454886512854e-05,
+      "loss": 2.6993,
+      "step": 3250
+    },
+    {
+      "epoch": 0.8356016965246565,
+      "grad_norm": 1.366295576095581,
+      "learning_rate": 4.0090977302569876e-05,
+      "loss": 2.551,
+      "step": 3300
+    },
+    {
+      "epoch": 0.8482623282901817,
+      "grad_norm": 1.1827855110168457,
+      "learning_rate": 3.98564997186269e-05,
+      "loss": 2.6131,
+      "step": 3350
+    },
+    {
+      "epoch": 0.8609229600557068,
+      "grad_norm": 1.2728627920150757,
+      "learning_rate": 3.9622022134683925e-05,
+      "loss": 2.6178,
+      "step": 3400
+    },
+    {
+      "epoch": 0.8609229600557068,
+      "eval_loss": 2.477010726928711,
+      "eval_runtime": 40.2504,
+      "eval_samples_per_second": 43.627,
+      "eval_steps_per_second": 43.627,
+      "step": 3400
+    },
+    {
+      "epoch": 0.8735835918212319,
+      "grad_norm": 1.341917634010315,
+      "learning_rate": 3.938754455074095e-05,
+      "loss": 2.5748,
+      "step": 3450
+    },
+    {
+      "epoch": 0.886244223586757,
+      "grad_norm": 1.4114609956741333,
+      "learning_rate": 3.9153066966797975e-05,
+      "loss": 2.667,
+      "step": 3500
+    },
+    {
+      "epoch": 0.898904855352282,
+      "grad_norm": 1.1211490631103516,
+      "learning_rate": 3.8918589382855e-05,
+      "loss": 2.5671,
+      "step": 3550
+    },
+    {
+      "epoch": 0.9115654871178072,
+      "grad_norm": 1.4166322946548462,
+      "learning_rate": 3.8684111798912024e-05,
+      "loss": 2.5945,
+      "step": 3600
+    },
+    {
+      "epoch": 0.9115654871178072,
+      "eval_loss": 2.47322940826416,
+      "eval_runtime": 40.2079,
+      "eval_samples_per_second": 43.673,
+      "eval_steps_per_second": 43.673,
+      "step": 3600
+    },
+    {
+      "epoch": 0.9242261188833323,
+      "grad_norm": 0.9144394993782043,
+      "learning_rate": 3.844963421496905e-05,
+      "loss": 2.6148,
+      "step": 3650
+    },
+    {
+      "epoch": 0.9368867506488574,
+      "grad_norm": 1.4106061458587646,
+      "learning_rate": 3.821515663102608e-05,
+      "loss": 2.6586,
+      "step": 3700
+    },
+    {
+      "epoch": 0.9495473824143825,
+      "grad_norm": 1.414415717124939,
+      "learning_rate": 3.79806790470831e-05,
+      "loss": 2.5874,
+      "step": 3750
+    },
+    {
+      "epoch": 0.9622080141799075,
+      "grad_norm": 1.5448992252349854,
+      "learning_rate": 3.774620146314012e-05,
+      "loss": 2.6422,
+      "step": 3800
+    },
+    {
+      "epoch": 0.9622080141799075,
+      "eval_loss": 2.4701173305511475,
+      "eval_runtime": 40.1267,
+      "eval_samples_per_second": 43.761,
+      "eval_steps_per_second": 43.761,
+      "step": 3800
+    },
+    {
+      "epoch": 0.9748686459454327,
+      "grad_norm": 1.1959314346313477,
+      "learning_rate": 3.751172387919715e-05,
+      "loss": 2.6975,
+      "step": 3850
+    },
+    {
+      "epoch": 0.9875292777109578,
+      "grad_norm": 0.9525274038314819,
+      "learning_rate": 3.727724629525417e-05,
+      "loss": 2.6675,
+      "step": 3900
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 4.733253479003906,
+      "learning_rate": 3.70427687113112e-05,
+      "loss": 2.566,
+      "step": 3950
+    },
+    {
+      "epoch": 1.0126606317655251,
+      "grad_norm": 1.2803192138671875,
+      "learning_rate": 3.680829112736823e-05,
+      "loss": 2.5659,
+      "step": 4000
+    },
+    {
+      "epoch": 1.0126606317655251,
+      "eval_loss": 2.4702188968658447,
+      "eval_runtime": 40.1387,
+      "eval_samples_per_second": 43.748,
+      "eval_steps_per_second": 43.748,
+      "step": 4000
+    },
+    {
+      "epoch": 1.0253212635310502,
+      "grad_norm": 1.446990966796875,
+      "learning_rate": 3.657381354342525e-05,
+      "loss": 2.627,
+      "step": 4050
+    },
+    {
+      "epoch": 1.0379818952965754,
+      "grad_norm": 1.3563008308410645,
+      "learning_rate": 3.633933595948227e-05,
+      "loss": 2.6252,
+      "step": 4100
+    },
+    {
+      "epoch": 1.0506425270621005,
+      "grad_norm": 1.5763463973999023,
+      "learning_rate": 3.61048583755393e-05,
+      "loss": 2.6593,
+      "step": 4150
+    },
+    {
+      "epoch": 1.0633031588276256,
+      "grad_norm": 1.0055335760116577,
+      "learning_rate": 3.587038079159633e-05,
+      "loss": 2.5955,
+      "step": 4200
+    },
+    {
+      "epoch": 1.0633031588276256,
+      "eval_loss": 2.4676930904388428,
+      "eval_runtime": 40.0342,
+      "eval_samples_per_second": 43.863,
+      "eval_steps_per_second": 43.863,
+      "step": 4200
+    },
+    {
+      "epoch": 1.0759637905931505,
+      "grad_norm": 1.7013343572616577,
+      "learning_rate": 3.563590320765335e-05,
+      "loss": 2.59,
+      "step": 4250
+    },
+    {
+      "epoch": 1.0886244223586756,
+      "grad_norm": 1.541069507598877,
+      "learning_rate": 3.540142562371038e-05,
+      "loss": 2.6192,
+      "step": 4300
+    },
+    {
+      "epoch": 1.1012850541242007,
+      "grad_norm": 1.2536805868148804,
+      "learning_rate": 3.51669480397674e-05,
+      "loss": 2.6225,
+      "step": 4350
+    },
+    {
+      "epoch": 1.1139456858897259,
+      "grad_norm": 1.8328826427459717,
+      "learning_rate": 3.493247045582442e-05,
+      "loss": 2.6022,
+      "step": 4400
+    },
+    {
+      "epoch": 1.1139456858897259,
+      "eval_loss": 2.465629816055298,
+      "eval_runtime": 39.8532,
+      "eval_samples_per_second": 44.062,
+      "eval_steps_per_second": 44.062,
+      "step": 4400
+    },
+    {
+      "epoch": 1.126606317655251,
+      "grad_norm": 1.8557270765304565,
+      "learning_rate": 3.469799287188145e-05,
+      "loss": 2.6496,
+      "step": 4450
+    },
+    {
+      "epoch": 1.139266949420776,
+      "grad_norm": 1.3255618810653687,
+      "learning_rate": 3.446820483961734e-05,
+      "loss": 2.5315,
+      "step": 4500
+    },
+    {
+      "epoch": 1.1519275811863012,
+      "grad_norm": 1.2192399501800537,
+      "learning_rate": 3.423372725567436e-05,
+      "loss": 2.5409,
+      "step": 4550
+    },
+    {
+      "epoch": 1.1645882129518264,
+      "grad_norm": 1.2533234357833862,
+      "learning_rate": 3.399924967173139e-05,
+      "loss": 2.6457,
+      "step": 4600
+    },
+    {
+      "epoch": 1.1645882129518264,
+      "eval_loss": 2.462162733078003,
+      "eval_runtime": 40.0542,
+      "eval_samples_per_second": 43.841,
+      "eval_steps_per_second": 43.841,
+      "step": 4600
+    },
+    {
+      "epoch": 1.1772488447173515,
+      "grad_norm": 1.8414678573608398,
+      "learning_rate": 3.376477208778841e-05,
+      "loss": 2.5658,
+      "step": 4650
+    },
+    {
+      "epoch": 1.1899094764828764,
+      "grad_norm": 1.568259596824646,
+      "learning_rate": 3.3530294503845436e-05,
+      "loss": 2.5771,
+      "step": 4700
+    },
+    {
+      "epoch": 1.2025701082484015,
+      "grad_norm": 1.3547483682632446,
+      "learning_rate": 3.3295816919902464e-05,
+      "loss": 2.6525,
+      "step": 4750
+    },
+    {
+      "epoch": 1.2152307400139266,
+      "grad_norm": 1.1655386686325073,
+      "learning_rate": 3.3061339335959486e-05,
+      "loss": 2.6421,
+      "step": 4800
+    },
+    {
+      "epoch": 1.2152307400139266,
+      "eval_loss": 2.461489200592041,
+      "eval_runtime": 39.9962,
+      "eval_samples_per_second": 43.904,
+      "eval_steps_per_second": 43.904,
+      "step": 4800
+    },
+    {
+      "epoch": 1.2278913717794517,
+      "grad_norm": 1.798033595085144,
+      "learning_rate": 3.282686175201651e-05,
+      "loss": 2.6091,
+      "step": 4850
+    },
+    {
+      "epoch": 1.2405520035449769,
+      "grad_norm": 3.2964117527008057,
+      "learning_rate": 3.2592384168073535e-05,
+      "loss": 2.5997,
+      "step": 4900
+    },
+    {
+      "epoch": 1.253212635310502,
+      "grad_norm": 1.0457675457000732,
+      "learning_rate": 3.2357906584130557e-05,
+      "loss": 2.6144,
+      "step": 4950
+    },
+    {
+      "epoch": 1.265873267076027,
+      "grad_norm": 0.9728056192398071,
+      "learning_rate": 3.2123429000187585e-05,
+      "loss": 2.5712,
+      "step": 5000
+    },
+    {
+      "epoch": 1.265873267076027,
+      "eval_loss": 2.460186719894409,
+      "eval_runtime": 39.8386,
+      "eval_samples_per_second": 44.078,
+      "eval_steps_per_second": 44.078,
+      "step": 5000
+    },
+    {
+      "epoch": 1.2785338988415522,
+      "grad_norm": 1.2350194454193115,
+      "learning_rate": 3.188895141624461e-05,
+      "loss": 2.5448,
+      "step": 5050
+    },
+    {
+      "epoch": 1.2911945306070773,
+      "grad_norm": 1.4210622310638428,
+      "learning_rate": 3.1654473832301634e-05,
+      "loss": 2.6031,
+      "step": 5100
+    },
+    {
+      "epoch": 1.3038551623726025,
+      "grad_norm": 2.226473093032837,
+      "learning_rate": 3.1419996248358656e-05,
+      "loss": 2.6597,
+      "step": 5150
+    },
+    {
+      "epoch": 1.3165157941381276,
+      "grad_norm": 2.4525105953216553,
+      "learning_rate": 3.1185518664415684e-05,
+      "loss": 2.596,
+      "step": 5200
+    },
+    {
+      "epoch": 1.3165157941381276,
+      "eval_loss": 2.454537868499756,
+      "eval_runtime": 39.805,
+      "eval_samples_per_second": 44.115,
+      "eval_steps_per_second": 44.115,
+      "step": 5200
+    },
+    {
+      "epoch": 1.3291764259036527,
+      "grad_norm": 1.265309453010559,
+      "learning_rate": 3.095104108047271e-05,
+      "loss": 2.5559,
+      "step": 5250
+    },
+    {
+      "epoch": 1.3418370576691778,
+      "grad_norm": 2.1364307403564453,
+      "learning_rate": 3.071656349652973e-05,
+      "loss": 2.5859,
+      "step": 5300
+    },
+    {
+      "epoch": 1.3544976894347027,
+      "grad_norm": 1.5945920944213867,
+      "learning_rate": 3.048208591258676e-05,
+      "loss": 2.5778,
+      "step": 5350
+    },
+    {
+      "epoch": 1.3671583212002278,
+      "grad_norm": 1.2479759454727173,
+      "learning_rate": 3.0247608328643783e-05,
+      "loss": 2.6846,
+      "step": 5400
+    },
+    {
+      "epoch": 1.3671583212002278,
+      "eval_loss": 2.4547293186187744,
+      "eval_runtime": 39.7806,
+      "eval_samples_per_second": 44.142,
+      "eval_steps_per_second": 44.142,
+      "step": 5400
+    },
+    {
+      "epoch": 1.379818952965753,
+      "grad_norm": 1.4845050573349,
+      "learning_rate": 3.0013130744700808e-05,
+      "loss": 2.5661,
+      "step": 5450
+    },
+    {
+      "epoch": 1.392479584731278,
+      "grad_norm": 1.5581985712051392,
+      "learning_rate": 2.9778653160757836e-05,
+      "loss": 2.5441,
+      "step": 5500
+    },
+    {
+      "epoch": 1.4051402164968032,
+      "grad_norm": 3.1663737297058105,
+      "learning_rate": 2.9544175576814857e-05,
+      "loss": 2.5044,
+      "step": 5550
+    },
+    {
+      "epoch": 1.4178008482623283,
+      "grad_norm": 1.2454484701156616,
+      "learning_rate": 2.9309697992871882e-05,
+      "loss": 2.5747,
+      "step": 5600
+    },
+    {
+      "epoch": 1.4178008482623283,
+      "eval_loss": 2.4544529914855957,
+      "eval_runtime": 39.9287,
+      "eval_samples_per_second": 43.978,
+      "eval_steps_per_second": 43.978,
+      "step": 5600
+    },
+    {
+      "epoch": 1.4304614800278534,
+      "grad_norm": 1.662784457206726,
+      "learning_rate": 2.907522040892891e-05,
+      "loss": 2.6064,
+      "step": 5650
+    },
+    {
+      "epoch": 1.4431221117933786,
+      "grad_norm": 1.618458867073059,
+      "learning_rate": 2.8840742824985935e-05,
+      "loss": 2.5191,
+      "step": 5700
+    },
+    {
+      "epoch": 1.4557827435589035,
+      "grad_norm": 1.3003348112106323,
+      "learning_rate": 2.8606265241042956e-05,
+      "loss": 2.5339,
+      "step": 5750
+    },
+    {
+      "epoch": 1.4684433753244286,
+      "grad_norm": 1.1443992853164673,
+      "learning_rate": 2.8371787657099984e-05,
+      "loss": 2.5914,
+      "step": 5800
+    },
+    {
+      "epoch": 1.4684433753244286,
+      "eval_loss": 2.453752279281616,
+      "eval_runtime": 39.8234,
+      "eval_samples_per_second": 44.095,
+      "eval_steps_per_second": 44.095,
+      "step": 5800
+    },
+    {
+      "epoch": 1.4811040070899537,
+      "grad_norm": 1.2574009895324707,
+      "learning_rate": 2.813731007315701e-05,
+      "loss": 2.6109,
+      "step": 5850
+    },
+    {
+      "epoch": 1.4937646388554788,
+      "grad_norm": 1.002815842628479,
+      "learning_rate": 2.790283248921403e-05,
+      "loss": 2.6075,
+      "step": 5900
+    },
+    {
+      "epoch": 1.506425270621004,
+      "grad_norm": 1.306024432182312,
+      "learning_rate": 2.766835490527106e-05,
+      "loss": 2.5733,
+      "step": 5950
+    },
+    {
+      "epoch": 1.519085902386529,
+      "grad_norm": 2.5023701190948486,
+      "learning_rate": 2.7433877321328083e-05,
+      "loss": 2.6274,
+      "step": 6000
+    },
+    {
+      "epoch": 1.519085902386529,
+      "eval_loss": 2.449617862701416,
+      "eval_runtime": 39.9401,
+      "eval_samples_per_second": 43.966,
+      "eval_steps_per_second": 43.966,
+      "step": 6000
+    },
+    {
+      "epoch": 1.5317465341520542,
+      "grad_norm": 1.9410326480865479,
+      "learning_rate": 2.7199399737385105e-05,
+      "loss": 2.5312,
+      "step": 6050
+    },
+    {
+      "epoch": 1.5444071659175793,
+      "grad_norm": 1.9793561697006226,
+      "learning_rate": 2.6964922153442136e-05,
+      "loss": 2.5759,
+      "step": 6100
+    },
+    {
+      "epoch": 1.5570677976831044,
+      "grad_norm": 1.290531873703003,
+      "learning_rate": 2.6730444569499157e-05,
+      "loss": 2.5817,
+      "step": 6150
+    },
+    {
+      "epoch": 1.5697284294486296,
+      "grad_norm": 2.11389422416687,
+      "learning_rate": 2.6495966985556182e-05,
+      "loss": 2.6287,
+      "step": 6200
+    },
+    {
+      "epoch": 1.5697284294486296,
+      "eval_loss": 2.4490554332733154,
+      "eval_runtime": 39.7474,
+      "eval_samples_per_second": 44.179,
+      "eval_steps_per_second": 44.179,
+      "step": 6200
+    },
+    {
+      "epoch": 1.5823890612141547,
+      "grad_norm": 1.6492938995361328,
+      "learning_rate": 2.626148940161321e-05,
+      "loss": 2.631,
+      "step": 6250
+    },
+    {
+      "epoch": 1.5950496929796798,
+      "grad_norm": 1.3233673572540283,
+      "learning_rate": 2.6027011817670232e-05,
+      "loss": 2.5654,
+      "step": 6300
+    },
+    {
+      "epoch": 1.607710324745205,
+      "grad_norm": 1.688264012336731,
+      "learning_rate": 2.5792534233727257e-05,
+      "loss": 2.6096,
+      "step": 6350
+    },
+    {
+      "epoch": 1.62037095651073,
+      "grad_norm": 2.064823865890503,
+      "learning_rate": 2.5558056649784285e-05,
+      "loss": 2.6275,
+      "step": 6400
+    },
+    {
+      "epoch": 1.62037095651073,
+      "eval_loss": 2.4488983154296875,
+      "eval_runtime": 39.6847,
+      "eval_samples_per_second": 44.249,
+      "eval_steps_per_second": 44.249,
+      "step": 6400
+    },
+    {
+      "epoch": 1.633031588276255,
+      "grad_norm": 1.5599696636199951,
+      "learning_rate": 2.5323579065841306e-05,
+      "loss": 2.6334,
+      "step": 6450
+    },
+    {
+      "epoch": 1.64569222004178,
+      "grad_norm": 1.3142633438110352,
+      "learning_rate": 2.508910148189833e-05,
+      "loss": 2.5496,
+      "step": 6500
+    },
+    {
+      "epoch": 1.6583528518073052,
+      "grad_norm": 1.474135160446167,
+      "learning_rate": 2.4854623897955356e-05,
+      "loss": 2.5628,
+      "step": 6550
+    },
+    {
+      "epoch": 1.6710134835728303,
+      "grad_norm": 1.3737610578536987,
+      "learning_rate": 2.4620146314012384e-05,
+      "loss": 2.5345,
+      "step": 6600
+    },
+    {
+      "epoch": 1.6710134835728303,
+      "eval_loss": 2.447559356689453,
+      "eval_runtime": 39.7021,
+      "eval_samples_per_second": 44.229,
+      "eval_steps_per_second": 44.229,
+      "step": 6600
+    },
+    {
+      "epoch": 1.6836741153383554,
+      "grad_norm": 1.2432060241699219,
+      "learning_rate": 2.4385668730069405e-05,
+      "loss": 2.5977,
+      "step": 6650
+    },
+    {
+      "epoch": 1.6963347471038803,
+      "grad_norm": 1.465063452720642,
+      "learning_rate": 2.415119114612643e-05,
+      "loss": 2.6118,
+      "step": 6700
+    },
+    {
+      "epoch": 1.7089953788694054,
+      "grad_norm": 1.5186200141906738,
+      "learning_rate": 2.3916713562183458e-05,
+      "loss": 2.6126,
+      "step": 6750
+    },
+    {
+      "epoch": 1.7216560106349306,
+      "grad_norm": 1.6869078874588013,
+      "learning_rate": 2.368223597824048e-05,
+      "loss": 2.576,
+      "step": 6800
+    },
+    {
+      "epoch": 1.7216560106349306,
+      "eval_loss": 2.4459502696990967,
+      "eval_runtime": 39.7653,
+      "eval_samples_per_second": 44.159,
+      "eval_steps_per_second": 44.159,
+      "step": 6800
+    },
+    {
+      "epoch": 1.7343166424004557,
+      "grad_norm": 1.2578104734420776,
+      "learning_rate": 2.3447758394297507e-05,
+      "loss": 2.6178,
+      "step": 6850
+    },
+    {
+      "epoch": 1.7469772741659808,
+      "grad_norm": 1.7597213983535767,
+      "learning_rate": 2.3213280810354532e-05,
+      "loss": 2.6358,
+      "step": 6900
+    },
+    {
+      "epoch": 1.759637905931506,
+      "grad_norm": 2.144465923309326,
+      "learning_rate": 2.2978803226411554e-05,
+      "loss": 2.5597,
+      "step": 6950
+    },
+    {
+      "epoch": 1.772298537697031,
+      "grad_norm": 1.1808464527130127,
+      "learning_rate": 2.2744325642468582e-05,
+      "loss": 2.6269,
+      "step": 7000
+    },
+    {
+      "epoch": 1.772298537697031,
+      "eval_loss": 2.4444611072540283,
+      "eval_runtime": 40.0709,
+      "eval_samples_per_second": 43.822,
+      "eval_steps_per_second": 43.822,
+      "step": 7000
+    },
+    {
+      "epoch": 1.7849591694625562,
+      "grad_norm": 1.4550806283950806,
+      "learning_rate": 2.2509848058525606e-05,
+      "loss": 2.6206,
+      "step": 7050
+    },
+    {
+      "epoch": 1.7976198012280813,
+      "grad_norm": 1.2635902166366577,
+      "learning_rate": 2.227537047458263e-05,
+      "loss": 2.5722,
+      "step": 7100
+    },
+    {
+      "epoch": 1.8102804329936064,
+      "grad_norm": 1.3835856914520264,
+      "learning_rate": 2.2040892890639656e-05,
+      "loss": 2.535,
+      "step": 7150
+    },
+    {
+      "epoch": 1.8229410647591315,
+      "grad_norm": 1.735004186630249,
+      "learning_rate": 2.180641530669668e-05,
+      "loss": 2.6086,
+      "step": 7200
+    },
+    {
+      "epoch": 1.8229410647591315,
+      "eval_loss": 2.443899154663086,
+      "eval_runtime": 40.91,
+      "eval_samples_per_second": 42.924,
+      "eval_steps_per_second": 42.924,
+      "step": 7200
+    },
+    {
+      "epoch": 1.8356016965246567,
+      "grad_norm": 1.263051986694336,
+      "learning_rate": 2.1571937722753706e-05,
+      "loss": 2.5544,
+      "step": 7250
+    },
+    {
+      "epoch": 1.8482623282901818,
+      "grad_norm": 1.0899442434310913,
+      "learning_rate": 2.133746013881073e-05,
+      "loss": 2.5603,
+      "step": 7300
+    },
+    {
+      "epoch": 1.860922960055707,
+      "grad_norm": 3.038811206817627,
+      "learning_rate": 2.1102982554867755e-05,
+      "loss": 2.5688,
+      "step": 7350
+    },
+    {
+      "epoch": 1.873583591821232,
+      "grad_norm": 1.6385984420776367,
+      "learning_rate": 2.086850497092478e-05,
+      "loss": 2.6006,
+      "step": 7400
+    },
+    {
+      "epoch": 1.873583591821232,
+      "eval_loss": 2.443300724029541,
+      "eval_runtime": 40.6649,
+      "eval_samples_per_second": 43.182,
+      "eval_steps_per_second": 43.182,
+      "step": 7400
+    },
+    {
+      "epoch": 1.8862442235867571,
+      "grad_norm": 1.2857129573822021,
+      "learning_rate": 2.0634027386981805e-05,
+      "loss": 2.5563,
+      "step": 7450
+    },
+    {
+      "epoch": 1.898904855352282,
+      "grad_norm": 1.0289497375488281,
+      "learning_rate": 2.0399549803038833e-05,
+      "loss": 2.5671,
+      "step": 7500
+    },
+    {
+      "epoch": 1.9115654871178072,
+      "grad_norm": 1.5041025876998901,
+      "learning_rate": 2.0165072219095854e-05,
+      "loss": 2.5689,
+      "step": 7550
+    },
+    {
+      "epoch": 1.9242261188833323,
+      "grad_norm": 1.6611964702606201,
+      "learning_rate": 1.993528418683174e-05,
+      "loss": 2.5801,
+      "step": 7600
+    },
+    {
+      "epoch": 1.9242261188833323,
+      "eval_loss": 2.443532943725586,
+      "eval_runtime": 39.931,
+      "eval_samples_per_second": 43.976,
+      "eval_steps_per_second": 43.976,
+      "step": 7600
+    },
+    {
+      "epoch": 1.9368867506488574,
+      "grad_norm": 1.521170735359192,
+      "learning_rate": 1.9700806602888767e-05,
+      "loss": 2.5969,
+      "step": 7650
+    },
+    {
+      "epoch": 1.9495473824143825,
+      "grad_norm": 1.3700034618377686,
+      "learning_rate": 1.946632901894579e-05,
+      "loss": 2.6306,
+      "step": 7700
+    },
+    {
+      "epoch": 1.9622080141799074,
+      "grad_norm": 2.311443328857422,
+      "learning_rate": 1.9231851435002814e-05,
+      "loss": 2.5608,
+      "step": 7750
+    },
+    {
+      "epoch": 1.9748686459454325,
+      "grad_norm": 1.6699820756912231,
+      "learning_rate": 1.8997373851059842e-05,
+      "loss": 2.5113,
+      "step": 7800
+    },
+    {
+      "epoch": 1.9748686459454325,
+      "eval_loss": 2.4421675205230713,
+      "eval_runtime": 40.1783,
+      "eval_samples_per_second": 43.705,
+      "eval_steps_per_second": 43.705,
+      "step": 7800
+    },
+    {
+      "epoch": 1.9875292777109577,
+      "grad_norm": 1.2560683488845825,
+      "learning_rate": 1.8762896267116863e-05,
+      "loss": 2.545,
+      "step": 7850
+    },
+    {
+      "epoch": 2.0,
+      "grad_norm": 2.176563262939453,
+      "learning_rate": 1.852841868317389e-05,
+      "loss": 2.5752,
+      "step": 7900
+    },
+    {
+      "epoch": 2.012660631765525,
+      "grad_norm": 1.2551178932189941,
+      "learning_rate": 1.8293941099230916e-05,
+      "loss": 2.5215,
+      "step": 7950
+    },
+    {
+      "epoch": 2.0253212635310502,
+      "grad_norm": 1.5646872520446777,
+      "learning_rate": 1.8059463515287937e-05,
+      "loss": 2.5838,
+      "step": 8000
+    },
+    {
+      "epoch": 2.0253212635310502,
+      "eval_loss": 2.441195249557495,
+      "eval_runtime": 39.701,
+      "eval_samples_per_second": 44.231,
+      "eval_steps_per_second": 44.231,
+      "step": 8000
+    },
+    {
+      "epoch": 2.0379818952965754,
+      "grad_norm": 1.4227900505065918,
+      "learning_rate": 1.7824985931344966e-05,
+      "loss": 2.5597,
+      "step": 8050
+    },
+    {
+      "epoch": 2.0506425270621005,
+      "grad_norm": 1.3013832569122314,
+      "learning_rate": 1.759050834740199e-05,
+      "loss": 2.7641,
+      "step": 8100
+    },
+    {
+      "epoch": 2.0633031588276256,
+      "grad_norm": 1.1282143592834473,
+      "learning_rate": 1.7356030763459015e-05,
+      "loss": 2.5875,
+      "step": 8150
+    },
+    {
+      "epoch": 2.0759637905931507,
+      "grad_norm": 2.079760789871216,
+      "learning_rate": 1.712155317951604e-05,
+      "loss": 2.4861,
+      "step": 8200
+    },
+    {
+      "epoch": 2.0759637905931507,
+      "eval_loss": 2.440812826156616,
+      "eval_runtime": 40.0764,
+      "eval_samples_per_second": 43.816,
+      "eval_steps_per_second": 43.816,
+      "step": 8200
+    },
+    {
+      "epoch": 2.088624422358676,
+      "grad_norm": 1.0884991884231567,
+      "learning_rate": 1.6887075595573065e-05,
+      "loss": 2.5941,
+      "step": 8250
+    },
+    {
+      "epoch": 2.101285054124201,
+      "grad_norm": 1.9202015399932861,
+      "learning_rate": 1.665259801163009e-05,
+      "loss": 2.5929,
+      "step": 8300
+    },
+    {
+      "epoch": 2.113945685889726,
+      "grad_norm": 1.5925830602645874,
+      "learning_rate": 1.6418120427687114e-05,
+      "loss": 2.5046,
+      "step": 8350
+    },
+    {
+      "epoch": 2.126606317655251,
+      "grad_norm": 1.5219184160232544,
+      "learning_rate": 1.618364284374414e-05,
+      "loss": 2.5628,
+      "step": 8400
+    },
+    {
+      "epoch": 2.126606317655251,
+      "eval_loss": 2.4396440982818604,
+      "eval_runtime": 40.0053,
+      "eval_samples_per_second": 43.894,
+      "eval_steps_per_second": 43.894,
+      "step": 8400
+    },
+    {
+      "epoch": 2.139266949420776,
+      "grad_norm": 1.4882445335388184,
+      "learning_rate": 1.5949165259801164e-05,
+      "loss": 2.6268,
+      "step": 8450
+    },
+    {
+      "epoch": 2.151927581186301,
+      "grad_norm": 1.3513301610946655,
+      "learning_rate": 1.571468767585819e-05,
+      "loss": 2.5277,
+      "step": 8500
+    },
+    {
+      "epoch": 2.164588212951826,
+      "grad_norm": 1.690974473953247,
+      "learning_rate": 1.5480210091915216e-05,
+      "loss": 2.5631,
+      "step": 8550
+    },
+    {
+      "epoch": 2.1772488447173513,
+      "grad_norm": 1.5311528444290161,
+      "learning_rate": 1.5245732507972238e-05,
+      "loss": 2.5454,
+      "step": 8600
+    },
+    {
+      "epoch": 2.1772488447173513,
+      "eval_loss": 2.4388718605041504,
+      "eval_runtime": 40.0289,
+      "eval_samples_per_second": 43.868,
+      "eval_steps_per_second": 43.868,
+      "step": 8600
+    },
+    {
+      "epoch": 2.1899094764828764,
+      "grad_norm": 2.1171281337738037,
+      "learning_rate": 1.5011254924029264e-05,
+      "loss": 2.6112,
+      "step": 8650
+    },
+    {
+      "epoch": 2.2025701082484015,
+      "grad_norm": 1.9706814289093018,
+      "learning_rate": 1.4776777340086289e-05,
+      "loss": 2.588,
+      "step": 8700
+    },
+    {
+      "epoch": 2.2152307400139266,
+      "grad_norm": 1.8991297483444214,
+      "learning_rate": 1.4542299756143312e-05,
+      "loss": 2.5655,
+      "step": 8750
+    },
+    {
+      "epoch": 2.2278913717794517,
+      "grad_norm": 1.5568820238113403,
+      "learning_rate": 1.4307822172200339e-05,
+      "loss": 2.5312,
+      "step": 8800
+    },
+    {
+      "epoch": 2.2278913717794517,
+      "eval_loss": 2.438715696334839,
+      "eval_runtime": 40.1748,
+      "eval_samples_per_second": 43.709,
+      "eval_steps_per_second": 43.709,
+      "step": 8800
+    },
+    {
+      "epoch": 2.240552003544977,
+      "grad_norm": 1.277051329612732,
+      "learning_rate": 1.4073344588257365e-05,
+      "loss": 2.5818,
+      "step": 8850
+    },
+    {
+      "epoch": 2.253212635310502,
+      "grad_norm": 1.8890128135681152,
+      "learning_rate": 1.3838867004314388e-05,
+      "loss": 2.5211,
+      "step": 8900
+    },
+    {
+      "epoch": 2.265873267076027,
+      "grad_norm": 1.8824830055236816,
+      "learning_rate": 1.3604389420371413e-05,
+      "loss": 2.53,
+      "step": 8950
+    },
+    {
+      "epoch": 2.278533898841552,
+      "grad_norm": 1.239490032196045,
+      "learning_rate": 1.336991183642844e-05,
+      "loss": 2.5889,
+      "step": 9000
+    },
+    {
+      "epoch": 2.278533898841552,
+      "eval_loss": 2.437577962875366,
+      "eval_runtime": 40.1654,
+      "eval_samples_per_second": 43.719,
+      "eval_steps_per_second": 43.719,
+      "step": 9000
+    },
+    {
+      "epoch": 2.2911945306070773,
+      "grad_norm": 1.7253328561782837,
+      "learning_rate": 1.3135434252485462e-05,
+      "loss": 2.5426,
+      "step": 9050
+    },
+    {
+      "epoch": 2.3038551623726025,
+      "grad_norm": 1.6971838474273682,
+      "learning_rate": 1.2900956668542489e-05,
+      "loss": 2.4953,
+      "step": 9100
+    },
+    {
+      "epoch": 2.3165157941381276,
+      "grad_norm": 1.4906270503997803,
+      "learning_rate": 1.2666479084599514e-05,
+      "loss": 2.606,
+      "step": 9150
+    },
+    {
+      "epoch": 2.3291764259036527,
+      "grad_norm": 1.658526062965393,
+      "learning_rate": 1.2432001500656538e-05,
+      "loss": 2.5483,
+      "step": 9200
+    },
+    {
+      "epoch": 2.3291764259036527,
+      "eval_loss": 2.437896490097046,
+      "eval_runtime": 40.7008,
+      "eval_samples_per_second": 43.144,
+      "eval_steps_per_second": 43.144,
+      "step": 9200
+    },
+    {
+      "epoch": 2.341837057669178,
+      "grad_norm": 1.0781177282333374,
+      "learning_rate": 1.2197523916713563e-05,
+      "loss": 2.5449,
+      "step": 9250
+    },
+    {
+      "epoch": 2.354497689434703,
+      "grad_norm": 2.1414873600006104,
+      "learning_rate": 1.1963046332770588e-05,
+      "loss": 2.5303,
+      "step": 9300
+    },
+    {
+      "epoch": 2.367158321200228,
+      "grad_norm": 2.063297986984253,
+      "learning_rate": 1.1728568748827613e-05,
+      "loss": 2.5837,
+      "step": 9350
+    },
+    {
+      "epoch": 2.3798189529657527,
+      "grad_norm": 1.2153489589691162,
+      "learning_rate": 1.1494091164884637e-05,
+      "loss": 2.6384,
+      "step": 9400
+    },
+    {
+      "epoch": 2.3798189529657527,
+      "eval_loss": 2.4365696907043457,
+      "eval_runtime": 40.2398,
+      "eval_samples_per_second": 43.638,
+      "eval_steps_per_second": 43.638,
+      "step": 9400
+    },
+    {
+      "epoch": 2.3924795847312783,
+      "grad_norm": 1.2976094484329224,
+      "learning_rate": 1.1259613580941662e-05,
+      "loss": 2.572,
+      "step": 9450
+    },
+    {
+      "epoch": 2.405140216496803,
+      "grad_norm": 1.2775920629501343,
+      "learning_rate": 1.1025135996998689e-05,
+      "loss": 2.6083,
+      "step": 9500
+    },
+    {
+      "epoch": 2.417800848262328,
+      "grad_norm": 1.358311653137207,
+      "learning_rate": 1.0790658413055713e-05,
+      "loss": 2.5206,
+      "step": 9550
+    },
+    {
+      "epoch": 2.4304614800278532,
+      "grad_norm": 1.3438369035720825,
+      "learning_rate": 1.0556180829112736e-05,
+      "loss": 2.4967,
+      "step": 9600
+    },
+    {
+      "epoch": 2.4304614800278532,
+      "eval_loss": 2.4359662532806396,
+      "eval_runtime": 40.0802,
+      "eval_samples_per_second": 43.812,
+      "eval_steps_per_second": 43.812,
+      "step": 9600
+    },
+    {
+      "epoch": 2.4431221117933783,
+      "grad_norm": 1.2618831396102905,
+      "learning_rate": 1.0321703245169763e-05,
+      "loss": 2.6169,
+      "step": 9650
+    },
+    {
+      "epoch": 2.4557827435589035,
+      "grad_norm": 1.3764727115631104,
+      "learning_rate": 1.0087225661226788e-05,
+      "loss": 2.5444,
+      "step": 9700
+    },
+    {
+      "epoch": 2.4684433753244286,
+      "grad_norm": 1.604864478111267,
+      "learning_rate": 9.852748077283812e-06,
+      "loss": 2.5343,
+      "step": 9750
+    },
+    {
+      "epoch": 2.4811040070899537,
+      "grad_norm": 1.390496850013733,
+      "learning_rate": 9.618270493340837e-06,
+      "loss": 2.5051,
+      "step": 9800
+    },
+    {
+      "epoch": 2.4811040070899537,
+      "eval_loss": 2.4353232383728027,
+      "eval_runtime": 40.1607,
+      "eval_samples_per_second": 43.724,
+      "eval_steps_per_second": 43.724,
+      "step": 9800
+    },
+    {
+      "epoch": 2.493764638855479,
+      "grad_norm": 2.1982169151306152,
+      "learning_rate": 9.383792909397862e-06,
+      "loss": 2.5036,
+      "step": 9850
+    },
+    {
+      "epoch": 2.506425270621004,
+      "grad_norm": 1.3033822774887085,
+      "learning_rate": 9.149315325454887e-06,
+      "loss": 2.5205,
+      "step": 9900
+    },
+    {
+      "epoch": 2.519085902386529,
+      "grad_norm": 1.682586431503296,
+      "learning_rate": 8.919527293190772e-06,
+      "loss": 2.6083,
+      "step": 9950
+    },
+    {
+      "epoch": 2.531746534152054,
+      "grad_norm": 3.184382200241089,
+      "learning_rate": 8.685049709247797e-06,
+      "loss": 2.5314,
+      "step": 10000
+    },
+    {
+      "epoch": 2.531746534152054,
+      "eval_loss": 2.434755802154541,
+      "eval_runtime": 40.2877,
+      "eval_samples_per_second": 43.587,
+      "eval_steps_per_second": 43.587,
+      "step": 10000
+    },
+    {
+      "epoch": 2.5444071659175793,
+      "grad_norm": 2.0026867389678955,
+      "learning_rate": 8.450572125304821e-06,
+      "loss": 2.5109,
+      "step": 10050
+    },
+    {
+      "epoch": 2.5570677976831044,
+      "grad_norm": 1.3833885192871094,
+      "learning_rate": 8.216094541361846e-06,
+      "loss": 2.5362,
+      "step": 10100
+    },
+    {
+      "epoch": 2.5697284294486296,
+      "grad_norm": 2.157984495162964,
+      "learning_rate": 7.981616957418871e-06,
+      "loss": 2.5423,
+      "step": 10150
+    },
+    {
+      "epoch": 2.5823890612141547,
+      "grad_norm": 1.682053565979004,
+      "learning_rate": 7.747139373475897e-06,
+      "loss": 2.5133,
+      "step": 10200
+    },
+    {
+      "epoch": 2.5823890612141547,
+      "eval_loss": 2.435208559036255,
+      "eval_runtime": 40.4768,
+      "eval_samples_per_second": 43.383,
+      "eval_steps_per_second": 43.383,
+      "step": 10200
+    },
+    {
+      "epoch": 2.59504969297968,
+      "grad_norm": 1.9720139503479004,
+      "learning_rate": 7.512661789532921e-06,
+      "loss": 2.6372,
+      "step": 10250
+    },
+    {
+      "epoch": 2.607710324745205,
+      "grad_norm": 1.6906607151031494,
+      "learning_rate": 7.278184205589945e-06,
+      "loss": 2.5505,
+      "step": 10300
+    },
+    {
+      "epoch": 2.62037095651073,
+      "grad_norm": 1.484045147895813,
+      "learning_rate": 7.043706621646972e-06,
+      "loss": 2.5095,
+      "step": 10350
+    },
+    {
+      "epoch": 2.633031588276255,
+      "grad_norm": 1.6676850318908691,
+      "learning_rate": 6.8092290377039955e-06,
+      "loss": 2.6487,
+      "step": 10400
+    },
+    {
+      "epoch": 2.633031588276255,
+      "eval_loss": 2.4344091415405273,
+      "eval_runtime": 39.869,
+      "eval_samples_per_second": 44.044,
+      "eval_steps_per_second": 44.044,
+      "step": 10400
+    },
+    {
+      "epoch": 2.64569222004178,
+      "grad_norm": 1.5012388229370117,
+      "learning_rate": 6.57475145376102e-06,
+      "loss": 2.5756,
+      "step": 10450
+    },
+    {
+      "epoch": 2.6583528518073054,
+      "grad_norm": 1.043954849243164,
+      "learning_rate": 6.340273869818046e-06,
+      "loss": 2.5843,
+      "step": 10500
+    },
+    {
+      "epoch": 2.67101348357283,
+      "grad_norm": 1.0455141067504883,
+      "learning_rate": 6.105796285875071e-06,
+      "loss": 2.5248,
+      "step": 10550
+    },
+    {
+      "epoch": 2.6836741153383556,
+      "grad_norm": 1.39467453956604,
+      "learning_rate": 5.871318701932095e-06,
+      "loss": 2.5091,
+      "step": 10600
+    },
+    {
+      "epoch": 2.6836741153383556,
+      "eval_loss": 2.4331610202789307,
+      "eval_runtime": 39.923,
+      "eval_samples_per_second": 43.985,
+      "eval_steps_per_second": 43.985,
+      "step": 10600
+    },
+    {
+      "epoch": 2.6963347471038803,
+      "grad_norm": 1.1417715549468994,
+      "learning_rate": 5.63684111798912e-06,
+      "loss": 2.5853,
+      "step": 10650
+    },
+    {
+      "epoch": 2.7089953788694054,
+      "grad_norm": 1.133244514465332,
+      "learning_rate": 5.402363534046146e-06,
+      "loss": 2.5457,
+      "step": 10700
+    },
+    {
+      "epoch": 2.7216560106349306,
+      "grad_norm": 1.2331452369689941,
+      "learning_rate": 5.1678859501031705e-06,
+      "loss": 2.5576,
+      "step": 10750
+    },
+    {
+      "epoch": 2.7343166424004557,
+      "grad_norm": 1.7164263725280762,
+      "learning_rate": 4.933408366160195e-06,
+      "loss": 2.5471,
+      "step": 10800
+    },
+    {
+      "epoch": 2.7343166424004557,
+      "eval_loss": 2.4340403079986572,
+      "eval_runtime": 40.3849,
+      "eval_samples_per_second": 43.482,
+      "eval_steps_per_second": 43.482,
+      "step": 10800
+    },
+    {
+      "epoch": 2.746977274165981,
+      "grad_norm": 1.3680106401443481,
+      "learning_rate": 4.69893078221722e-06,
+      "loss": 2.5562,
+      "step": 10850
+    },
+    {
+      "epoch": 2.759637905931506,
+      "grad_norm": 1.0978279113769531,
+      "learning_rate": 4.464453198274246e-06,
+      "loss": 2.5185,
+      "step": 10900
+    },
+    {
+      "epoch": 2.772298537697031,
+      "grad_norm": 1.2212647199630737,
+      "learning_rate": 4.2299756143312695e-06,
+      "loss": 2.6371,
+      "step": 10950
+    },
+    {
+      "epoch": 2.784959169462556,
+      "grad_norm": 1.6452165842056274,
+      "learning_rate": 3.995498030388295e-06,
+      "loss": 2.681,
+      "step": 11000
+    },
+    {
+      "epoch": 2.784959169462556,
+      "eval_loss": 2.4337143898010254,
+      "eval_runtime": 40.4235,
+      "eval_samples_per_second": 43.44,
+      "eval_steps_per_second": 43.44,
+      "step": 11000
+    },
+    {
+      "epoch": 2.7976198012280813,
+      "grad_norm": 1.7757978439331055,
+      "learning_rate": 3.7610204464453203e-06,
+      "loss": 2.5746,
+      "step": 11050
+    },
+    {
+      "epoch": 2.8102804329936064,
+      "grad_norm": 1.2373579740524292,
+      "learning_rate": 3.5265428625023455e-06,
+      "loss": 2.5412,
+      "step": 11100
+    },
+    {
+      "epoch": 2.8229410647591315,
+      "grad_norm": 1.1407558917999268,
+      "learning_rate": 3.29206527855937e-06,
+      "loss": 2.5973,
+      "step": 11150
+    },
+    {
+      "epoch": 2.8356016965246567,
+      "grad_norm": 2.399686813354492,
+      "learning_rate": 3.057587694616395e-06,
+      "loss": 2.5566,
+      "step": 11200
+    },
+    {
+      "epoch": 2.8356016965246567,
+      "eval_loss": 2.4338231086730957,
+      "eval_runtime": 40.4877,
+      "eval_samples_per_second": 43.371,
+      "eval_steps_per_second": 43.371,
+      "step": 11200
+    },
+    {
+      "epoch": 2.8482623282901818,
+      "grad_norm": 1.7053141593933105,
+      "learning_rate": 2.8231101106734197e-06,
+      "loss": 2.6224,
+      "step": 11250
+    },
+    {
+      "epoch": 2.860922960055707,
+      "grad_norm": 1.8215903043746948,
+      "learning_rate": 2.5886325267304445e-06,
+      "loss": 2.5108,
+      "step": 11300
+    },
+    {
+      "epoch": 2.873583591821232,
+      "grad_norm": 1.1648200750350952,
+      "learning_rate": 2.3541549427874697e-06,
+      "loss": 2.557,
+      "step": 11350
+    },
+    {
+      "epoch": 2.886244223586757,
+      "grad_norm": 1.5225868225097656,
+      "learning_rate": 2.1196773588444944e-06,
+      "loss": 2.6285,
+      "step": 11400
+    },
+    {
+      "epoch": 2.886244223586757,
+      "eval_loss": 2.4334514141082764,
+      "eval_runtime": 40.3985,
+      "eval_samples_per_second": 43.467,
+      "eval_steps_per_second": 43.467,
+      "step": 11400
+    },
+    {
+      "epoch": 2.8989048553522823,
+      "grad_norm": 1.4937622547149658,
+      "learning_rate": 1.8851997749015194e-06,
+      "loss": 2.481,
+      "step": 11450
+    },
+    {
+      "epoch": 2.911565487117807,
+      "grad_norm": 1.9169902801513672,
+      "learning_rate": 1.6507221909585446e-06,
+      "loss": 2.5412,
+      "step": 11500
+    },
+    {
+      "epoch": 2.9242261188833325,
+      "grad_norm": 1.6611114740371704,
+      "learning_rate": 1.4162446070155693e-06,
+      "loss": 2.5086,
+      "step": 11550
+    },
+    {
+      "epoch": 2.936886750648857,
+      "grad_norm": 1.3464007377624512,
+      "learning_rate": 1.1817670230725943e-06,
+      "loss": 2.6063,
+      "step": 11600
+    },
+    {
+      "epoch": 2.936886750648857,
+      "eval_loss": 2.4329476356506348,
+      "eval_runtime": 40.4334,
+      "eval_samples_per_second": 43.429,
+      "eval_steps_per_second": 43.429,
+      "step": 11600
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 11847,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6118984656617472.0,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-11600/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d46b7107c3dfd17ff0fb12b8cdcc256a49e8d4594358d4495904a5fcb069382
+size 5304

checkpoint-11800/README.md

@@ -0,0 +1,202 @@
+---
+base_model: gpt2
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.14.0

checkpoint-11800/adapter_config.json

@@ -0,0 +1,32 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "gpt2",
+  "bias": "none",
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_proj",
+    "c_attn"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-11800/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1cbeee3dfc268cbdbc22f727c37945ebd4275a3ea49512bd0ee049c583c3112b
+size 3253104

checkpoint-11800/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2508e6c71eefe4bf3e6d34287d87494c23ed52ffdc88e1f2b4e1870ad89b4ce2
+size 6548858

checkpoint-11800/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7627c0686ccb211fe33205e3cd82a7c42fc538760ecf8cec401a983447f69f8d
+size 14244

checkpoint-11800/scaler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:573e7fd5dc49ed59f25d4becc07e4193e7f82ea7e1f2e73baca290d86a26a80a
+size 988

checkpoint-11800/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:779c8896e02a376108997a6c5096158c8cea4a1d22071fa3e06e889ac960dfa8
+size 1064

checkpoint-11800/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-11800/tokenizer_config.json

@@ -0,0 +1,21 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-11800/trainer_state.json

@@ -0,0 +1,2158 @@
+{
+  "best_global_step": 11800,
+  "best_metric": 2.432849645614624,
+  "best_model_checkpoint": "/kaggle/working/senko_adaptive/checkpoint-11800",
+  "epoch": 2.9875292777109577,
+  "eval_steps": 200,
+  "global_step": 11800,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0126606317655251,
+      "grad_norm": 0.4274106025695801,
+      "learning_rate": 2.067510548523207e-06,
+      "loss": 3.4405,
+      "step": 50
+    },
+    {
+      "epoch": 0.0253212635310502,
+      "grad_norm": 0.5292551517486572,
+      "learning_rate": 4.177215189873418e-06,
+      "loss": 3.4567,
+      "step": 100
+    },
+    {
+      "epoch": 0.0379818952965753,
+      "grad_norm": 0.7541739344596863,
+      "learning_rate": 6.28691983122363e-06,
+      "loss": 3.4683,
+      "step": 150
+    },
+    {
+      "epoch": 0.0506425270621004,
+      "grad_norm": 0.8833445906639099,
+      "learning_rate": 8.39662447257384e-06,
+      "loss": 3.5084,
+      "step": 200
+    },
+    {
+      "epoch": 0.0506425270621004,
+      "eval_loss": 3.5248923301696777,
+      "eval_runtime": 39.9384,
+      "eval_samples_per_second": 43.968,
+      "eval_steps_per_second": 43.968,
+      "step": 200
+    },
+    {
+      "epoch": 0.0633031588276255,
+      "grad_norm": 0.9998921155929565,
+      "learning_rate": 1.0506329113924052e-05,
+      "loss": 3.359,
+      "step": 250
+    },
+    {
+      "epoch": 0.0759637905931506,
+      "grad_norm": 0.8041885495185852,
+      "learning_rate": 1.2616033755274262e-05,
+      "loss": 3.351,
+      "step": 300
+    },
+    {
+      "epoch": 0.0886244223586757,
+      "grad_norm": 0.9213416576385498,
+      "learning_rate": 1.4725738396624473e-05,
+      "loss": 3.2244,
+      "step": 350
+    },
+    {
+      "epoch": 0.1012850541242008,
+      "grad_norm": 1.0922213792800903,
+      "learning_rate": 1.6835443037974685e-05,
+      "loss": 3.1565,
+      "step": 400
+    },
+    {
+      "epoch": 0.1012850541242008,
+      "eval_loss": 3.151216983795166,
+      "eval_runtime": 39.7515,
+      "eval_samples_per_second": 44.174,
+      "eval_steps_per_second": 44.174,
+      "step": 400
+    },
+    {
+      "epoch": 0.1139456858897259,
+      "grad_norm": 1.4199283123016357,
+      "learning_rate": 1.8945147679324897e-05,
+      "loss": 3.0154,
+      "step": 450
+    },
+    {
+      "epoch": 0.126606317655251,
+      "grad_norm": 1.077143907546997,
+      "learning_rate": 2.1054852320675106e-05,
+      "loss": 3.0456,
+      "step": 500
+    },
+    {
+      "epoch": 0.1392669494207761,
+      "grad_norm": 1.5466052293777466,
+      "learning_rate": 2.3164556962025318e-05,
+      "loss": 2.9099,
+      "step": 550
+    },
+    {
+      "epoch": 0.1519275811863012,
+      "grad_norm": 1.2139467000961304,
+      "learning_rate": 2.5274261603375527e-05,
+      "loss": 2.8839,
+      "step": 600
+    },
+    {
+      "epoch": 0.1519275811863012,
+      "eval_loss": 2.793567657470703,
+      "eval_runtime": 40.2573,
+      "eval_samples_per_second": 43.619,
+      "eval_steps_per_second": 43.619,
+      "step": 600
+    },
+    {
+      "epoch": 0.1645882129518263,
+      "grad_norm": 1.0270315408706665,
+      "learning_rate": 2.738396624472574e-05,
+      "loss": 2.8389,
+      "step": 650
+    },
+    {
+      "epoch": 0.1772488447173514,
+      "grad_norm": 1.5865377187728882,
+      "learning_rate": 2.949367088607595e-05,
+      "loss": 2.8228,
+      "step": 700
+    },
+    {
+      "epoch": 0.18990947648287648,
+      "grad_norm": 1.076073408126831,
+      "learning_rate": 3.160337552742616e-05,
+      "loss": 2.9255,
+      "step": 750
+    },
+    {
+      "epoch": 0.2025701082484016,
+      "grad_norm": 1.4510694742202759,
+      "learning_rate": 3.3713080168776376e-05,
+      "loss": 2.8165,
+      "step": 800
+    },
+    {
+      "epoch": 0.2025701082484016,
+      "eval_loss": 2.667301893234253,
+      "eval_runtime": 40.0906,
+      "eval_samples_per_second": 43.801,
+      "eval_steps_per_second": 43.801,
+      "step": 800
+    },
+    {
+      "epoch": 0.2152307400139267,
+      "grad_norm": 1.5206592082977295,
+      "learning_rate": 3.5822784810126585e-05,
+      "loss": 2.8022,
+      "step": 850
+    },
+    {
+      "epoch": 0.2278913717794518,
+      "grad_norm": 1.173909068107605,
+      "learning_rate": 3.7932489451476794e-05,
+      "loss": 2.8034,
+      "step": 900
+    },
+    {
+      "epoch": 0.24055200354497688,
+      "grad_norm": 1.4551103115081787,
+      "learning_rate": 4.004219409282701e-05,
+      "loss": 2.774,
+      "step": 950
+    },
+    {
+      "epoch": 0.253212635310502,
+      "grad_norm": 1.509749412536621,
+      "learning_rate": 4.215189873417722e-05,
+      "loss": 2.7978,
+      "step": 1000
+    },
+    {
+      "epoch": 0.253212635310502,
+      "eval_loss": 2.603116273880005,
+      "eval_runtime": 39.9617,
+      "eval_samples_per_second": 43.942,
+      "eval_steps_per_second": 43.942,
+      "step": 1000
+    },
+    {
+      "epoch": 0.2658732670760271,
+      "grad_norm": 1.745764136314392,
+      "learning_rate": 4.426160337552743e-05,
+      "loss": 2.7249,
+      "step": 1050
+    },
+    {
+      "epoch": 0.2785338988415522,
+      "grad_norm": 1.6712589263916016,
+      "learning_rate": 4.637130801687764e-05,
+      "loss": 2.7618,
+      "step": 1100
+    },
+    {
+      "epoch": 0.2911945306070773,
+      "grad_norm": 2.256267786026001,
+      "learning_rate": 4.8481012658227845e-05,
+      "loss": 2.7431,
+      "step": 1150
+    },
+    {
+      "epoch": 0.3038551623726024,
+      "grad_norm": 1.5181586742401123,
+      "learning_rate": 4.993434627649597e-05,
+      "loss": 2.778,
+      "step": 1200
+    },
+    {
+      "epoch": 0.3038551623726024,
+      "eval_loss": 2.5704379081726074,
+      "eval_runtime": 40.1791,
+      "eval_samples_per_second": 43.704,
+      "eval_steps_per_second": 43.704,
+      "step": 1200
+    },
+    {
+      "epoch": 0.31651579413812747,
+      "grad_norm": 1.1885608434677124,
+      "learning_rate": 4.969986869255299e-05,
+      "loss": 2.7224,
+      "step": 1250
+    },
+    {
+      "epoch": 0.3291764259036526,
+      "grad_norm": 1.2136404514312744,
+      "learning_rate": 4.946539110861002e-05,
+      "loss": 2.6823,
+      "step": 1300
+    },
+    {
+      "epoch": 0.3418370576691777,
+      "grad_norm": 0.8780732750892639,
+      "learning_rate": 4.9230913524667046e-05,
+      "loss": 2.6961,
+      "step": 1350
+    },
+    {
+      "epoch": 0.3544976894347028,
+      "grad_norm": 1.0844959020614624,
+      "learning_rate": 4.899643594072407e-05,
+      "loss": 2.7014,
+      "step": 1400
+    },
+    {
+      "epoch": 0.3544976894347028,
+      "eval_loss": 2.54587721824646,
+      "eval_runtime": 39.8803,
+      "eval_samples_per_second": 44.032,
+      "eval_steps_per_second": 44.032,
+      "step": 1400
+    },
+    {
+      "epoch": 0.3671583212002279,
+      "grad_norm": 1.3518335819244385,
+      "learning_rate": 4.8761958356781096e-05,
+      "loss": 2.6393,
+      "step": 1450
+    },
+    {
+      "epoch": 0.37981895296575297,
+      "grad_norm": 1.1389687061309814,
+      "learning_rate": 4.852748077283812e-05,
+      "loss": 2.7002,
+      "step": 1500
+    },
+    {
+      "epoch": 0.3924795847312781,
+      "grad_norm": 1.6295430660247803,
+      "learning_rate": 4.829300318889514e-05,
+      "loss": 2.6754,
+      "step": 1550
+    },
+    {
+      "epoch": 0.4051402164968032,
+      "grad_norm": 1.387499451637268,
+      "learning_rate": 4.8058525604952173e-05,
+      "loss": 2.6853,
+      "step": 1600
+    },
+    {
+      "epoch": 0.4051402164968032,
+      "eval_loss": 2.5297553539276123,
+      "eval_runtime": 39.722,
+      "eval_samples_per_second": 44.207,
+      "eval_steps_per_second": 44.207,
+      "step": 1600
+    },
+    {
+      "epoch": 0.4178008482623283,
+      "grad_norm": 1.014020323753357,
+      "learning_rate": 4.7824048021009195e-05,
+      "loss": 2.7275,
+      "step": 1650
+    },
+    {
+      "epoch": 0.4304614800278534,
+      "grad_norm": 1.1505990028381348,
+      "learning_rate": 4.7589570437066216e-05,
+      "loss": 2.6651,
+      "step": 1700
+    },
+    {
+      "epoch": 0.4431221117933785,
+      "grad_norm": 1.1389458179473877,
+      "learning_rate": 4.7355092853123244e-05,
+      "loss": 2.6993,
+      "step": 1750
+    },
+    {
+      "epoch": 0.4557827435589036,
+      "grad_norm": 1.2159587144851685,
+      "learning_rate": 4.7120615269180266e-05,
+      "loss": 2.7239,
+      "step": 1800
+    },
+    {
+      "epoch": 0.4557827435589036,
+      "eval_loss": 2.5201079845428467,
+      "eval_runtime": 39.8313,
+      "eval_samples_per_second": 44.086,
+      "eval_steps_per_second": 44.086,
+      "step": 1800
+    },
+    {
+      "epoch": 0.4684433753244287,
+      "grad_norm": 1.1873971223831177,
+      "learning_rate": 4.6886137685237294e-05,
+      "loss": 2.6368,
+      "step": 1850
+    },
+    {
+      "epoch": 0.48110400708995377,
+      "grad_norm": 1.5109103918075562,
+      "learning_rate": 4.665166010129432e-05,
+      "loss": 2.6827,
+      "step": 1900
+    },
+    {
+      "epoch": 0.4937646388554789,
+      "grad_norm": 1.9981125593185425,
+      "learning_rate": 4.641718251735134e-05,
+      "loss": 2.6513,
+      "step": 1950
+    },
+    {
+      "epoch": 0.506425270621004,
+      "grad_norm": 1.4879294633865356,
+      "learning_rate": 4.6182704933408365e-05,
+      "loss": 2.6433,
+      "step": 2000
+    },
+    {
+      "epoch": 0.506425270621004,
+      "eval_loss": 2.5107176303863525,
+      "eval_runtime": 40.0643,
+      "eval_samples_per_second": 43.83,
+      "eval_steps_per_second": 43.83,
+      "step": 2000
+    },
+    {
+      "epoch": 0.5190859023865291,
+      "grad_norm": 1.2832767963409424,
+      "learning_rate": 4.5952916901144253e-05,
+      "loss": 2.6225,
+      "step": 2050
+    },
+    {
+      "epoch": 0.5317465341520542,
+      "grad_norm": 1.2915899753570557,
+      "learning_rate": 4.5718439317201275e-05,
+      "loss": 2.6592,
+      "step": 2100
+    },
+    {
+      "epoch": 0.5444071659175793,
+      "grad_norm": 1.229929804801941,
+      "learning_rate": 4.54839617332583e-05,
+      "loss": 2.6411,
+      "step": 2150
+    },
+    {
+      "epoch": 0.5570677976831044,
+      "grad_norm": 1.2569608688354492,
+      "learning_rate": 4.524948414931533e-05,
+      "loss": 2.6436,
+      "step": 2200
+    },
+    {
+      "epoch": 0.5570677976831044,
+      "eval_loss": 2.504101514816284,
+      "eval_runtime": 39.8694,
+      "eval_samples_per_second": 44.044,
+      "eval_steps_per_second": 44.044,
+      "step": 2200
+    },
+    {
+      "epoch": 0.5697284294486294,
+      "grad_norm": 1.3688510656356812,
+      "learning_rate": 4.501500656537235e-05,
+      "loss": 2.6819,
+      "step": 2250
+    },
+    {
+      "epoch": 0.5823890612141546,
+      "grad_norm": 1.1405905485153198,
+      "learning_rate": 4.4780528981429374e-05,
+      "loss": 2.6116,
+      "step": 2300
+    },
+    {
+      "epoch": 0.5950496929796797,
+      "grad_norm": 1.453338861465454,
+      "learning_rate": 4.45460513974864e-05,
+      "loss": 2.6154,
+      "step": 2350
+    },
+    {
+      "epoch": 0.6077103247452048,
+      "grad_norm": 1.0401395559310913,
+      "learning_rate": 4.431157381354343e-05,
+      "loss": 2.6018,
+      "step": 2400
+    },
+    {
+      "epoch": 0.6077103247452048,
+      "eval_loss": 2.498344898223877,
+      "eval_runtime": 39.9496,
+      "eval_samples_per_second": 43.955,
+      "eval_steps_per_second": 43.955,
+      "step": 2400
+    },
+    {
+      "epoch": 0.6203709565107299,
+      "grad_norm": 1.4646718502044678,
+      "learning_rate": 4.407709622960045e-05,
+      "loss": 2.5734,
+      "step": 2450
+    },
+    {
+      "epoch": 0.6330315882762549,
+      "grad_norm": 1.3828164339065552,
+      "learning_rate": 4.384261864565748e-05,
+      "loss": 2.6445,
+      "step": 2500
+    },
+    {
+      "epoch": 0.6456922200417801,
+      "grad_norm": 2.1768596172332764,
+      "learning_rate": 4.36081410617145e-05,
+      "loss": 2.6618,
+      "step": 2550
+    },
+    {
+      "epoch": 0.6583528518073052,
+      "grad_norm": 1.6110296249389648,
+      "learning_rate": 4.337366347777152e-05,
+      "loss": 2.6509,
+      "step": 2600
+    },
+    {
+      "epoch": 0.6583528518073052,
+      "eval_loss": 2.4937028884887695,
+      "eval_runtime": 39.8698,
+      "eval_samples_per_second": 44.043,
+      "eval_steps_per_second": 44.043,
+      "step": 2600
+    },
+    {
+      "epoch": 0.6710134835728303,
+      "grad_norm": 1.2363536357879639,
+      "learning_rate": 4.313918589382856e-05,
+      "loss": 2.6274,
+      "step": 2650
+    },
+    {
+      "epoch": 0.6836741153383554,
+      "grad_norm": 2.192110538482666,
+      "learning_rate": 4.290470830988558e-05,
+      "loss": 2.6932,
+      "step": 2700
+    },
+    {
+      "epoch": 0.6963347471038804,
+      "grad_norm": 1.2024074792861938,
+      "learning_rate": 4.26702307259426e-05,
+      "loss": 2.6221,
+      "step": 2750
+    },
+    {
+      "epoch": 0.7089953788694056,
+      "grad_norm": 1.8665797710418701,
+      "learning_rate": 4.243575314199963e-05,
+      "loss": 2.6313,
+      "step": 2800
+    },
+    {
+      "epoch": 0.7089953788694056,
+      "eval_loss": 2.4876773357391357,
+      "eval_runtime": 40.026,
+      "eval_samples_per_second": 43.871,
+      "eval_steps_per_second": 43.871,
+      "step": 2800
+    },
+    {
+      "epoch": 0.7216560106349307,
+      "grad_norm": 1.4088993072509766,
+      "learning_rate": 4.220127555805665e-05,
+      "loss": 2.5675,
+      "step": 2850
+    },
+    {
+      "epoch": 0.7343166424004558,
+      "grad_norm": 1.3225140571594238,
+      "learning_rate": 4.196679797411368e-05,
+      "loss": 2.56,
+      "step": 2900
+    },
+    {
+      "epoch": 0.7469772741659809,
+      "grad_norm": 1.3416539430618286,
+      "learning_rate": 4.1732320390170706e-05,
+      "loss": 2.6517,
+      "step": 2950
+    },
+    {
+      "epoch": 0.7596379059315059,
+      "grad_norm": 1.079567790031433,
+      "learning_rate": 4.149784280622773e-05,
+      "loss": 2.698,
+      "step": 3000
+    },
+    {
+      "epoch": 0.7596379059315059,
+      "eval_loss": 2.4842560291290283,
+      "eval_runtime": 39.7988,
+      "eval_samples_per_second": 44.122,
+      "eval_steps_per_second": 44.122,
+      "step": 3000
+    },
+    {
+      "epoch": 0.772298537697031,
+      "grad_norm": 1.4532116651535034,
+      "learning_rate": 4.126336522228475e-05,
+      "loss": 2.6232,
+      "step": 3050
+    },
+    {
+      "epoch": 0.7849591694625562,
+      "grad_norm": 1.5380038022994995,
+      "learning_rate": 4.102888763834178e-05,
+      "loss": 2.6212,
+      "step": 3100
+    },
+    {
+      "epoch": 0.7976198012280813,
+      "grad_norm": 1.3965916633605957,
+      "learning_rate": 4.0794410054398805e-05,
+      "loss": 2.5804,
+      "step": 3150
+    },
+    {
+      "epoch": 0.8102804329936064,
+      "grad_norm": 1.4798463582992554,
+      "learning_rate": 4.0559932470455826e-05,
+      "loss": 2.6724,
+      "step": 3200
+    },
+    {
+      "epoch": 0.8102804329936064,
+      "eval_loss": 2.480894088745117,
+      "eval_runtime": 39.9604,
+      "eval_samples_per_second": 43.943,
+      "eval_steps_per_second": 43.943,
+      "step": 3200
+    },
+    {
+      "epoch": 0.8229410647591315,
+      "grad_norm": 1.2598360776901245,
+      "learning_rate": 4.0325454886512854e-05,
+      "loss": 2.6993,
+      "step": 3250
+    },
+    {
+      "epoch": 0.8356016965246565,
+      "grad_norm": 1.366295576095581,
+      "learning_rate": 4.0090977302569876e-05,
+      "loss": 2.551,
+      "step": 3300
+    },
+    {
+      "epoch": 0.8482623282901817,
+      "grad_norm": 1.1827855110168457,
+      "learning_rate": 3.98564997186269e-05,
+      "loss": 2.6131,
+      "step": 3350
+    },
+    {
+      "epoch": 0.8609229600557068,
+      "grad_norm": 1.2728627920150757,
+      "learning_rate": 3.9622022134683925e-05,
+      "loss": 2.6178,
+      "step": 3400
+    },
+    {
+      "epoch": 0.8609229600557068,
+      "eval_loss": 2.477010726928711,
+      "eval_runtime": 40.2504,
+      "eval_samples_per_second": 43.627,
+      "eval_steps_per_second": 43.627,
+      "step": 3400
+    },
+    {
+      "epoch": 0.8735835918212319,
+      "grad_norm": 1.341917634010315,
+      "learning_rate": 3.938754455074095e-05,
+      "loss": 2.5748,
+      "step": 3450
+    },
+    {
+      "epoch": 0.886244223586757,
+      "grad_norm": 1.4114609956741333,
+      "learning_rate": 3.9153066966797975e-05,
+      "loss": 2.667,
+      "step": 3500
+    },
+    {
+      "epoch": 0.898904855352282,
+      "grad_norm": 1.1211490631103516,
+      "learning_rate": 3.8918589382855e-05,
+      "loss": 2.5671,
+      "step": 3550
+    },
+    {
+      "epoch": 0.9115654871178072,
+      "grad_norm": 1.4166322946548462,
+      "learning_rate": 3.8684111798912024e-05,
+      "loss": 2.5945,
+      "step": 3600
+    },
+    {
+      "epoch": 0.9115654871178072,
+      "eval_loss": 2.47322940826416,
+      "eval_runtime": 40.2079,
+      "eval_samples_per_second": 43.673,
+      "eval_steps_per_second": 43.673,
+      "step": 3600
+    },
+    {
+      "epoch": 0.9242261188833323,
+      "grad_norm": 0.9144394993782043,
+      "learning_rate": 3.844963421496905e-05,
+      "loss": 2.6148,
+      "step": 3650
+    },
+    {
+      "epoch": 0.9368867506488574,
+      "grad_norm": 1.4106061458587646,
+      "learning_rate": 3.821515663102608e-05,
+      "loss": 2.6586,
+      "step": 3700
+    },
+    {
+      "epoch": 0.9495473824143825,
+      "grad_norm": 1.414415717124939,
+      "learning_rate": 3.79806790470831e-05,
+      "loss": 2.5874,
+      "step": 3750
+    },
+    {
+      "epoch": 0.9622080141799075,
+      "grad_norm": 1.5448992252349854,
+      "learning_rate": 3.774620146314012e-05,
+      "loss": 2.6422,
+      "step": 3800
+    },
+    {
+      "epoch": 0.9622080141799075,
+      "eval_loss": 2.4701173305511475,
+      "eval_runtime": 40.1267,
+      "eval_samples_per_second": 43.761,
+      "eval_steps_per_second": 43.761,
+      "step": 3800
+    },
+    {
+      "epoch": 0.9748686459454327,
+      "grad_norm": 1.1959314346313477,
+      "learning_rate": 3.751172387919715e-05,
+      "loss": 2.6975,
+      "step": 3850
+    },
+    {
+      "epoch": 0.9875292777109578,
+      "grad_norm": 0.9525274038314819,
+      "learning_rate": 3.727724629525417e-05,
+      "loss": 2.6675,
+      "step": 3900
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 4.733253479003906,
+      "learning_rate": 3.70427687113112e-05,
+      "loss": 2.566,
+      "step": 3950
+    },
+    {
+      "epoch": 1.0126606317655251,
+      "grad_norm": 1.2803192138671875,
+      "learning_rate": 3.680829112736823e-05,
+      "loss": 2.5659,
+      "step": 4000
+    },
+    {
+      "epoch": 1.0126606317655251,
+      "eval_loss": 2.4702188968658447,
+      "eval_runtime": 40.1387,
+      "eval_samples_per_second": 43.748,
+      "eval_steps_per_second": 43.748,
+      "step": 4000
+    },
+    {
+      "epoch": 1.0253212635310502,
+      "grad_norm": 1.446990966796875,
+      "learning_rate": 3.657381354342525e-05,
+      "loss": 2.627,
+      "step": 4050
+    },
+    {
+      "epoch": 1.0379818952965754,
+      "grad_norm": 1.3563008308410645,
+      "learning_rate": 3.633933595948227e-05,
+      "loss": 2.6252,
+      "step": 4100
+    },
+    {
+      "epoch": 1.0506425270621005,
+      "grad_norm": 1.5763463973999023,
+      "learning_rate": 3.61048583755393e-05,
+      "loss": 2.6593,
+      "step": 4150
+    },
+    {
+      "epoch": 1.0633031588276256,
+      "grad_norm": 1.0055335760116577,
+      "learning_rate": 3.587038079159633e-05,
+      "loss": 2.5955,
+      "step": 4200
+    },
+    {
+      "epoch": 1.0633031588276256,
+      "eval_loss": 2.4676930904388428,
+      "eval_runtime": 40.0342,
+      "eval_samples_per_second": 43.863,
+      "eval_steps_per_second": 43.863,
+      "step": 4200
+    },
+    {
+      "epoch": 1.0759637905931505,
+      "grad_norm": 1.7013343572616577,
+      "learning_rate": 3.563590320765335e-05,
+      "loss": 2.59,
+      "step": 4250
+    },
+    {
+      "epoch": 1.0886244223586756,
+      "grad_norm": 1.541069507598877,
+      "learning_rate": 3.540142562371038e-05,
+      "loss": 2.6192,
+      "step": 4300
+    },
+    {
+      "epoch": 1.1012850541242007,
+      "grad_norm": 1.2536805868148804,
+      "learning_rate": 3.51669480397674e-05,
+      "loss": 2.6225,
+      "step": 4350
+    },
+    {
+      "epoch": 1.1139456858897259,
+      "grad_norm": 1.8328826427459717,
+      "learning_rate": 3.493247045582442e-05,
+      "loss": 2.6022,
+      "step": 4400
+    },
+    {
+      "epoch": 1.1139456858897259,
+      "eval_loss": 2.465629816055298,
+      "eval_runtime": 39.8532,
+      "eval_samples_per_second": 44.062,
+      "eval_steps_per_second": 44.062,
+      "step": 4400
+    },
+    {
+      "epoch": 1.126606317655251,
+      "grad_norm": 1.8557270765304565,
+      "learning_rate": 3.469799287188145e-05,
+      "loss": 2.6496,
+      "step": 4450
+    },
+    {
+      "epoch": 1.139266949420776,
+      "grad_norm": 1.3255618810653687,
+      "learning_rate": 3.446820483961734e-05,
+      "loss": 2.5315,
+      "step": 4500
+    },
+    {
+      "epoch": 1.1519275811863012,
+      "grad_norm": 1.2192399501800537,
+      "learning_rate": 3.423372725567436e-05,
+      "loss": 2.5409,
+      "step": 4550
+    },
+    {
+      "epoch": 1.1645882129518264,
+      "grad_norm": 1.2533234357833862,
+      "learning_rate": 3.399924967173139e-05,
+      "loss": 2.6457,
+      "step": 4600
+    },
+    {
+      "epoch": 1.1645882129518264,
+      "eval_loss": 2.462162733078003,
+      "eval_runtime": 40.0542,
+      "eval_samples_per_second": 43.841,
+      "eval_steps_per_second": 43.841,
+      "step": 4600
+    },
+    {
+      "epoch": 1.1772488447173515,
+      "grad_norm": 1.8414678573608398,
+      "learning_rate": 3.376477208778841e-05,
+      "loss": 2.5658,
+      "step": 4650
+    },
+    {
+      "epoch": 1.1899094764828764,
+      "grad_norm": 1.568259596824646,
+      "learning_rate": 3.3530294503845436e-05,
+      "loss": 2.5771,
+      "step": 4700
+    },
+    {
+      "epoch": 1.2025701082484015,
+      "grad_norm": 1.3547483682632446,
+      "learning_rate": 3.3295816919902464e-05,
+      "loss": 2.6525,
+      "step": 4750
+    },
+    {
+      "epoch": 1.2152307400139266,
+      "grad_norm": 1.1655386686325073,
+      "learning_rate": 3.3061339335959486e-05,
+      "loss": 2.6421,
+      "step": 4800
+    },
+    {
+      "epoch": 1.2152307400139266,
+      "eval_loss": 2.461489200592041,
+      "eval_runtime": 39.9962,
+      "eval_samples_per_second": 43.904,
+      "eval_steps_per_second": 43.904,
+      "step": 4800
+    },
+    {
+      "epoch": 1.2278913717794517,
+      "grad_norm": 1.798033595085144,
+      "learning_rate": 3.282686175201651e-05,
+      "loss": 2.6091,
+      "step": 4850
+    },
+    {
+      "epoch": 1.2405520035449769,
+      "grad_norm": 3.2964117527008057,
+      "learning_rate": 3.2592384168073535e-05,
+      "loss": 2.5997,
+      "step": 4900
+    },
+    {
+      "epoch": 1.253212635310502,
+      "grad_norm": 1.0457675457000732,
+      "learning_rate": 3.2357906584130557e-05,
+      "loss": 2.6144,
+      "step": 4950
+    },
+    {
+      "epoch": 1.265873267076027,
+      "grad_norm": 0.9728056192398071,
+      "learning_rate": 3.2123429000187585e-05,
+      "loss": 2.5712,
+      "step": 5000
+    },
+    {
+      "epoch": 1.265873267076027,
+      "eval_loss": 2.460186719894409,
+      "eval_runtime": 39.8386,
+      "eval_samples_per_second": 44.078,
+      "eval_steps_per_second": 44.078,
+      "step": 5000
+    },
+    {
+      "epoch": 1.2785338988415522,
+      "grad_norm": 1.2350194454193115,
+      "learning_rate": 3.188895141624461e-05,
+      "loss": 2.5448,
+      "step": 5050
+    },
+    {
+      "epoch": 1.2911945306070773,
+      "grad_norm": 1.4210622310638428,
+      "learning_rate": 3.1654473832301634e-05,
+      "loss": 2.6031,
+      "step": 5100
+    },
+    {
+      "epoch": 1.3038551623726025,
+      "grad_norm": 2.226473093032837,
+      "learning_rate": 3.1419996248358656e-05,
+      "loss": 2.6597,
+      "step": 5150
+    },
+    {
+      "epoch": 1.3165157941381276,
+      "grad_norm": 2.4525105953216553,
+      "learning_rate": 3.1185518664415684e-05,
+      "loss": 2.596,
+      "step": 5200
+    },
+    {
+      "epoch": 1.3165157941381276,
+      "eval_loss": 2.454537868499756,
+      "eval_runtime": 39.805,
+      "eval_samples_per_second": 44.115,
+      "eval_steps_per_second": 44.115,
+      "step": 5200
+    },
+    {
+      "epoch": 1.3291764259036527,
+      "grad_norm": 1.265309453010559,
+      "learning_rate": 3.095104108047271e-05,
+      "loss": 2.5559,
+      "step": 5250
+    },
+    {
+      "epoch": 1.3418370576691778,
+      "grad_norm": 2.1364307403564453,
+      "learning_rate": 3.071656349652973e-05,
+      "loss": 2.5859,
+      "step": 5300
+    },
+    {
+      "epoch": 1.3544976894347027,
+      "grad_norm": 1.5945920944213867,
+      "learning_rate": 3.048208591258676e-05,
+      "loss": 2.5778,
+      "step": 5350
+    },
+    {
+      "epoch": 1.3671583212002278,
+      "grad_norm": 1.2479759454727173,
+      "learning_rate": 3.0247608328643783e-05,
+      "loss": 2.6846,
+      "step": 5400
+    },
+    {
+      "epoch": 1.3671583212002278,
+      "eval_loss": 2.4547293186187744,
+      "eval_runtime": 39.7806,
+      "eval_samples_per_second": 44.142,
+      "eval_steps_per_second": 44.142,
+      "step": 5400
+    },
+    {
+      "epoch": 1.379818952965753,
+      "grad_norm": 1.4845050573349,
+      "learning_rate": 3.0013130744700808e-05,
+      "loss": 2.5661,
+      "step": 5450
+    },
+    {
+      "epoch": 1.392479584731278,
+      "grad_norm": 1.5581985712051392,
+      "learning_rate": 2.9778653160757836e-05,
+      "loss": 2.5441,
+      "step": 5500
+    },
+    {
+      "epoch": 1.4051402164968032,
+      "grad_norm": 3.1663737297058105,
+      "learning_rate": 2.9544175576814857e-05,
+      "loss": 2.5044,
+      "step": 5550
+    },
+    {
+      "epoch": 1.4178008482623283,
+      "grad_norm": 1.2454484701156616,
+      "learning_rate": 2.9309697992871882e-05,
+      "loss": 2.5747,
+      "step": 5600
+    },
+    {
+      "epoch": 1.4178008482623283,
+      "eval_loss": 2.4544529914855957,
+      "eval_runtime": 39.9287,
+      "eval_samples_per_second": 43.978,
+      "eval_steps_per_second": 43.978,
+      "step": 5600
+    },
+    {
+      "epoch": 1.4304614800278534,
+      "grad_norm": 1.662784457206726,
+      "learning_rate": 2.907522040892891e-05,
+      "loss": 2.6064,
+      "step": 5650
+    },
+    {
+      "epoch": 1.4431221117933786,
+      "grad_norm": 1.618458867073059,
+      "learning_rate": 2.8840742824985935e-05,
+      "loss": 2.5191,
+      "step": 5700
+    },
+    {
+      "epoch": 1.4557827435589035,
+      "grad_norm": 1.3003348112106323,
+      "learning_rate": 2.8606265241042956e-05,
+      "loss": 2.5339,
+      "step": 5750
+    },
+    {
+      "epoch": 1.4684433753244286,
+      "grad_norm": 1.1443992853164673,
+      "learning_rate": 2.8371787657099984e-05,
+      "loss": 2.5914,
+      "step": 5800
+    },
+    {
+      "epoch": 1.4684433753244286,
+      "eval_loss": 2.453752279281616,
+      "eval_runtime": 39.8234,
+      "eval_samples_per_second": 44.095,
+      "eval_steps_per_second": 44.095,
+      "step": 5800
+    },
+    {
+      "epoch": 1.4811040070899537,
+      "grad_norm": 1.2574009895324707,
+      "learning_rate": 2.813731007315701e-05,
+      "loss": 2.6109,
+      "step": 5850
+    },
+    {
+      "epoch": 1.4937646388554788,
+      "grad_norm": 1.002815842628479,
+      "learning_rate": 2.790283248921403e-05,
+      "loss": 2.6075,
+      "step": 5900
+    },
+    {
+      "epoch": 1.506425270621004,
+      "grad_norm": 1.306024432182312,
+      "learning_rate": 2.766835490527106e-05,
+      "loss": 2.5733,
+      "step": 5950
+    },
+    {
+      "epoch": 1.519085902386529,
+      "grad_norm": 2.5023701190948486,
+      "learning_rate": 2.7433877321328083e-05,
+      "loss": 2.6274,
+      "step": 6000
+    },
+    {
+      "epoch": 1.519085902386529,
+      "eval_loss": 2.449617862701416,
+      "eval_runtime": 39.9401,
+      "eval_samples_per_second": 43.966,
+      "eval_steps_per_second": 43.966,
+      "step": 6000
+    },
+    {
+      "epoch": 1.5317465341520542,
+      "grad_norm": 1.9410326480865479,
+      "learning_rate": 2.7199399737385105e-05,
+      "loss": 2.5312,
+      "step": 6050
+    },
+    {
+      "epoch": 1.5444071659175793,
+      "grad_norm": 1.9793561697006226,
+      "learning_rate": 2.6964922153442136e-05,
+      "loss": 2.5759,
+      "step": 6100
+    },
+    {
+      "epoch": 1.5570677976831044,
+      "grad_norm": 1.290531873703003,
+      "learning_rate": 2.6730444569499157e-05,
+      "loss": 2.5817,
+      "step": 6150
+    },
+    {
+      "epoch": 1.5697284294486296,
+      "grad_norm": 2.11389422416687,
+      "learning_rate": 2.6495966985556182e-05,
+      "loss": 2.6287,
+      "step": 6200
+    },
+    {
+      "epoch": 1.5697284294486296,
+      "eval_loss": 2.4490554332733154,
+      "eval_runtime": 39.7474,
+      "eval_samples_per_second": 44.179,
+      "eval_steps_per_second": 44.179,
+      "step": 6200
+    },
+    {
+      "epoch": 1.5823890612141547,
+      "grad_norm": 1.6492938995361328,
+      "learning_rate": 2.626148940161321e-05,
+      "loss": 2.631,
+      "step": 6250
+    },
+    {
+      "epoch": 1.5950496929796798,
+      "grad_norm": 1.3233673572540283,
+      "learning_rate": 2.6027011817670232e-05,
+      "loss": 2.5654,
+      "step": 6300
+    },
+    {
+      "epoch": 1.607710324745205,
+      "grad_norm": 1.688264012336731,
+      "learning_rate": 2.5792534233727257e-05,
+      "loss": 2.6096,
+      "step": 6350
+    },
+    {
+      "epoch": 1.62037095651073,
+      "grad_norm": 2.064823865890503,
+      "learning_rate": 2.5558056649784285e-05,
+      "loss": 2.6275,
+      "step": 6400
+    },
+    {
+      "epoch": 1.62037095651073,
+      "eval_loss": 2.4488983154296875,
+      "eval_runtime": 39.6847,
+      "eval_samples_per_second": 44.249,
+      "eval_steps_per_second": 44.249,
+      "step": 6400
+    },
+    {
+      "epoch": 1.633031588276255,
+      "grad_norm": 1.5599696636199951,
+      "learning_rate": 2.5323579065841306e-05,
+      "loss": 2.6334,
+      "step": 6450
+    },
+    {
+      "epoch": 1.64569222004178,
+      "grad_norm": 1.3142633438110352,
+      "learning_rate": 2.508910148189833e-05,
+      "loss": 2.5496,
+      "step": 6500
+    },
+    {
+      "epoch": 1.6583528518073052,
+      "grad_norm": 1.474135160446167,
+      "learning_rate": 2.4854623897955356e-05,
+      "loss": 2.5628,
+      "step": 6550
+    },
+    {
+      "epoch": 1.6710134835728303,
+      "grad_norm": 1.3737610578536987,
+      "learning_rate": 2.4620146314012384e-05,
+      "loss": 2.5345,
+      "step": 6600
+    },
+    {
+      "epoch": 1.6710134835728303,
+      "eval_loss": 2.447559356689453,
+      "eval_runtime": 39.7021,
+      "eval_samples_per_second": 44.229,
+      "eval_steps_per_second": 44.229,
+      "step": 6600
+    },
+    {
+      "epoch": 1.6836741153383554,
+      "grad_norm": 1.2432060241699219,
+      "learning_rate": 2.4385668730069405e-05,
+      "loss": 2.5977,
+      "step": 6650
+    },
+    {
+      "epoch": 1.6963347471038803,
+      "grad_norm": 1.465063452720642,
+      "learning_rate": 2.415119114612643e-05,
+      "loss": 2.6118,
+      "step": 6700
+    },
+    {
+      "epoch": 1.7089953788694054,
+      "grad_norm": 1.5186200141906738,
+      "learning_rate": 2.3916713562183458e-05,
+      "loss": 2.6126,
+      "step": 6750
+    },
+    {
+      "epoch": 1.7216560106349306,
+      "grad_norm": 1.6869078874588013,
+      "learning_rate": 2.368223597824048e-05,
+      "loss": 2.576,
+      "step": 6800
+    },
+    {
+      "epoch": 1.7216560106349306,
+      "eval_loss": 2.4459502696990967,
+      "eval_runtime": 39.7653,
+      "eval_samples_per_second": 44.159,
+      "eval_steps_per_second": 44.159,
+      "step": 6800
+    },
+    {
+      "epoch": 1.7343166424004557,
+      "grad_norm": 1.2578104734420776,
+      "learning_rate": 2.3447758394297507e-05,
+      "loss": 2.6178,
+      "step": 6850
+    },
+    {
+      "epoch": 1.7469772741659808,
+      "grad_norm": 1.7597213983535767,
+      "learning_rate": 2.3213280810354532e-05,
+      "loss": 2.6358,
+      "step": 6900
+    },
+    {
+      "epoch": 1.759637905931506,
+      "grad_norm": 2.144465923309326,
+      "learning_rate": 2.2978803226411554e-05,
+      "loss": 2.5597,
+      "step": 6950
+    },
+    {
+      "epoch": 1.772298537697031,
+      "grad_norm": 1.1808464527130127,
+      "learning_rate": 2.2744325642468582e-05,
+      "loss": 2.6269,
+      "step": 7000
+    },
+    {
+      "epoch": 1.772298537697031,
+      "eval_loss": 2.4444611072540283,
+      "eval_runtime": 40.0709,
+      "eval_samples_per_second": 43.822,
+      "eval_steps_per_second": 43.822,
+      "step": 7000
+    },
+    {
+      "epoch": 1.7849591694625562,
+      "grad_norm": 1.4550806283950806,
+      "learning_rate": 2.2509848058525606e-05,
+      "loss": 2.6206,
+      "step": 7050
+    },
+    {
+      "epoch": 1.7976198012280813,
+      "grad_norm": 1.2635902166366577,
+      "learning_rate": 2.227537047458263e-05,
+      "loss": 2.5722,
+      "step": 7100
+    },
+    {
+      "epoch": 1.8102804329936064,
+      "grad_norm": 1.3835856914520264,
+      "learning_rate": 2.2040892890639656e-05,
+      "loss": 2.535,
+      "step": 7150
+    },
+    {
+      "epoch": 1.8229410647591315,
+      "grad_norm": 1.735004186630249,
+      "learning_rate": 2.180641530669668e-05,
+      "loss": 2.6086,
+      "step": 7200
+    },
+    {
+      "epoch": 1.8229410647591315,
+      "eval_loss": 2.443899154663086,
+      "eval_runtime": 40.91,
+      "eval_samples_per_second": 42.924,
+      "eval_steps_per_second": 42.924,
+      "step": 7200
+    },
+    {
+      "epoch": 1.8356016965246567,
+      "grad_norm": 1.263051986694336,
+      "learning_rate": 2.1571937722753706e-05,
+      "loss": 2.5544,
+      "step": 7250
+    },
+    {
+      "epoch": 1.8482623282901818,
+      "grad_norm": 1.0899442434310913,
+      "learning_rate": 2.133746013881073e-05,
+      "loss": 2.5603,
+      "step": 7300
+    },
+    {
+      "epoch": 1.860922960055707,
+      "grad_norm": 3.038811206817627,
+      "learning_rate": 2.1102982554867755e-05,
+      "loss": 2.5688,
+      "step": 7350
+    },
+    {
+      "epoch": 1.873583591821232,
+      "grad_norm": 1.6385984420776367,
+      "learning_rate": 2.086850497092478e-05,
+      "loss": 2.6006,
+      "step": 7400
+    },
+    {
+      "epoch": 1.873583591821232,
+      "eval_loss": 2.443300724029541,
+      "eval_runtime": 40.6649,
+      "eval_samples_per_second": 43.182,
+      "eval_steps_per_second": 43.182,
+      "step": 7400
+    },
+    {
+      "epoch": 1.8862442235867571,
+      "grad_norm": 1.2857129573822021,
+      "learning_rate": 2.0634027386981805e-05,
+      "loss": 2.5563,
+      "step": 7450
+    },
+    {
+      "epoch": 1.898904855352282,
+      "grad_norm": 1.0289497375488281,
+      "learning_rate": 2.0399549803038833e-05,
+      "loss": 2.5671,
+      "step": 7500
+    },
+    {
+      "epoch": 1.9115654871178072,
+      "grad_norm": 1.5041025876998901,
+      "learning_rate": 2.0165072219095854e-05,
+      "loss": 2.5689,
+      "step": 7550
+    },
+    {
+      "epoch": 1.9242261188833323,
+      "grad_norm": 1.6611964702606201,
+      "learning_rate": 1.993528418683174e-05,
+      "loss": 2.5801,
+      "step": 7600
+    },
+    {
+      "epoch": 1.9242261188833323,
+      "eval_loss": 2.443532943725586,
+      "eval_runtime": 39.931,
+      "eval_samples_per_second": 43.976,
+      "eval_steps_per_second": 43.976,
+      "step": 7600
+    },
+    {
+      "epoch": 1.9368867506488574,
+      "grad_norm": 1.521170735359192,
+      "learning_rate": 1.9700806602888767e-05,
+      "loss": 2.5969,
+      "step": 7650
+    },
+    {
+      "epoch": 1.9495473824143825,
+      "grad_norm": 1.3700034618377686,
+      "learning_rate": 1.946632901894579e-05,
+      "loss": 2.6306,
+      "step": 7700
+    },
+    {
+      "epoch": 1.9622080141799074,
+      "grad_norm": 2.311443328857422,
+      "learning_rate": 1.9231851435002814e-05,
+      "loss": 2.5608,
+      "step": 7750
+    },
+    {
+      "epoch": 1.9748686459454325,
+      "grad_norm": 1.6699820756912231,
+      "learning_rate": 1.8997373851059842e-05,
+      "loss": 2.5113,
+      "step": 7800
+    },
+    {
+      "epoch": 1.9748686459454325,
+      "eval_loss": 2.4421675205230713,
+      "eval_runtime": 40.1783,
+      "eval_samples_per_second": 43.705,
+      "eval_steps_per_second": 43.705,
+      "step": 7800
+    },
+    {
+      "epoch": 1.9875292777109577,
+      "grad_norm": 1.2560683488845825,
+      "learning_rate": 1.8762896267116863e-05,
+      "loss": 2.545,
+      "step": 7850
+    },
+    {
+      "epoch": 2.0,
+      "grad_norm": 2.176563262939453,
+      "learning_rate": 1.852841868317389e-05,
+      "loss": 2.5752,
+      "step": 7900
+    },
+    {
+      "epoch": 2.012660631765525,
+      "grad_norm": 1.2551178932189941,
+      "learning_rate": 1.8293941099230916e-05,
+      "loss": 2.5215,
+      "step": 7950
+    },
+    {
+      "epoch": 2.0253212635310502,
+      "grad_norm": 1.5646872520446777,
+      "learning_rate": 1.8059463515287937e-05,
+      "loss": 2.5838,
+      "step": 8000
+    },
+    {
+      "epoch": 2.0253212635310502,
+      "eval_loss": 2.441195249557495,
+      "eval_runtime": 39.701,
+      "eval_samples_per_second": 44.231,
+      "eval_steps_per_second": 44.231,
+      "step": 8000
+    },
+    {
+      "epoch": 2.0379818952965754,
+      "grad_norm": 1.4227900505065918,
+      "learning_rate": 1.7824985931344966e-05,
+      "loss": 2.5597,
+      "step": 8050
+    },
+    {
+      "epoch": 2.0506425270621005,
+      "grad_norm": 1.3013832569122314,
+      "learning_rate": 1.759050834740199e-05,
+      "loss": 2.7641,
+      "step": 8100
+    },
+    {
+      "epoch": 2.0633031588276256,
+      "grad_norm": 1.1282143592834473,
+      "learning_rate": 1.7356030763459015e-05,
+      "loss": 2.5875,
+      "step": 8150
+    },
+    {
+      "epoch": 2.0759637905931507,
+      "grad_norm": 2.079760789871216,
+      "learning_rate": 1.712155317951604e-05,
+      "loss": 2.4861,
+      "step": 8200
+    },
+    {
+      "epoch": 2.0759637905931507,
+      "eval_loss": 2.440812826156616,
+      "eval_runtime": 40.0764,
+      "eval_samples_per_second": 43.816,
+      "eval_steps_per_second": 43.816,
+      "step": 8200
+    },
+    {
+      "epoch": 2.088624422358676,
+      "grad_norm": 1.0884991884231567,
+      "learning_rate": 1.6887075595573065e-05,
+      "loss": 2.5941,
+      "step": 8250
+    },
+    {
+      "epoch": 2.101285054124201,
+      "grad_norm": 1.9202015399932861,
+      "learning_rate": 1.665259801163009e-05,
+      "loss": 2.5929,
+      "step": 8300
+    },
+    {
+      "epoch": 2.113945685889726,
+      "grad_norm": 1.5925830602645874,
+      "learning_rate": 1.6418120427687114e-05,
+      "loss": 2.5046,
+      "step": 8350
+    },
+    {
+      "epoch": 2.126606317655251,
+      "grad_norm": 1.5219184160232544,
+      "learning_rate": 1.618364284374414e-05,
+      "loss": 2.5628,
+      "step": 8400
+    },
+    {
+      "epoch": 2.126606317655251,
+      "eval_loss": 2.4396440982818604,
+      "eval_runtime": 40.0053,
+      "eval_samples_per_second": 43.894,
+      "eval_steps_per_second": 43.894,
+      "step": 8400
+    },
+    {
+      "epoch": 2.139266949420776,
+      "grad_norm": 1.4882445335388184,
+      "learning_rate": 1.5949165259801164e-05,
+      "loss": 2.6268,
+      "step": 8450
+    },
+    {
+      "epoch": 2.151927581186301,
+      "grad_norm": 1.3513301610946655,
+      "learning_rate": 1.571468767585819e-05,
+      "loss": 2.5277,
+      "step": 8500
+    },
+    {
+      "epoch": 2.164588212951826,
+      "grad_norm": 1.690974473953247,
+      "learning_rate": 1.5480210091915216e-05,
+      "loss": 2.5631,
+      "step": 8550
+    },
+    {
+      "epoch": 2.1772488447173513,
+      "grad_norm": 1.5311528444290161,
+      "learning_rate": 1.5245732507972238e-05,
+      "loss": 2.5454,
+      "step": 8600
+    },
+    {
+      "epoch": 2.1772488447173513,
+      "eval_loss": 2.4388718605041504,
+      "eval_runtime": 40.0289,
+      "eval_samples_per_second": 43.868,
+      "eval_steps_per_second": 43.868,
+      "step": 8600
+    },
+    {
+      "epoch": 2.1899094764828764,
+      "grad_norm": 2.1171281337738037,
+      "learning_rate": 1.5011254924029264e-05,
+      "loss": 2.6112,
+      "step": 8650
+    },
+    {
+      "epoch": 2.2025701082484015,
+      "grad_norm": 1.9706814289093018,
+      "learning_rate": 1.4776777340086289e-05,
+      "loss": 2.588,
+      "step": 8700
+    },
+    {
+      "epoch": 2.2152307400139266,
+      "grad_norm": 1.8991297483444214,
+      "learning_rate": 1.4542299756143312e-05,
+      "loss": 2.5655,
+      "step": 8750
+    },
+    {
+      "epoch": 2.2278913717794517,
+      "grad_norm": 1.5568820238113403,
+      "learning_rate": 1.4307822172200339e-05,
+      "loss": 2.5312,
+      "step": 8800
+    },
+    {
+      "epoch": 2.2278913717794517,
+      "eval_loss": 2.438715696334839,
+      "eval_runtime": 40.1748,
+      "eval_samples_per_second": 43.709,
+      "eval_steps_per_second": 43.709,
+      "step": 8800
+    },
+    {
+      "epoch": 2.240552003544977,
+      "grad_norm": 1.277051329612732,
+      "learning_rate": 1.4073344588257365e-05,
+      "loss": 2.5818,
+      "step": 8850
+    },
+    {
+      "epoch": 2.253212635310502,
+      "grad_norm": 1.8890128135681152,
+      "learning_rate": 1.3838867004314388e-05,
+      "loss": 2.5211,
+      "step": 8900
+    },
+    {
+      "epoch": 2.265873267076027,
+      "grad_norm": 1.8824830055236816,
+      "learning_rate": 1.3604389420371413e-05,
+      "loss": 2.53,
+      "step": 8950
+    },
+    {
+      "epoch": 2.278533898841552,
+      "grad_norm": 1.239490032196045,
+      "learning_rate": 1.336991183642844e-05,
+      "loss": 2.5889,
+      "step": 9000
+    },
+    {
+      "epoch": 2.278533898841552,
+      "eval_loss": 2.437577962875366,
+      "eval_runtime": 40.1654,
+      "eval_samples_per_second": 43.719,
+      "eval_steps_per_second": 43.719,
+      "step": 9000
+    },
+    {
+      "epoch": 2.2911945306070773,
+      "grad_norm": 1.7253328561782837,
+      "learning_rate": 1.3135434252485462e-05,
+      "loss": 2.5426,
+      "step": 9050
+    },
+    {
+      "epoch": 2.3038551623726025,
+      "grad_norm": 1.6971838474273682,
+      "learning_rate": 1.2900956668542489e-05,
+      "loss": 2.4953,
+      "step": 9100
+    },
+    {
+      "epoch": 2.3165157941381276,
+      "grad_norm": 1.4906270503997803,
+      "learning_rate": 1.2666479084599514e-05,
+      "loss": 2.606,
+      "step": 9150
+    },
+    {
+      "epoch": 2.3291764259036527,
+      "grad_norm": 1.658526062965393,
+      "learning_rate": 1.2432001500656538e-05,
+      "loss": 2.5483,
+      "step": 9200
+    },
+    {
+      "epoch": 2.3291764259036527,
+      "eval_loss": 2.437896490097046,
+      "eval_runtime": 40.7008,
+      "eval_samples_per_second": 43.144,
+      "eval_steps_per_second": 43.144,
+      "step": 9200
+    },
+    {
+      "epoch": 2.341837057669178,
+      "grad_norm": 1.0781177282333374,
+      "learning_rate": 1.2197523916713563e-05,
+      "loss": 2.5449,
+      "step": 9250
+    },
+    {
+      "epoch": 2.354497689434703,
+      "grad_norm": 2.1414873600006104,
+      "learning_rate": 1.1963046332770588e-05,
+      "loss": 2.5303,
+      "step": 9300
+    },
+    {
+      "epoch": 2.367158321200228,
+      "grad_norm": 2.063297986984253,
+      "learning_rate": 1.1728568748827613e-05,
+      "loss": 2.5837,
+      "step": 9350
+    },
+    {
+      "epoch": 2.3798189529657527,
+      "grad_norm": 1.2153489589691162,
+      "learning_rate": 1.1494091164884637e-05,
+      "loss": 2.6384,
+      "step": 9400
+    },
+    {
+      "epoch": 2.3798189529657527,
+      "eval_loss": 2.4365696907043457,
+      "eval_runtime": 40.2398,
+      "eval_samples_per_second": 43.638,
+      "eval_steps_per_second": 43.638,
+      "step": 9400
+    },
+    {
+      "epoch": 2.3924795847312783,
+      "grad_norm": 1.2976094484329224,
+      "learning_rate": 1.1259613580941662e-05,
+      "loss": 2.572,
+      "step": 9450
+    },
+    {
+      "epoch": 2.405140216496803,
+      "grad_norm": 1.2775920629501343,
+      "learning_rate": 1.1025135996998689e-05,
+      "loss": 2.6083,
+      "step": 9500
+    },
+    {
+      "epoch": 2.417800848262328,
+      "grad_norm": 1.358311653137207,
+      "learning_rate": 1.0790658413055713e-05,
+      "loss": 2.5206,
+      "step": 9550
+    },
+    {
+      "epoch": 2.4304614800278532,
+      "grad_norm": 1.3438369035720825,
+      "learning_rate": 1.0556180829112736e-05,
+      "loss": 2.4967,
+      "step": 9600
+    },
+    {
+      "epoch": 2.4304614800278532,
+      "eval_loss": 2.4359662532806396,
+      "eval_runtime": 40.0802,
+      "eval_samples_per_second": 43.812,
+      "eval_steps_per_second": 43.812,
+      "step": 9600
+    },
+    {
+      "epoch": 2.4431221117933783,
+      "grad_norm": 1.2618831396102905,
+      "learning_rate": 1.0321703245169763e-05,
+      "loss": 2.6169,
+      "step": 9650
+    },
+    {
+      "epoch": 2.4557827435589035,
+      "grad_norm": 1.3764727115631104,
+      "learning_rate": 1.0087225661226788e-05,
+      "loss": 2.5444,
+      "step": 9700
+    },
+    {
+      "epoch": 2.4684433753244286,
+      "grad_norm": 1.604864478111267,
+      "learning_rate": 9.852748077283812e-06,
+      "loss": 2.5343,
+      "step": 9750
+    },
+    {
+      "epoch": 2.4811040070899537,
+      "grad_norm": 1.390496850013733,
+      "learning_rate": 9.618270493340837e-06,
+      "loss": 2.5051,
+      "step": 9800
+    },
+    {
+      "epoch": 2.4811040070899537,
+      "eval_loss": 2.4353232383728027,
+      "eval_runtime": 40.1607,
+      "eval_samples_per_second": 43.724,
+      "eval_steps_per_second": 43.724,
+      "step": 9800
+    },
+    {
+      "epoch": 2.493764638855479,
+      "grad_norm": 2.1982169151306152,
+      "learning_rate": 9.383792909397862e-06,
+      "loss": 2.5036,
+      "step": 9850
+    },
+    {
+      "epoch": 2.506425270621004,
+      "grad_norm": 1.3033822774887085,
+      "learning_rate": 9.149315325454887e-06,
+      "loss": 2.5205,
+      "step": 9900
+    },
+    {
+      "epoch": 2.519085902386529,
+      "grad_norm": 1.682586431503296,
+      "learning_rate": 8.919527293190772e-06,
+      "loss": 2.6083,
+      "step": 9950
+    },
+    {
+      "epoch": 2.531746534152054,
+      "grad_norm": 3.184382200241089,
+      "learning_rate": 8.685049709247797e-06,
+      "loss": 2.5314,
+      "step": 10000
+    },
+    {
+      "epoch": 2.531746534152054,
+      "eval_loss": 2.434755802154541,
+      "eval_runtime": 40.2877,
+      "eval_samples_per_second": 43.587,
+      "eval_steps_per_second": 43.587,
+      "step": 10000
+    },
+    {
+      "epoch": 2.5444071659175793,
+      "grad_norm": 2.0026867389678955,
+      "learning_rate": 8.450572125304821e-06,
+      "loss": 2.5109,
+      "step": 10050
+    },
+    {
+      "epoch": 2.5570677976831044,
+      "grad_norm": 1.3833885192871094,
+      "learning_rate": 8.216094541361846e-06,
+      "loss": 2.5362,
+      "step": 10100
+    },
+    {
+      "epoch": 2.5697284294486296,
+      "grad_norm": 2.157984495162964,
+      "learning_rate": 7.981616957418871e-06,
+      "loss": 2.5423,
+      "step": 10150
+    },
+    {
+      "epoch": 2.5823890612141547,
+      "grad_norm": 1.682053565979004,
+      "learning_rate": 7.747139373475897e-06,
+      "loss": 2.5133,
+      "step": 10200
+    },
+    {
+      "epoch": 2.5823890612141547,
+      "eval_loss": 2.435208559036255,
+      "eval_runtime": 40.4768,
+      "eval_samples_per_second": 43.383,
+      "eval_steps_per_second": 43.383,
+      "step": 10200
+    },
+    {
+      "epoch": 2.59504969297968,
+      "grad_norm": 1.9720139503479004,
+      "learning_rate": 7.512661789532921e-06,
+      "loss": 2.6372,
+      "step": 10250
+    },
+    {
+      "epoch": 2.607710324745205,
+      "grad_norm": 1.6906607151031494,
+      "learning_rate": 7.278184205589945e-06,
+      "loss": 2.5505,
+      "step": 10300
+    },
+    {
+      "epoch": 2.62037095651073,
+      "grad_norm": 1.484045147895813,
+      "learning_rate": 7.043706621646972e-06,
+      "loss": 2.5095,
+      "step": 10350
+    },
+    {
+      "epoch": 2.633031588276255,
+      "grad_norm": 1.6676850318908691,
+      "learning_rate": 6.8092290377039955e-06,
+      "loss": 2.6487,
+      "step": 10400
+    },
+    {
+      "epoch": 2.633031588276255,
+      "eval_loss": 2.4344091415405273,
+      "eval_runtime": 39.869,
+      "eval_samples_per_second": 44.044,
+      "eval_steps_per_second": 44.044,
+      "step": 10400
+    },
+    {
+      "epoch": 2.64569222004178,
+      "grad_norm": 1.5012388229370117,
+      "learning_rate": 6.57475145376102e-06,
+      "loss": 2.5756,
+      "step": 10450
+    },
+    {
+      "epoch": 2.6583528518073054,
+      "grad_norm": 1.043954849243164,
+      "learning_rate": 6.340273869818046e-06,
+      "loss": 2.5843,
+      "step": 10500
+    },
+    {
+      "epoch": 2.67101348357283,
+      "grad_norm": 1.0455141067504883,
+      "learning_rate": 6.105796285875071e-06,
+      "loss": 2.5248,
+      "step": 10550
+    },
+    {
+      "epoch": 2.6836741153383556,
+      "grad_norm": 1.39467453956604,
+      "learning_rate": 5.871318701932095e-06,
+      "loss": 2.5091,
+      "step": 10600
+    },
+    {
+      "epoch": 2.6836741153383556,
+      "eval_loss": 2.4331610202789307,
+      "eval_runtime": 39.923,
+      "eval_samples_per_second": 43.985,
+      "eval_steps_per_second": 43.985,
+      "step": 10600
+    },
+    {
+      "epoch": 2.6963347471038803,
+      "grad_norm": 1.1417715549468994,
+      "learning_rate": 5.63684111798912e-06,
+      "loss": 2.5853,
+      "step": 10650
+    },
+    {
+      "epoch": 2.7089953788694054,
+      "grad_norm": 1.133244514465332,
+      "learning_rate": 5.402363534046146e-06,
+      "loss": 2.5457,
+      "step": 10700
+    },
+    {
+      "epoch": 2.7216560106349306,
+      "grad_norm": 1.2331452369689941,
+      "learning_rate": 5.1678859501031705e-06,
+      "loss": 2.5576,
+      "step": 10750
+    },
+    {
+      "epoch": 2.7343166424004557,
+      "grad_norm": 1.7164263725280762,
+      "learning_rate": 4.933408366160195e-06,
+      "loss": 2.5471,
+      "step": 10800
+    },
+    {
+      "epoch": 2.7343166424004557,
+      "eval_loss": 2.4340403079986572,
+      "eval_runtime": 40.3849,
+      "eval_samples_per_second": 43.482,
+      "eval_steps_per_second": 43.482,
+      "step": 10800
+    },
+    {
+      "epoch": 2.746977274165981,
+      "grad_norm": 1.3680106401443481,
+      "learning_rate": 4.69893078221722e-06,
+      "loss": 2.5562,
+      "step": 10850
+    },
+    {
+      "epoch": 2.759637905931506,
+      "grad_norm": 1.0978279113769531,
+      "learning_rate": 4.464453198274246e-06,
+      "loss": 2.5185,
+      "step": 10900
+    },
+    {
+      "epoch": 2.772298537697031,
+      "grad_norm": 1.2212647199630737,
+      "learning_rate": 4.2299756143312695e-06,
+      "loss": 2.6371,
+      "step": 10950
+    },
+    {
+      "epoch": 2.784959169462556,
+      "grad_norm": 1.6452165842056274,
+      "learning_rate": 3.995498030388295e-06,
+      "loss": 2.681,
+      "step": 11000
+    },
+    {
+      "epoch": 2.784959169462556,
+      "eval_loss": 2.4337143898010254,
+      "eval_runtime": 40.4235,
+      "eval_samples_per_second": 43.44,
+      "eval_steps_per_second": 43.44,
+      "step": 11000
+    },
+    {
+      "epoch": 2.7976198012280813,
+      "grad_norm": 1.7757978439331055,
+      "learning_rate": 3.7610204464453203e-06,
+      "loss": 2.5746,
+      "step": 11050
+    },
+    {
+      "epoch": 2.8102804329936064,
+      "grad_norm": 1.2373579740524292,
+      "learning_rate": 3.5265428625023455e-06,
+      "loss": 2.5412,
+      "step": 11100
+    },
+    {
+      "epoch": 2.8229410647591315,
+      "grad_norm": 1.1407558917999268,
+      "learning_rate": 3.29206527855937e-06,
+      "loss": 2.5973,
+      "step": 11150
+    },
+    {
+      "epoch": 2.8356016965246567,
+      "grad_norm": 2.399686813354492,
+      "learning_rate": 3.057587694616395e-06,
+      "loss": 2.5566,
+      "step": 11200
+    },
+    {
+      "epoch": 2.8356016965246567,
+      "eval_loss": 2.4338231086730957,
+      "eval_runtime": 40.4877,
+      "eval_samples_per_second": 43.371,
+      "eval_steps_per_second": 43.371,
+      "step": 11200
+    },
+    {
+      "epoch": 2.8482623282901818,
+      "grad_norm": 1.7053141593933105,
+      "learning_rate": 2.8231101106734197e-06,
+      "loss": 2.6224,
+      "step": 11250
+    },
+    {
+      "epoch": 2.860922960055707,
+      "grad_norm": 1.8215903043746948,
+      "learning_rate": 2.5886325267304445e-06,
+      "loss": 2.5108,
+      "step": 11300
+    },
+    {
+      "epoch": 2.873583591821232,
+      "grad_norm": 1.1648200750350952,
+      "learning_rate": 2.3541549427874697e-06,
+      "loss": 2.557,
+      "step": 11350
+    },
+    {
+      "epoch": 2.886244223586757,
+      "grad_norm": 1.5225868225097656,
+      "learning_rate": 2.1196773588444944e-06,
+      "loss": 2.6285,
+      "step": 11400
+    },
+    {
+      "epoch": 2.886244223586757,
+      "eval_loss": 2.4334514141082764,
+      "eval_runtime": 40.3985,
+      "eval_samples_per_second": 43.467,
+      "eval_steps_per_second": 43.467,
+      "step": 11400
+    },
+    {
+      "epoch": 2.8989048553522823,
+      "grad_norm": 1.4937622547149658,
+      "learning_rate": 1.8851997749015194e-06,
+      "loss": 2.481,
+      "step": 11450
+    },
+    {
+      "epoch": 2.911565487117807,
+      "grad_norm": 1.9169902801513672,
+      "learning_rate": 1.6507221909585446e-06,
+      "loss": 2.5412,
+      "step": 11500
+    },
+    {
+      "epoch": 2.9242261188833325,
+      "grad_norm": 1.6611114740371704,
+      "learning_rate": 1.4162446070155693e-06,
+      "loss": 2.5086,
+      "step": 11550
+    },
+    {
+      "epoch": 2.936886750648857,
+      "grad_norm": 1.3464007377624512,
+      "learning_rate": 1.1817670230725943e-06,
+      "loss": 2.6063,
+      "step": 11600
+    },
+    {
+      "epoch": 2.936886750648857,
+      "eval_loss": 2.4329476356506348,
+      "eval_runtime": 40.4334,
+      "eval_samples_per_second": 43.429,
+      "eval_steps_per_second": 43.429,
+      "step": 11600
+    },
+    {
+      "epoch": 2.9495473824143827,
+      "grad_norm": 1.453385829925537,
+      "learning_rate": 9.472894391296193e-07,
+      "loss": 2.5012,
+      "step": 11650
+    },
+    {
+      "epoch": 2.9622080141799074,
+      "grad_norm": 1.6921356916427612,
+      "learning_rate": 7.128118551866442e-07,
+      "loss": 2.5589,
+      "step": 11700
+    },
+    {
+      "epoch": 2.9748686459454325,
+      "grad_norm": 1.0562982559204102,
+      "learning_rate": 4.783342712436691e-07,
+      "loss": 2.6015,
+      "step": 11750
+    },
+    {
+      "epoch": 2.9875292777109577,
+      "grad_norm": 1.457960844039917,
+      "learning_rate": 2.4385668730069406e-07,
+      "loss": 2.6224,
+      "step": 11800
+    },
+    {
+      "epoch": 2.9875292777109577,
+      "eval_loss": 2.432849645614624,
+      "eval_runtime": 40.4518,
+      "eval_samples_per_second": 43.41,
+      "eval_steps_per_second": 43.41,
+      "step": 11800
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 11847,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6224498036047872.0,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-11800/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d46b7107c3dfd17ff0fb12b8cdcc256a49e8d4594358d4495904a5fcb069382
+size 5304

checkpoint-11847/README.md

@@ -0,0 +1,202 @@
+---
+base_model: gpt2
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.14.0

checkpoint-11847/adapter_config.json

@@ -0,0 +1,32 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "gpt2",
+  "bias": "none",
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_proj",
+    "c_attn"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-11847/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2096525bcf5c5b06858aea39ffe9f1d86e5e5f698c630fe8a7b6968326c4147d
+size 3253104

checkpoint-11847/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d317fa4b44b54d0a2990dc5b855d3892cbb408e7299889f7dfde8d83f90dec55
+size 6548858

checkpoint-11847/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47d7b1d7efd16ee4ff475df79309ca7861c4a0fa87b8f21e5bc85746b2437479
+size 14244

checkpoint-11847/scaler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ace690d17ecc0bd723919a7caf76c676aa6af99ac2aad28a489002b232ed59bd
+size 988

checkpoint-11847/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:31739f88076d66774095d513dd74fc5f8222fb41697cf1f0e9e3ad7cd14e52b4
+size 1064

checkpoint-11847/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-11847/tokenizer_config.json

@@ -0,0 +1,21 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-11847/trainer_state.json

@@ -0,0 +1,2158 @@
+{
+  "best_global_step": 11800,
+  "best_metric": 2.432849645614624,
+  "best_model_checkpoint": "/kaggle/working/senko_adaptive/checkpoint-11800",
+  "epoch": 2.999430271570551,
+  "eval_steps": 200,
+  "global_step": 11847,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0126606317655251,
+      "grad_norm": 0.4274106025695801,
+      "learning_rate": 2.067510548523207e-06,
+      "loss": 3.4405,
+      "step": 50
+    },
+    {
+      "epoch": 0.0253212635310502,
+      "grad_norm": 0.5292551517486572,
+      "learning_rate": 4.177215189873418e-06,
+      "loss": 3.4567,
+      "step": 100
+    },
+    {
+      "epoch": 0.0379818952965753,
+      "grad_norm": 0.7541739344596863,
+      "learning_rate": 6.28691983122363e-06,
+      "loss": 3.4683,
+      "step": 150
+    },
+    {
+      "epoch": 0.0506425270621004,
+      "grad_norm": 0.8833445906639099,
+      "learning_rate": 8.39662447257384e-06,
+      "loss": 3.5084,
+      "step": 200
+    },
+    {
+      "epoch": 0.0506425270621004,
+      "eval_loss": 3.5248923301696777,
+      "eval_runtime": 39.9384,
+      "eval_samples_per_second": 43.968,
+      "eval_steps_per_second": 43.968,
+      "step": 200
+    },
+    {
+      "epoch": 0.0633031588276255,
+      "grad_norm": 0.9998921155929565,
+      "learning_rate": 1.0506329113924052e-05,
+      "loss": 3.359,
+      "step": 250
+    },
+    {
+      "epoch": 0.0759637905931506,
+      "grad_norm": 0.8041885495185852,
+      "learning_rate": 1.2616033755274262e-05,
+      "loss": 3.351,
+      "step": 300
+    },
+    {
+      "epoch": 0.0886244223586757,
+      "grad_norm": 0.9213416576385498,
+      "learning_rate": 1.4725738396624473e-05,
+      "loss": 3.2244,
+      "step": 350
+    },
+    {
+      "epoch": 0.1012850541242008,
+      "grad_norm": 1.0922213792800903,
+      "learning_rate": 1.6835443037974685e-05,
+      "loss": 3.1565,
+      "step": 400
+    },
+    {
+      "epoch": 0.1012850541242008,
+      "eval_loss": 3.151216983795166,
+      "eval_runtime": 39.7515,
+      "eval_samples_per_second": 44.174,
+      "eval_steps_per_second": 44.174,
+      "step": 400
+    },
+    {
+      "epoch": 0.1139456858897259,
+      "grad_norm": 1.4199283123016357,
+      "learning_rate": 1.8945147679324897e-05,
+      "loss": 3.0154,
+      "step": 450
+    },
+    {
+      "epoch": 0.126606317655251,
+      "grad_norm": 1.077143907546997,
+      "learning_rate": 2.1054852320675106e-05,
+      "loss": 3.0456,
+      "step": 500
+    },
+    {
+      "epoch": 0.1392669494207761,
+      "grad_norm": 1.5466052293777466,
+      "learning_rate": 2.3164556962025318e-05,
+      "loss": 2.9099,
+      "step": 550
+    },
+    {
+      "epoch": 0.1519275811863012,
+      "grad_norm": 1.2139467000961304,
+      "learning_rate": 2.5274261603375527e-05,
+      "loss": 2.8839,
+      "step": 600
+    },
+    {
+      "epoch": 0.1519275811863012,
+      "eval_loss": 2.793567657470703,
+      "eval_runtime": 40.2573,
+      "eval_samples_per_second": 43.619,
+      "eval_steps_per_second": 43.619,
+      "step": 600
+    },
+    {
+      "epoch": 0.1645882129518263,
+      "grad_norm": 1.0270315408706665,
+      "learning_rate": 2.738396624472574e-05,
+      "loss": 2.8389,
+      "step": 650
+    },
+    {
+      "epoch": 0.1772488447173514,
+      "grad_norm": 1.5865377187728882,
+      "learning_rate": 2.949367088607595e-05,
+      "loss": 2.8228,
+      "step": 700
+    },
+    {
+      "epoch": 0.18990947648287648,
+      "grad_norm": 1.076073408126831,
+      "learning_rate": 3.160337552742616e-05,
+      "loss": 2.9255,
+      "step": 750
+    },
+    {
+      "epoch": 0.2025701082484016,
+      "grad_norm": 1.4510694742202759,
+      "learning_rate": 3.3713080168776376e-05,
+      "loss": 2.8165,
+      "step": 800
+    },
+    {
+      "epoch": 0.2025701082484016,
+      "eval_loss": 2.667301893234253,
+      "eval_runtime": 40.0906,
+      "eval_samples_per_second": 43.801,
+      "eval_steps_per_second": 43.801,
+      "step": 800
+    },
+    {
+      "epoch": 0.2152307400139267,
+      "grad_norm": 1.5206592082977295,
+      "learning_rate": 3.5822784810126585e-05,
+      "loss": 2.8022,
+      "step": 850
+    },
+    {
+      "epoch": 0.2278913717794518,
+      "grad_norm": 1.173909068107605,
+      "learning_rate": 3.7932489451476794e-05,
+      "loss": 2.8034,
+      "step": 900
+    },
+    {
+      "epoch": 0.24055200354497688,
+      "grad_norm": 1.4551103115081787,
+      "learning_rate": 4.004219409282701e-05,
+      "loss": 2.774,
+      "step": 950
+    },
+    {
+      "epoch": 0.253212635310502,
+      "grad_norm": 1.509749412536621,
+      "learning_rate": 4.215189873417722e-05,
+      "loss": 2.7978,
+      "step": 1000
+    },
+    {
+      "epoch": 0.253212635310502,
+      "eval_loss": 2.603116273880005,
+      "eval_runtime": 39.9617,
+      "eval_samples_per_second": 43.942,
+      "eval_steps_per_second": 43.942,
+      "step": 1000
+    },
+    {
+      "epoch": 0.2658732670760271,
+      "grad_norm": 1.745764136314392,
+      "learning_rate": 4.426160337552743e-05,
+      "loss": 2.7249,
+      "step": 1050
+    },
+    {
+      "epoch": 0.2785338988415522,
+      "grad_norm": 1.6712589263916016,
+      "learning_rate": 4.637130801687764e-05,
+      "loss": 2.7618,
+      "step": 1100
+    },
+    {
+      "epoch": 0.2911945306070773,
+      "grad_norm": 2.256267786026001,
+      "learning_rate": 4.8481012658227845e-05,
+      "loss": 2.7431,
+      "step": 1150
+    },
+    {
+      "epoch": 0.3038551623726024,
+      "grad_norm": 1.5181586742401123,
+      "learning_rate": 4.993434627649597e-05,
+      "loss": 2.778,
+      "step": 1200
+    },
+    {
+      "epoch": 0.3038551623726024,
+      "eval_loss": 2.5704379081726074,
+      "eval_runtime": 40.1791,
+      "eval_samples_per_second": 43.704,
+      "eval_steps_per_second": 43.704,
+      "step": 1200
+    },
+    {
+      "epoch": 0.31651579413812747,
+      "grad_norm": 1.1885608434677124,
+      "learning_rate": 4.969986869255299e-05,
+      "loss": 2.7224,
+      "step": 1250
+    },
+    {
+      "epoch": 0.3291764259036526,
+      "grad_norm": 1.2136404514312744,
+      "learning_rate": 4.946539110861002e-05,
+      "loss": 2.6823,
+      "step": 1300
+    },
+    {
+      "epoch": 0.3418370576691777,
+      "grad_norm": 0.8780732750892639,
+      "learning_rate": 4.9230913524667046e-05,
+      "loss": 2.6961,
+      "step": 1350
+    },
+    {
+      "epoch": 0.3544976894347028,
+      "grad_norm": 1.0844959020614624,
+      "learning_rate": 4.899643594072407e-05,
+      "loss": 2.7014,
+      "step": 1400
+    },
+    {
+      "epoch": 0.3544976894347028,
+      "eval_loss": 2.54587721824646,
+      "eval_runtime": 39.8803,
+      "eval_samples_per_second": 44.032,
+      "eval_steps_per_second": 44.032,
+      "step": 1400
+    },
+    {
+      "epoch": 0.3671583212002279,
+      "grad_norm": 1.3518335819244385,
+      "learning_rate": 4.8761958356781096e-05,
+      "loss": 2.6393,
+      "step": 1450
+    },
+    {
+      "epoch": 0.37981895296575297,
+      "grad_norm": 1.1389687061309814,
+      "learning_rate": 4.852748077283812e-05,
+      "loss": 2.7002,
+      "step": 1500
+    },
+    {
+      "epoch": 0.3924795847312781,
+      "grad_norm": 1.6295430660247803,
+      "learning_rate": 4.829300318889514e-05,
+      "loss": 2.6754,
+      "step": 1550
+    },
+    {
+      "epoch": 0.4051402164968032,
+      "grad_norm": 1.387499451637268,
+      "learning_rate": 4.8058525604952173e-05,
+      "loss": 2.6853,
+      "step": 1600
+    },
+    {
+      "epoch": 0.4051402164968032,
+      "eval_loss": 2.5297553539276123,
+      "eval_runtime": 39.722,
+      "eval_samples_per_second": 44.207,
+      "eval_steps_per_second": 44.207,
+      "step": 1600
+    },
+    {
+      "epoch": 0.4178008482623283,
+      "grad_norm": 1.014020323753357,
+      "learning_rate": 4.7824048021009195e-05,
+      "loss": 2.7275,
+      "step": 1650
+    },
+    {
+      "epoch": 0.4304614800278534,
+      "grad_norm": 1.1505990028381348,
+      "learning_rate": 4.7589570437066216e-05,
+      "loss": 2.6651,
+      "step": 1700
+    },
+    {
+      "epoch": 0.4431221117933785,
+      "grad_norm": 1.1389458179473877,
+      "learning_rate": 4.7355092853123244e-05,
+      "loss": 2.6993,
+      "step": 1750
+    },
+    {
+      "epoch": 0.4557827435589036,
+      "grad_norm": 1.2159587144851685,
+      "learning_rate": 4.7120615269180266e-05,
+      "loss": 2.7239,
+      "step": 1800
+    },
+    {
+      "epoch": 0.4557827435589036,
+      "eval_loss": 2.5201079845428467,
+      "eval_runtime": 39.8313,
+      "eval_samples_per_second": 44.086,
+      "eval_steps_per_second": 44.086,
+      "step": 1800
+    },
+    {
+      "epoch": 0.4684433753244287,
+      "grad_norm": 1.1873971223831177,
+      "learning_rate": 4.6886137685237294e-05,
+      "loss": 2.6368,
+      "step": 1850
+    },
+    {
+      "epoch": 0.48110400708995377,
+      "grad_norm": 1.5109103918075562,
+      "learning_rate": 4.665166010129432e-05,
+      "loss": 2.6827,
+      "step": 1900
+    },
+    {
+      "epoch": 0.4937646388554789,
+      "grad_norm": 1.9981125593185425,
+      "learning_rate": 4.641718251735134e-05,
+      "loss": 2.6513,
+      "step": 1950
+    },
+    {
+      "epoch": 0.506425270621004,
+      "grad_norm": 1.4879294633865356,
+      "learning_rate": 4.6182704933408365e-05,
+      "loss": 2.6433,
+      "step": 2000
+    },
+    {
+      "epoch": 0.506425270621004,
+      "eval_loss": 2.5107176303863525,
+      "eval_runtime": 40.0643,
+      "eval_samples_per_second": 43.83,
+      "eval_steps_per_second": 43.83,
+      "step": 2000
+    },
+    {
+      "epoch": 0.5190859023865291,
+      "grad_norm": 1.2832767963409424,
+      "learning_rate": 4.5952916901144253e-05,
+      "loss": 2.6225,
+      "step": 2050
+    },
+    {
+      "epoch": 0.5317465341520542,
+      "grad_norm": 1.2915899753570557,
+      "learning_rate": 4.5718439317201275e-05,
+      "loss": 2.6592,
+      "step": 2100
+    },
+    {
+      "epoch": 0.5444071659175793,
+      "grad_norm": 1.229929804801941,
+      "learning_rate": 4.54839617332583e-05,
+      "loss": 2.6411,
+      "step": 2150
+    },
+    {
+      "epoch": 0.5570677976831044,
+      "grad_norm": 1.2569608688354492,
+      "learning_rate": 4.524948414931533e-05,
+      "loss": 2.6436,
+      "step": 2200
+    },
+    {
+      "epoch": 0.5570677976831044,
+      "eval_loss": 2.504101514816284,
+      "eval_runtime": 39.8694,
+      "eval_samples_per_second": 44.044,
+      "eval_steps_per_second": 44.044,
+      "step": 2200
+    },
+    {
+      "epoch": 0.5697284294486294,
+      "grad_norm": 1.3688510656356812,
+      "learning_rate": 4.501500656537235e-05,
+      "loss": 2.6819,
+      "step": 2250
+    },
+    {
+      "epoch": 0.5823890612141546,
+      "grad_norm": 1.1405905485153198,
+      "learning_rate": 4.4780528981429374e-05,
+      "loss": 2.6116,
+      "step": 2300
+    },
+    {
+      "epoch": 0.5950496929796797,
+      "grad_norm": 1.453338861465454,
+      "learning_rate": 4.45460513974864e-05,
+      "loss": 2.6154,
+      "step": 2350
+    },
+    {
+      "epoch": 0.6077103247452048,
+      "grad_norm": 1.0401395559310913,
+      "learning_rate": 4.431157381354343e-05,
+      "loss": 2.6018,
+      "step": 2400
+    },
+    {
+      "epoch": 0.6077103247452048,
+      "eval_loss": 2.498344898223877,
+      "eval_runtime": 39.9496,
+      "eval_samples_per_second": 43.955,
+      "eval_steps_per_second": 43.955,
+      "step": 2400
+    },
+    {
+      "epoch": 0.6203709565107299,
+      "grad_norm": 1.4646718502044678,
+      "learning_rate": 4.407709622960045e-05,
+      "loss": 2.5734,
+      "step": 2450
+    },
+    {
+      "epoch": 0.6330315882762549,
+      "grad_norm": 1.3828164339065552,
+      "learning_rate": 4.384261864565748e-05,
+      "loss": 2.6445,
+      "step": 2500
+    },
+    {
+      "epoch": 0.6456922200417801,
+      "grad_norm": 2.1768596172332764,
+      "learning_rate": 4.36081410617145e-05,
+      "loss": 2.6618,
+      "step": 2550
+    },
+    {
+      "epoch": 0.6583528518073052,
+      "grad_norm": 1.6110296249389648,
+      "learning_rate": 4.337366347777152e-05,
+      "loss": 2.6509,
+      "step": 2600
+    },
+    {
+      "epoch": 0.6583528518073052,
+      "eval_loss": 2.4937028884887695,
+      "eval_runtime": 39.8698,
+      "eval_samples_per_second": 44.043,
+      "eval_steps_per_second": 44.043,
+      "step": 2600
+    },
+    {
+      "epoch": 0.6710134835728303,
+      "grad_norm": 1.2363536357879639,
+      "learning_rate": 4.313918589382856e-05,
+      "loss": 2.6274,
+      "step": 2650
+    },
+    {
+      "epoch": 0.6836741153383554,
+      "grad_norm": 2.192110538482666,
+      "learning_rate": 4.290470830988558e-05,
+      "loss": 2.6932,
+      "step": 2700
+    },
+    {
+      "epoch": 0.6963347471038804,
+      "grad_norm": 1.2024074792861938,
+      "learning_rate": 4.26702307259426e-05,
+      "loss": 2.6221,
+      "step": 2750
+    },
+    {
+      "epoch": 0.7089953788694056,
+      "grad_norm": 1.8665797710418701,
+      "learning_rate": 4.243575314199963e-05,
+      "loss": 2.6313,
+      "step": 2800
+    },
+    {
+      "epoch": 0.7089953788694056,
+      "eval_loss": 2.4876773357391357,
+      "eval_runtime": 40.026,
+      "eval_samples_per_second": 43.871,
+      "eval_steps_per_second": 43.871,
+      "step": 2800
+    },
+    {
+      "epoch": 0.7216560106349307,
+      "grad_norm": 1.4088993072509766,
+      "learning_rate": 4.220127555805665e-05,
+      "loss": 2.5675,
+      "step": 2850
+    },
+    {
+      "epoch": 0.7343166424004558,
+      "grad_norm": 1.3225140571594238,
+      "learning_rate": 4.196679797411368e-05,
+      "loss": 2.56,
+      "step": 2900
+    },
+    {
+      "epoch": 0.7469772741659809,
+      "grad_norm": 1.3416539430618286,
+      "learning_rate": 4.1732320390170706e-05,
+      "loss": 2.6517,
+      "step": 2950
+    },
+    {
+      "epoch": 0.7596379059315059,
+      "grad_norm": 1.079567790031433,
+      "learning_rate": 4.149784280622773e-05,
+      "loss": 2.698,
+      "step": 3000
+    },
+    {
+      "epoch": 0.7596379059315059,
+      "eval_loss": 2.4842560291290283,
+      "eval_runtime": 39.7988,
+      "eval_samples_per_second": 44.122,
+      "eval_steps_per_second": 44.122,
+      "step": 3000
+    },
+    {
+      "epoch": 0.772298537697031,
+      "grad_norm": 1.4532116651535034,
+      "learning_rate": 4.126336522228475e-05,
+      "loss": 2.6232,
+      "step": 3050
+    },
+    {
+      "epoch": 0.7849591694625562,
+      "grad_norm": 1.5380038022994995,
+      "learning_rate": 4.102888763834178e-05,
+      "loss": 2.6212,
+      "step": 3100
+    },
+    {
+      "epoch": 0.7976198012280813,
+      "grad_norm": 1.3965916633605957,
+      "learning_rate": 4.0794410054398805e-05,
+      "loss": 2.5804,
+      "step": 3150
+    },
+    {
+      "epoch": 0.8102804329936064,
+      "grad_norm": 1.4798463582992554,
+      "learning_rate": 4.0559932470455826e-05,
+      "loss": 2.6724,
+      "step": 3200
+    },
+    {
+      "epoch": 0.8102804329936064,
+      "eval_loss": 2.480894088745117,
+      "eval_runtime": 39.9604,
+      "eval_samples_per_second": 43.943,
+      "eval_steps_per_second": 43.943,
+      "step": 3200
+    },
+    {
+      "epoch": 0.8229410647591315,
+      "grad_norm": 1.2598360776901245,
+      "learning_rate": 4.0325454886512854e-05,
+      "loss": 2.6993,
+      "step": 3250
+    },
+    {
+      "epoch": 0.8356016965246565,
+      "grad_norm": 1.366295576095581,
+      "learning_rate": 4.0090977302569876e-05,
+      "loss": 2.551,
+      "step": 3300
+    },
+    {
+      "epoch": 0.8482623282901817,
+      "grad_norm": 1.1827855110168457,
+      "learning_rate": 3.98564997186269e-05,
+      "loss": 2.6131,
+      "step": 3350
+    },
+    {
+      "epoch": 0.8609229600557068,
+      "grad_norm": 1.2728627920150757,
+      "learning_rate": 3.9622022134683925e-05,
+      "loss": 2.6178,
+      "step": 3400
+    },
+    {
+      "epoch": 0.8609229600557068,
+      "eval_loss": 2.477010726928711,
+      "eval_runtime": 40.2504,
+      "eval_samples_per_second": 43.627,
+      "eval_steps_per_second": 43.627,
+      "step": 3400
+    },
+    {
+      "epoch": 0.8735835918212319,
+      "grad_norm": 1.341917634010315,
+      "learning_rate": 3.938754455074095e-05,
+      "loss": 2.5748,
+      "step": 3450
+    },
+    {
+      "epoch": 0.886244223586757,
+      "grad_norm": 1.4114609956741333,
+      "learning_rate": 3.9153066966797975e-05,
+      "loss": 2.667,
+      "step": 3500
+    },
+    {
+      "epoch": 0.898904855352282,
+      "grad_norm": 1.1211490631103516,
+      "learning_rate": 3.8918589382855e-05,
+      "loss": 2.5671,
+      "step": 3550
+    },
+    {
+      "epoch": 0.9115654871178072,
+      "grad_norm": 1.4166322946548462,
+      "learning_rate": 3.8684111798912024e-05,
+      "loss": 2.5945,
+      "step": 3600
+    },
+    {
+      "epoch": 0.9115654871178072,
+      "eval_loss": 2.47322940826416,
+      "eval_runtime": 40.2079,
+      "eval_samples_per_second": 43.673,
+      "eval_steps_per_second": 43.673,
+      "step": 3600
+    },
+    {
+      "epoch": 0.9242261188833323,
+      "grad_norm": 0.9144394993782043,
+      "learning_rate": 3.844963421496905e-05,
+      "loss": 2.6148,
+      "step": 3650
+    },
+    {
+      "epoch": 0.9368867506488574,
+      "grad_norm": 1.4106061458587646,
+      "learning_rate": 3.821515663102608e-05,
+      "loss": 2.6586,
+      "step": 3700
+    },
+    {
+      "epoch": 0.9495473824143825,
+      "grad_norm": 1.414415717124939,
+      "learning_rate": 3.79806790470831e-05,
+      "loss": 2.5874,
+      "step": 3750
+    },
+    {
+      "epoch": 0.9622080141799075,
+      "grad_norm": 1.5448992252349854,
+      "learning_rate": 3.774620146314012e-05,
+      "loss": 2.6422,
+      "step": 3800
+    },
+    {
+      "epoch": 0.9622080141799075,
+      "eval_loss": 2.4701173305511475,
+      "eval_runtime": 40.1267,
+      "eval_samples_per_second": 43.761,
+      "eval_steps_per_second": 43.761,
+      "step": 3800
+    },
+    {
+      "epoch": 0.9748686459454327,
+      "grad_norm": 1.1959314346313477,
+      "learning_rate": 3.751172387919715e-05,
+      "loss": 2.6975,
+      "step": 3850
+    },
+    {
+      "epoch": 0.9875292777109578,
+      "grad_norm": 0.9525274038314819,
+      "learning_rate": 3.727724629525417e-05,
+      "loss": 2.6675,
+      "step": 3900
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 4.733253479003906,
+      "learning_rate": 3.70427687113112e-05,
+      "loss": 2.566,
+      "step": 3950
+    },
+    {
+      "epoch": 1.0126606317655251,
+      "grad_norm": 1.2803192138671875,
+      "learning_rate": 3.680829112736823e-05,
+      "loss": 2.5659,
+      "step": 4000
+    },
+    {
+      "epoch": 1.0126606317655251,
+      "eval_loss": 2.4702188968658447,
+      "eval_runtime": 40.1387,
+      "eval_samples_per_second": 43.748,
+      "eval_steps_per_second": 43.748,
+      "step": 4000
+    },
+    {
+      "epoch": 1.0253212635310502,
+      "grad_norm": 1.446990966796875,
+      "learning_rate": 3.657381354342525e-05,
+      "loss": 2.627,
+      "step": 4050
+    },
+    {
+      "epoch": 1.0379818952965754,
+      "grad_norm": 1.3563008308410645,
+      "learning_rate": 3.633933595948227e-05,
+      "loss": 2.6252,
+      "step": 4100
+    },
+    {
+      "epoch": 1.0506425270621005,
+      "grad_norm": 1.5763463973999023,
+      "learning_rate": 3.61048583755393e-05,
+      "loss": 2.6593,
+      "step": 4150
+    },
+    {
+      "epoch": 1.0633031588276256,
+      "grad_norm": 1.0055335760116577,
+      "learning_rate": 3.587038079159633e-05,
+      "loss": 2.5955,
+      "step": 4200
+    },
+    {
+      "epoch": 1.0633031588276256,
+      "eval_loss": 2.4676930904388428,
+      "eval_runtime": 40.0342,
+      "eval_samples_per_second": 43.863,
+      "eval_steps_per_second": 43.863,
+      "step": 4200
+    },
+    {
+      "epoch": 1.0759637905931505,
+      "grad_norm": 1.7013343572616577,
+      "learning_rate": 3.563590320765335e-05,
+      "loss": 2.59,
+      "step": 4250
+    },
+    {
+      "epoch": 1.0886244223586756,
+      "grad_norm": 1.541069507598877,
+      "learning_rate": 3.540142562371038e-05,
+      "loss": 2.6192,
+      "step": 4300
+    },
+    {
+      "epoch": 1.1012850541242007,
+      "grad_norm": 1.2536805868148804,
+      "learning_rate": 3.51669480397674e-05,
+      "loss": 2.6225,
+      "step": 4350
+    },
+    {
+      "epoch": 1.1139456858897259,
+      "grad_norm": 1.8328826427459717,
+      "learning_rate": 3.493247045582442e-05,
+      "loss": 2.6022,
+      "step": 4400
+    },
+    {
+      "epoch": 1.1139456858897259,
+      "eval_loss": 2.465629816055298,
+      "eval_runtime": 39.8532,
+      "eval_samples_per_second": 44.062,
+      "eval_steps_per_second": 44.062,
+      "step": 4400
+    },
+    {
+      "epoch": 1.126606317655251,
+      "grad_norm": 1.8557270765304565,
+      "learning_rate": 3.469799287188145e-05,
+      "loss": 2.6496,
+      "step": 4450
+    },
+    {
+      "epoch": 1.139266949420776,
+      "grad_norm": 1.3255618810653687,
+      "learning_rate": 3.446820483961734e-05,
+      "loss": 2.5315,
+      "step": 4500
+    },
+    {
+      "epoch": 1.1519275811863012,
+      "grad_norm": 1.2192399501800537,
+      "learning_rate": 3.423372725567436e-05,
+      "loss": 2.5409,
+      "step": 4550
+    },
+    {
+      "epoch": 1.1645882129518264,
+      "grad_norm": 1.2533234357833862,
+      "learning_rate": 3.399924967173139e-05,
+      "loss": 2.6457,
+      "step": 4600
+    },
+    {
+      "epoch": 1.1645882129518264,
+      "eval_loss": 2.462162733078003,
+      "eval_runtime": 40.0542,
+      "eval_samples_per_second": 43.841,
+      "eval_steps_per_second": 43.841,
+      "step": 4600
+    },
+    {
+      "epoch": 1.1772488447173515,
+      "grad_norm": 1.8414678573608398,
+      "learning_rate": 3.376477208778841e-05,
+      "loss": 2.5658,
+      "step": 4650
+    },
+    {
+      "epoch": 1.1899094764828764,
+      "grad_norm": 1.568259596824646,
+      "learning_rate": 3.3530294503845436e-05,
+      "loss": 2.5771,
+      "step": 4700
+    },
+    {
+      "epoch": 1.2025701082484015,
+      "grad_norm": 1.3547483682632446,
+      "learning_rate": 3.3295816919902464e-05,
+      "loss": 2.6525,
+      "step": 4750
+    },
+    {
+      "epoch": 1.2152307400139266,
+      "grad_norm": 1.1655386686325073,
+      "learning_rate": 3.3061339335959486e-05,
+      "loss": 2.6421,
+      "step": 4800
+    },
+    {
+      "epoch": 1.2152307400139266,
+      "eval_loss": 2.461489200592041,
+      "eval_runtime": 39.9962,
+      "eval_samples_per_second": 43.904,
+      "eval_steps_per_second": 43.904,
+      "step": 4800
+    },
+    {
+      "epoch": 1.2278913717794517,
+      "grad_norm": 1.798033595085144,
+      "learning_rate": 3.282686175201651e-05,
+      "loss": 2.6091,
+      "step": 4850
+    },
+    {
+      "epoch": 1.2405520035449769,
+      "grad_norm": 3.2964117527008057,
+      "learning_rate": 3.2592384168073535e-05,
+      "loss": 2.5997,
+      "step": 4900
+    },
+    {
+      "epoch": 1.253212635310502,
+      "grad_norm": 1.0457675457000732,
+      "learning_rate": 3.2357906584130557e-05,
+      "loss": 2.6144,
+      "step": 4950
+    },
+    {
+      "epoch": 1.265873267076027,
+      "grad_norm": 0.9728056192398071,
+      "learning_rate": 3.2123429000187585e-05,
+      "loss": 2.5712,
+      "step": 5000
+    },
+    {
+      "epoch": 1.265873267076027,
+      "eval_loss": 2.460186719894409,
+      "eval_runtime": 39.8386,
+      "eval_samples_per_second": 44.078,
+      "eval_steps_per_second": 44.078,
+      "step": 5000
+    },
+    {
+      "epoch": 1.2785338988415522,
+      "grad_norm": 1.2350194454193115,
+      "learning_rate": 3.188895141624461e-05,
+      "loss": 2.5448,
+      "step": 5050
+    },
+    {
+      "epoch": 1.2911945306070773,
+      "grad_norm": 1.4210622310638428,
+      "learning_rate": 3.1654473832301634e-05,
+      "loss": 2.6031,
+      "step": 5100
+    },
+    {
+      "epoch": 1.3038551623726025,
+      "grad_norm": 2.226473093032837,
+      "learning_rate": 3.1419996248358656e-05,
+      "loss": 2.6597,
+      "step": 5150
+    },
+    {
+      "epoch": 1.3165157941381276,
+      "grad_norm": 2.4525105953216553,
+      "learning_rate": 3.1185518664415684e-05,
+      "loss": 2.596,
+      "step": 5200
+    },
+    {
+      "epoch": 1.3165157941381276,
+      "eval_loss": 2.454537868499756,
+      "eval_runtime": 39.805,
+      "eval_samples_per_second": 44.115,
+      "eval_steps_per_second": 44.115,
+      "step": 5200
+    },
+    {
+      "epoch": 1.3291764259036527,
+      "grad_norm": 1.265309453010559,
+      "learning_rate": 3.095104108047271e-05,
+      "loss": 2.5559,
+      "step": 5250
+    },
+    {
+      "epoch": 1.3418370576691778,
+      "grad_norm": 2.1364307403564453,
+      "learning_rate": 3.071656349652973e-05,
+      "loss": 2.5859,
+      "step": 5300
+    },
+    {
+      "epoch": 1.3544976894347027,
+      "grad_norm": 1.5945920944213867,
+      "learning_rate": 3.048208591258676e-05,
+      "loss": 2.5778,
+      "step": 5350
+    },
+    {
+      "epoch": 1.3671583212002278,
+      "grad_norm": 1.2479759454727173,
+      "learning_rate": 3.0247608328643783e-05,
+      "loss": 2.6846,
+      "step": 5400
+    },
+    {
+      "epoch": 1.3671583212002278,
+      "eval_loss": 2.4547293186187744,
+      "eval_runtime": 39.7806,
+      "eval_samples_per_second": 44.142,
+      "eval_steps_per_second": 44.142,
+      "step": 5400
+    },
+    {
+      "epoch": 1.379818952965753,
+      "grad_norm": 1.4845050573349,
+      "learning_rate": 3.0013130744700808e-05,
+      "loss": 2.5661,
+      "step": 5450
+    },
+    {
+      "epoch": 1.392479584731278,
+      "grad_norm": 1.5581985712051392,
+      "learning_rate": 2.9778653160757836e-05,
+      "loss": 2.5441,
+      "step": 5500
+    },
+    {
+      "epoch": 1.4051402164968032,
+      "grad_norm": 3.1663737297058105,
+      "learning_rate": 2.9544175576814857e-05,
+      "loss": 2.5044,
+      "step": 5550
+    },
+    {
+      "epoch": 1.4178008482623283,
+      "grad_norm": 1.2454484701156616,
+      "learning_rate": 2.9309697992871882e-05,
+      "loss": 2.5747,
+      "step": 5600
+    },
+    {
+      "epoch": 1.4178008482623283,
+      "eval_loss": 2.4544529914855957,
+      "eval_runtime": 39.9287,
+      "eval_samples_per_second": 43.978,
+      "eval_steps_per_second": 43.978,
+      "step": 5600
+    },
+    {
+      "epoch": 1.4304614800278534,
+      "grad_norm": 1.662784457206726,
+      "learning_rate": 2.907522040892891e-05,
+      "loss": 2.6064,
+      "step": 5650
+    },
+    {
+      "epoch": 1.4431221117933786,
+      "grad_norm": 1.618458867073059,
+      "learning_rate": 2.8840742824985935e-05,
+      "loss": 2.5191,
+      "step": 5700
+    },
+    {
+      "epoch": 1.4557827435589035,
+      "grad_norm": 1.3003348112106323,
+      "learning_rate": 2.8606265241042956e-05,
+      "loss": 2.5339,
+      "step": 5750
+    },
+    {
+      "epoch": 1.4684433753244286,
+      "grad_norm": 1.1443992853164673,
+      "learning_rate": 2.8371787657099984e-05,
+      "loss": 2.5914,
+      "step": 5800
+    },
+    {
+      "epoch": 1.4684433753244286,
+      "eval_loss": 2.453752279281616,
+      "eval_runtime": 39.8234,
+      "eval_samples_per_second": 44.095,
+      "eval_steps_per_second": 44.095,
+      "step": 5800
+    },
+    {
+      "epoch": 1.4811040070899537,
+      "grad_norm": 1.2574009895324707,
+      "learning_rate": 2.813731007315701e-05,
+      "loss": 2.6109,
+      "step": 5850
+    },
+    {
+      "epoch": 1.4937646388554788,
+      "grad_norm": 1.002815842628479,
+      "learning_rate": 2.790283248921403e-05,
+      "loss": 2.6075,
+      "step": 5900
+    },
+    {
+      "epoch": 1.506425270621004,
+      "grad_norm": 1.306024432182312,
+      "learning_rate": 2.766835490527106e-05,
+      "loss": 2.5733,
+      "step": 5950
+    },
+    {
+      "epoch": 1.519085902386529,
+      "grad_norm": 2.5023701190948486,
+      "learning_rate": 2.7433877321328083e-05,
+      "loss": 2.6274,
+      "step": 6000
+    },
+    {
+      "epoch": 1.519085902386529,
+      "eval_loss": 2.449617862701416,
+      "eval_runtime": 39.9401,
+      "eval_samples_per_second": 43.966,
+      "eval_steps_per_second": 43.966,
+      "step": 6000
+    },
+    {
+      "epoch": 1.5317465341520542,
+      "grad_norm": 1.9410326480865479,
+      "learning_rate": 2.7199399737385105e-05,
+      "loss": 2.5312,
+      "step": 6050
+    },
+    {
+      "epoch": 1.5444071659175793,
+      "grad_norm": 1.9793561697006226,
+      "learning_rate": 2.6964922153442136e-05,
+      "loss": 2.5759,
+      "step": 6100
+    },
+    {
+      "epoch": 1.5570677976831044,
+      "grad_norm": 1.290531873703003,
+      "learning_rate": 2.6730444569499157e-05,
+      "loss": 2.5817,
+      "step": 6150
+    },
+    {
+      "epoch": 1.5697284294486296,
+      "grad_norm": 2.11389422416687,
+      "learning_rate": 2.6495966985556182e-05,
+      "loss": 2.6287,
+      "step": 6200
+    },
+    {
+      "epoch": 1.5697284294486296,
+      "eval_loss": 2.4490554332733154,
+      "eval_runtime": 39.7474,
+      "eval_samples_per_second": 44.179,
+      "eval_steps_per_second": 44.179,
+      "step": 6200
+    },
+    {
+      "epoch": 1.5823890612141547,
+      "grad_norm": 1.6492938995361328,
+      "learning_rate": 2.626148940161321e-05,
+      "loss": 2.631,
+      "step": 6250
+    },
+    {
+      "epoch": 1.5950496929796798,
+      "grad_norm": 1.3233673572540283,
+      "learning_rate": 2.6027011817670232e-05,
+      "loss": 2.5654,
+      "step": 6300
+    },
+    {
+      "epoch": 1.607710324745205,
+      "grad_norm": 1.688264012336731,
+      "learning_rate": 2.5792534233727257e-05,
+      "loss": 2.6096,
+      "step": 6350
+    },
+    {
+      "epoch": 1.62037095651073,
+      "grad_norm": 2.064823865890503,
+      "learning_rate": 2.5558056649784285e-05,
+      "loss": 2.6275,
+      "step": 6400
+    },
+    {
+      "epoch": 1.62037095651073,
+      "eval_loss": 2.4488983154296875,
+      "eval_runtime": 39.6847,
+      "eval_samples_per_second": 44.249,
+      "eval_steps_per_second": 44.249,
+      "step": 6400
+    },
+    {
+      "epoch": 1.633031588276255,
+      "grad_norm": 1.5599696636199951,
+      "learning_rate": 2.5323579065841306e-05,
+      "loss": 2.6334,
+      "step": 6450
+    },
+    {
+      "epoch": 1.64569222004178,
+      "grad_norm": 1.3142633438110352,
+      "learning_rate": 2.508910148189833e-05,
+      "loss": 2.5496,
+      "step": 6500
+    },
+    {
+      "epoch": 1.6583528518073052,
+      "grad_norm": 1.474135160446167,
+      "learning_rate": 2.4854623897955356e-05,
+      "loss": 2.5628,
+      "step": 6550
+    },
+    {
+      "epoch": 1.6710134835728303,
+      "grad_norm": 1.3737610578536987,
+      "learning_rate": 2.4620146314012384e-05,
+      "loss": 2.5345,
+      "step": 6600
+    },
+    {
+      "epoch": 1.6710134835728303,
+      "eval_loss": 2.447559356689453,
+      "eval_runtime": 39.7021,
+      "eval_samples_per_second": 44.229,
+      "eval_steps_per_second": 44.229,
+      "step": 6600
+    },
+    {
+      "epoch": 1.6836741153383554,
+      "grad_norm": 1.2432060241699219,
+      "learning_rate": 2.4385668730069405e-05,
+      "loss": 2.5977,
+      "step": 6650
+    },
+    {
+      "epoch": 1.6963347471038803,
+      "grad_norm": 1.465063452720642,
+      "learning_rate": 2.415119114612643e-05,
+      "loss": 2.6118,
+      "step": 6700
+    },
+    {
+      "epoch": 1.7089953788694054,
+      "grad_norm": 1.5186200141906738,
+      "learning_rate": 2.3916713562183458e-05,
+      "loss": 2.6126,
+      "step": 6750
+    },
+    {
+      "epoch": 1.7216560106349306,
+      "grad_norm": 1.6869078874588013,
+      "learning_rate": 2.368223597824048e-05,
+      "loss": 2.576,
+      "step": 6800
+    },
+    {
+      "epoch": 1.7216560106349306,
+      "eval_loss": 2.4459502696990967,
+      "eval_runtime": 39.7653,
+      "eval_samples_per_second": 44.159,
+      "eval_steps_per_second": 44.159,
+      "step": 6800
+    },
+    {
+      "epoch": 1.7343166424004557,
+      "grad_norm": 1.2578104734420776,
+      "learning_rate": 2.3447758394297507e-05,
+      "loss": 2.6178,
+      "step": 6850
+    },
+    {
+      "epoch": 1.7469772741659808,
+      "grad_norm": 1.7597213983535767,
+      "learning_rate": 2.3213280810354532e-05,
+      "loss": 2.6358,
+      "step": 6900
+    },
+    {
+      "epoch": 1.759637905931506,
+      "grad_norm": 2.144465923309326,
+      "learning_rate": 2.2978803226411554e-05,
+      "loss": 2.5597,
+      "step": 6950
+    },
+    {
+      "epoch": 1.772298537697031,
+      "grad_norm": 1.1808464527130127,
+      "learning_rate": 2.2744325642468582e-05,
+      "loss": 2.6269,
+      "step": 7000
+    },
+    {
+      "epoch": 1.772298537697031,
+      "eval_loss": 2.4444611072540283,
+      "eval_runtime": 40.0709,
+      "eval_samples_per_second": 43.822,
+      "eval_steps_per_second": 43.822,
+      "step": 7000
+    },
+    {
+      "epoch": 1.7849591694625562,
+      "grad_norm": 1.4550806283950806,
+      "learning_rate": 2.2509848058525606e-05,
+      "loss": 2.6206,
+      "step": 7050
+    },
+    {
+      "epoch": 1.7976198012280813,
+      "grad_norm": 1.2635902166366577,
+      "learning_rate": 2.227537047458263e-05,
+      "loss": 2.5722,
+      "step": 7100
+    },
+    {
+      "epoch": 1.8102804329936064,
+      "grad_norm": 1.3835856914520264,
+      "learning_rate": 2.2040892890639656e-05,
+      "loss": 2.535,
+      "step": 7150
+    },
+    {
+      "epoch": 1.8229410647591315,
+      "grad_norm": 1.735004186630249,
+      "learning_rate": 2.180641530669668e-05,
+      "loss": 2.6086,
+      "step": 7200
+    },
+    {
+      "epoch": 1.8229410647591315,
+      "eval_loss": 2.443899154663086,
+      "eval_runtime": 40.91,
+      "eval_samples_per_second": 42.924,
+      "eval_steps_per_second": 42.924,
+      "step": 7200
+    },
+    {
+      "epoch": 1.8356016965246567,
+      "grad_norm": 1.263051986694336,
+      "learning_rate": 2.1571937722753706e-05,
+      "loss": 2.5544,
+      "step": 7250
+    },
+    {
+      "epoch": 1.8482623282901818,
+      "grad_norm": 1.0899442434310913,
+      "learning_rate": 2.133746013881073e-05,
+      "loss": 2.5603,
+      "step": 7300
+    },
+    {
+      "epoch": 1.860922960055707,
+      "grad_norm": 3.038811206817627,
+      "learning_rate": 2.1102982554867755e-05,
+      "loss": 2.5688,
+      "step": 7350
+    },
+    {
+      "epoch": 1.873583591821232,
+      "grad_norm": 1.6385984420776367,
+      "learning_rate": 2.086850497092478e-05,
+      "loss": 2.6006,
+      "step": 7400
+    },
+    {
+      "epoch": 1.873583591821232,
+      "eval_loss": 2.443300724029541,
+      "eval_runtime": 40.6649,
+      "eval_samples_per_second": 43.182,
+      "eval_steps_per_second": 43.182,
+      "step": 7400
+    },
+    {
+      "epoch": 1.8862442235867571,
+      "grad_norm": 1.2857129573822021,
+      "learning_rate": 2.0634027386981805e-05,
+      "loss": 2.5563,
+      "step": 7450
+    },
+    {
+      "epoch": 1.898904855352282,
+      "grad_norm": 1.0289497375488281,
+      "learning_rate": 2.0399549803038833e-05,
+      "loss": 2.5671,
+      "step": 7500
+    },
+    {
+      "epoch": 1.9115654871178072,
+      "grad_norm": 1.5041025876998901,
+      "learning_rate": 2.0165072219095854e-05,
+      "loss": 2.5689,
+      "step": 7550
+    },
+    {
+      "epoch": 1.9242261188833323,
+      "grad_norm": 1.6611964702606201,
+      "learning_rate": 1.993528418683174e-05,
+      "loss": 2.5801,
+      "step": 7600
+    },
+    {
+      "epoch": 1.9242261188833323,
+      "eval_loss": 2.443532943725586,
+      "eval_runtime": 39.931,
+      "eval_samples_per_second": 43.976,
+      "eval_steps_per_second": 43.976,
+      "step": 7600
+    },
+    {
+      "epoch": 1.9368867506488574,
+      "grad_norm": 1.521170735359192,
+      "learning_rate": 1.9700806602888767e-05,
+      "loss": 2.5969,
+      "step": 7650
+    },
+    {
+      "epoch": 1.9495473824143825,
+      "grad_norm": 1.3700034618377686,
+      "learning_rate": 1.946632901894579e-05,
+      "loss": 2.6306,
+      "step": 7700
+    },
+    {
+      "epoch": 1.9622080141799074,
+      "grad_norm": 2.311443328857422,
+      "learning_rate": 1.9231851435002814e-05,
+      "loss": 2.5608,
+      "step": 7750
+    },
+    {
+      "epoch": 1.9748686459454325,
+      "grad_norm": 1.6699820756912231,
+      "learning_rate": 1.8997373851059842e-05,
+      "loss": 2.5113,
+      "step": 7800
+    },
+    {
+      "epoch": 1.9748686459454325,
+      "eval_loss": 2.4421675205230713,
+      "eval_runtime": 40.1783,
+      "eval_samples_per_second": 43.705,
+      "eval_steps_per_second": 43.705,
+      "step": 7800
+    },
+    {
+      "epoch": 1.9875292777109577,
+      "grad_norm": 1.2560683488845825,
+      "learning_rate": 1.8762896267116863e-05,
+      "loss": 2.545,
+      "step": 7850
+    },
+    {
+      "epoch": 2.0,
+      "grad_norm": 2.176563262939453,
+      "learning_rate": 1.852841868317389e-05,
+      "loss": 2.5752,
+      "step": 7900
+    },
+    {
+      "epoch": 2.012660631765525,
+      "grad_norm": 1.2551178932189941,
+      "learning_rate": 1.8293941099230916e-05,
+      "loss": 2.5215,
+      "step": 7950
+    },
+    {
+      "epoch": 2.0253212635310502,
+      "grad_norm": 1.5646872520446777,
+      "learning_rate": 1.8059463515287937e-05,
+      "loss": 2.5838,
+      "step": 8000
+    },
+    {
+      "epoch": 2.0253212635310502,
+      "eval_loss": 2.441195249557495,
+      "eval_runtime": 39.701,
+      "eval_samples_per_second": 44.231,
+      "eval_steps_per_second": 44.231,
+      "step": 8000
+    },
+    {
+      "epoch": 2.0379818952965754,
+      "grad_norm": 1.4227900505065918,
+      "learning_rate": 1.7824985931344966e-05,
+      "loss": 2.5597,
+      "step": 8050
+    },
+    {
+      "epoch": 2.0506425270621005,
+      "grad_norm": 1.3013832569122314,
+      "learning_rate": 1.759050834740199e-05,
+      "loss": 2.7641,
+      "step": 8100
+    },
+    {
+      "epoch": 2.0633031588276256,
+      "grad_norm": 1.1282143592834473,
+      "learning_rate": 1.7356030763459015e-05,
+      "loss": 2.5875,
+      "step": 8150
+    },
+    {
+      "epoch": 2.0759637905931507,
+      "grad_norm": 2.079760789871216,
+      "learning_rate": 1.712155317951604e-05,
+      "loss": 2.4861,
+      "step": 8200
+    },
+    {
+      "epoch": 2.0759637905931507,
+      "eval_loss": 2.440812826156616,
+      "eval_runtime": 40.0764,
+      "eval_samples_per_second": 43.816,
+      "eval_steps_per_second": 43.816,
+      "step": 8200
+    },
+    {
+      "epoch": 2.088624422358676,
+      "grad_norm": 1.0884991884231567,
+      "learning_rate": 1.6887075595573065e-05,
+      "loss": 2.5941,
+      "step": 8250
+    },
+    {
+      "epoch": 2.101285054124201,
+      "grad_norm": 1.9202015399932861,
+      "learning_rate": 1.665259801163009e-05,
+      "loss": 2.5929,
+      "step": 8300
+    },
+    {
+      "epoch": 2.113945685889726,
+      "grad_norm": 1.5925830602645874,
+      "learning_rate": 1.6418120427687114e-05,
+      "loss": 2.5046,
+      "step": 8350
+    },
+    {
+      "epoch": 2.126606317655251,
+      "grad_norm": 1.5219184160232544,
+      "learning_rate": 1.618364284374414e-05,
+      "loss": 2.5628,
+      "step": 8400
+    },
+    {
+      "epoch": 2.126606317655251,
+      "eval_loss": 2.4396440982818604,
+      "eval_runtime": 40.0053,
+      "eval_samples_per_second": 43.894,
+      "eval_steps_per_second": 43.894,
+      "step": 8400
+    },
+    {
+      "epoch": 2.139266949420776,
+      "grad_norm": 1.4882445335388184,
+      "learning_rate": 1.5949165259801164e-05,
+      "loss": 2.6268,
+      "step": 8450
+    },
+    {
+      "epoch": 2.151927581186301,
+      "grad_norm": 1.3513301610946655,
+      "learning_rate": 1.571468767585819e-05,
+      "loss": 2.5277,
+      "step": 8500
+    },
+    {
+      "epoch": 2.164588212951826,
+      "grad_norm": 1.690974473953247,
+      "learning_rate": 1.5480210091915216e-05,
+      "loss": 2.5631,
+      "step": 8550
+    },
+    {
+      "epoch": 2.1772488447173513,
+      "grad_norm": 1.5311528444290161,
+      "learning_rate": 1.5245732507972238e-05,
+      "loss": 2.5454,
+      "step": 8600
+    },
+    {
+      "epoch": 2.1772488447173513,
+      "eval_loss": 2.4388718605041504,
+      "eval_runtime": 40.0289,
+      "eval_samples_per_second": 43.868,
+      "eval_steps_per_second": 43.868,
+      "step": 8600
+    },
+    {
+      "epoch": 2.1899094764828764,
+      "grad_norm": 2.1171281337738037,
+      "learning_rate": 1.5011254924029264e-05,
+      "loss": 2.6112,
+      "step": 8650
+    },
+    {
+      "epoch": 2.2025701082484015,
+      "grad_norm": 1.9706814289093018,
+      "learning_rate": 1.4776777340086289e-05,
+      "loss": 2.588,
+      "step": 8700
+    },
+    {
+      "epoch": 2.2152307400139266,
+      "grad_norm": 1.8991297483444214,
+      "learning_rate": 1.4542299756143312e-05,
+      "loss": 2.5655,
+      "step": 8750
+    },
+    {
+      "epoch": 2.2278913717794517,
+      "grad_norm": 1.5568820238113403,
+      "learning_rate": 1.4307822172200339e-05,
+      "loss": 2.5312,
+      "step": 8800
+    },
+    {
+      "epoch": 2.2278913717794517,
+      "eval_loss": 2.438715696334839,
+      "eval_runtime": 40.1748,
+      "eval_samples_per_second": 43.709,
+      "eval_steps_per_second": 43.709,
+      "step": 8800
+    },
+    {
+      "epoch": 2.240552003544977,
+      "grad_norm": 1.277051329612732,
+      "learning_rate": 1.4073344588257365e-05,
+      "loss": 2.5818,
+      "step": 8850
+    },
+    {
+      "epoch": 2.253212635310502,
+      "grad_norm": 1.8890128135681152,
+      "learning_rate": 1.3838867004314388e-05,
+      "loss": 2.5211,
+      "step": 8900
+    },
+    {
+      "epoch": 2.265873267076027,
+      "grad_norm": 1.8824830055236816,
+      "learning_rate": 1.3604389420371413e-05,
+      "loss": 2.53,
+      "step": 8950
+    },
+    {
+      "epoch": 2.278533898841552,
+      "grad_norm": 1.239490032196045,
+      "learning_rate": 1.336991183642844e-05,
+      "loss": 2.5889,
+      "step": 9000
+    },
+    {
+      "epoch": 2.278533898841552,
+      "eval_loss": 2.437577962875366,
+      "eval_runtime": 40.1654,
+      "eval_samples_per_second": 43.719,
+      "eval_steps_per_second": 43.719,
+      "step": 9000
+    },
+    {
+      "epoch": 2.2911945306070773,
+      "grad_norm": 1.7253328561782837,
+      "learning_rate": 1.3135434252485462e-05,
+      "loss": 2.5426,
+      "step": 9050
+    },
+    {
+      "epoch": 2.3038551623726025,
+      "grad_norm": 1.6971838474273682,
+      "learning_rate": 1.2900956668542489e-05,
+      "loss": 2.4953,
+      "step": 9100
+    },
+    {
+      "epoch": 2.3165157941381276,
+      "grad_norm": 1.4906270503997803,
+      "learning_rate": 1.2666479084599514e-05,
+      "loss": 2.606,
+      "step": 9150
+    },
+    {
+      "epoch": 2.3291764259036527,
+      "grad_norm": 1.658526062965393,
+      "learning_rate": 1.2432001500656538e-05,
+      "loss": 2.5483,
+      "step": 9200
+    },
+    {
+      "epoch": 2.3291764259036527,
+      "eval_loss": 2.437896490097046,
+      "eval_runtime": 40.7008,
+      "eval_samples_per_second": 43.144,
+      "eval_steps_per_second": 43.144,
+      "step": 9200
+    },
+    {
+      "epoch": 2.341837057669178,
+      "grad_norm": 1.0781177282333374,
+      "learning_rate": 1.2197523916713563e-05,
+      "loss": 2.5449,
+      "step": 9250
+    },
+    {
+      "epoch": 2.354497689434703,
+      "grad_norm": 2.1414873600006104,
+      "learning_rate": 1.1963046332770588e-05,
+      "loss": 2.5303,
+      "step": 9300
+    },
+    {
+      "epoch": 2.367158321200228,
+      "grad_norm": 2.063297986984253,
+      "learning_rate": 1.1728568748827613e-05,
+      "loss": 2.5837,
+      "step": 9350
+    },
+    {
+      "epoch": 2.3798189529657527,
+      "grad_norm": 1.2153489589691162,
+      "learning_rate": 1.1494091164884637e-05,
+      "loss": 2.6384,
+      "step": 9400
+    },
+    {
+      "epoch": 2.3798189529657527,
+      "eval_loss": 2.4365696907043457,
+      "eval_runtime": 40.2398,
+      "eval_samples_per_second": 43.638,
+      "eval_steps_per_second": 43.638,
+      "step": 9400
+    },
+    {
+      "epoch": 2.3924795847312783,
+      "grad_norm": 1.2976094484329224,
+      "learning_rate": 1.1259613580941662e-05,
+      "loss": 2.572,
+      "step": 9450
+    },
+    {
+      "epoch": 2.405140216496803,
+      "grad_norm": 1.2775920629501343,
+      "learning_rate": 1.1025135996998689e-05,
+      "loss": 2.6083,
+      "step": 9500
+    },
+    {
+      "epoch": 2.417800848262328,
+      "grad_norm": 1.358311653137207,
+      "learning_rate": 1.0790658413055713e-05,
+      "loss": 2.5206,
+      "step": 9550
+    },
+    {
+      "epoch": 2.4304614800278532,
+      "grad_norm": 1.3438369035720825,
+      "learning_rate": 1.0556180829112736e-05,
+      "loss": 2.4967,
+      "step": 9600
+    },
+    {
+      "epoch": 2.4304614800278532,
+      "eval_loss": 2.4359662532806396,
+      "eval_runtime": 40.0802,
+      "eval_samples_per_second": 43.812,
+      "eval_steps_per_second": 43.812,
+      "step": 9600
+    },
+    {
+      "epoch": 2.4431221117933783,
+      "grad_norm": 1.2618831396102905,
+      "learning_rate": 1.0321703245169763e-05,
+      "loss": 2.6169,
+      "step": 9650
+    },
+    {
+      "epoch": 2.4557827435589035,
+      "grad_norm": 1.3764727115631104,
+      "learning_rate": 1.0087225661226788e-05,
+      "loss": 2.5444,
+      "step": 9700
+    },
+    {
+      "epoch": 2.4684433753244286,
+      "grad_norm": 1.604864478111267,
+      "learning_rate": 9.852748077283812e-06,
+      "loss": 2.5343,
+      "step": 9750
+    },
+    {
+      "epoch": 2.4811040070899537,
+      "grad_norm": 1.390496850013733,
+      "learning_rate": 9.618270493340837e-06,
+      "loss": 2.5051,
+      "step": 9800
+    },
+    {
+      "epoch": 2.4811040070899537,
+      "eval_loss": 2.4353232383728027,
+      "eval_runtime": 40.1607,
+      "eval_samples_per_second": 43.724,
+      "eval_steps_per_second": 43.724,
+      "step": 9800
+    },
+    {
+      "epoch": 2.493764638855479,
+      "grad_norm": 2.1982169151306152,
+      "learning_rate": 9.383792909397862e-06,
+      "loss": 2.5036,
+      "step": 9850
+    },
+    {
+      "epoch": 2.506425270621004,
+      "grad_norm": 1.3033822774887085,
+      "learning_rate": 9.149315325454887e-06,
+      "loss": 2.5205,
+      "step": 9900
+    },
+    {
+      "epoch": 2.519085902386529,
+      "grad_norm": 1.682586431503296,
+      "learning_rate": 8.919527293190772e-06,
+      "loss": 2.6083,
+      "step": 9950
+    },
+    {
+      "epoch": 2.531746534152054,
+      "grad_norm": 3.184382200241089,
+      "learning_rate": 8.685049709247797e-06,
+      "loss": 2.5314,
+      "step": 10000
+    },
+    {
+      "epoch": 2.531746534152054,
+      "eval_loss": 2.434755802154541,
+      "eval_runtime": 40.2877,
+      "eval_samples_per_second": 43.587,
+      "eval_steps_per_second": 43.587,
+      "step": 10000
+    },
+    {
+      "epoch": 2.5444071659175793,
+      "grad_norm": 2.0026867389678955,
+      "learning_rate": 8.450572125304821e-06,
+      "loss": 2.5109,
+      "step": 10050
+    },
+    {
+      "epoch": 2.5570677976831044,
+      "grad_norm": 1.3833885192871094,
+      "learning_rate": 8.216094541361846e-06,
+      "loss": 2.5362,
+      "step": 10100
+    },
+    {
+      "epoch": 2.5697284294486296,
+      "grad_norm": 2.157984495162964,
+      "learning_rate": 7.981616957418871e-06,
+      "loss": 2.5423,
+      "step": 10150
+    },
+    {
+      "epoch": 2.5823890612141547,
+      "grad_norm": 1.682053565979004,
+      "learning_rate": 7.747139373475897e-06,
+      "loss": 2.5133,
+      "step": 10200
+    },
+    {
+      "epoch": 2.5823890612141547,
+      "eval_loss": 2.435208559036255,
+      "eval_runtime": 40.4768,
+      "eval_samples_per_second": 43.383,
+      "eval_steps_per_second": 43.383,
+      "step": 10200
+    },
+    {
+      "epoch": 2.59504969297968,
+      "grad_norm": 1.9720139503479004,
+      "learning_rate": 7.512661789532921e-06,
+      "loss": 2.6372,
+      "step": 10250
+    },
+    {
+      "epoch": 2.607710324745205,
+      "grad_norm": 1.6906607151031494,
+      "learning_rate": 7.278184205589945e-06,
+      "loss": 2.5505,
+      "step": 10300
+    },
+    {
+      "epoch": 2.62037095651073,
+      "grad_norm": 1.484045147895813,
+      "learning_rate": 7.043706621646972e-06,
+      "loss": 2.5095,
+      "step": 10350
+    },
+    {
+      "epoch": 2.633031588276255,
+      "grad_norm": 1.6676850318908691,
+      "learning_rate": 6.8092290377039955e-06,
+      "loss": 2.6487,
+      "step": 10400
+    },
+    {
+      "epoch": 2.633031588276255,
+      "eval_loss": 2.4344091415405273,
+      "eval_runtime": 39.869,
+      "eval_samples_per_second": 44.044,
+      "eval_steps_per_second": 44.044,
+      "step": 10400
+    },
+    {
+      "epoch": 2.64569222004178,
+      "grad_norm": 1.5012388229370117,
+      "learning_rate": 6.57475145376102e-06,
+      "loss": 2.5756,
+      "step": 10450
+    },
+    {
+      "epoch": 2.6583528518073054,
+      "grad_norm": 1.043954849243164,
+      "learning_rate": 6.340273869818046e-06,
+      "loss": 2.5843,
+      "step": 10500
+    },
+    {
+      "epoch": 2.67101348357283,
+      "grad_norm": 1.0455141067504883,
+      "learning_rate": 6.105796285875071e-06,
+      "loss": 2.5248,
+      "step": 10550
+    },
+    {
+      "epoch": 2.6836741153383556,
+      "grad_norm": 1.39467453956604,
+      "learning_rate": 5.871318701932095e-06,
+      "loss": 2.5091,
+      "step": 10600
+    },
+    {
+      "epoch": 2.6836741153383556,
+      "eval_loss": 2.4331610202789307,
+      "eval_runtime": 39.923,
+      "eval_samples_per_second": 43.985,
+      "eval_steps_per_second": 43.985,
+      "step": 10600
+    },
+    {
+      "epoch": 2.6963347471038803,
+      "grad_norm": 1.1417715549468994,
+      "learning_rate": 5.63684111798912e-06,
+      "loss": 2.5853,
+      "step": 10650
+    },
+    {
+      "epoch": 2.7089953788694054,
+      "grad_norm": 1.133244514465332,
+      "learning_rate": 5.402363534046146e-06,
+      "loss": 2.5457,
+      "step": 10700
+    },
+    {
+      "epoch": 2.7216560106349306,
+      "grad_norm": 1.2331452369689941,
+      "learning_rate": 5.1678859501031705e-06,
+      "loss": 2.5576,
+      "step": 10750
+    },
+    {
+      "epoch": 2.7343166424004557,
+      "grad_norm": 1.7164263725280762,
+      "learning_rate": 4.933408366160195e-06,
+      "loss": 2.5471,
+      "step": 10800
+    },
+    {
+      "epoch": 2.7343166424004557,
+      "eval_loss": 2.4340403079986572,
+      "eval_runtime": 40.3849,
+      "eval_samples_per_second": 43.482,
+      "eval_steps_per_second": 43.482,
+      "step": 10800
+    },
+    {
+      "epoch": 2.746977274165981,
+      "grad_norm": 1.3680106401443481,
+      "learning_rate": 4.69893078221722e-06,
+      "loss": 2.5562,
+      "step": 10850
+    },
+    {
+      "epoch": 2.759637905931506,
+      "grad_norm": 1.0978279113769531,
+      "learning_rate": 4.464453198274246e-06,
+      "loss": 2.5185,
+      "step": 10900
+    },
+    {
+      "epoch": 2.772298537697031,
+      "grad_norm": 1.2212647199630737,
+      "learning_rate": 4.2299756143312695e-06,
+      "loss": 2.6371,
+      "step": 10950
+    },
+    {
+      "epoch": 2.784959169462556,
+      "grad_norm": 1.6452165842056274,
+      "learning_rate": 3.995498030388295e-06,
+      "loss": 2.681,
+      "step": 11000
+    },
+    {
+      "epoch": 2.784959169462556,
+      "eval_loss": 2.4337143898010254,
+      "eval_runtime": 40.4235,
+      "eval_samples_per_second": 43.44,
+      "eval_steps_per_second": 43.44,
+      "step": 11000
+    },
+    {
+      "epoch": 2.7976198012280813,
+      "grad_norm": 1.7757978439331055,
+      "learning_rate": 3.7610204464453203e-06,
+      "loss": 2.5746,
+      "step": 11050
+    },
+    {
+      "epoch": 2.8102804329936064,
+      "grad_norm": 1.2373579740524292,
+      "learning_rate": 3.5265428625023455e-06,
+      "loss": 2.5412,
+      "step": 11100
+    },
+    {
+      "epoch": 2.8229410647591315,
+      "grad_norm": 1.1407558917999268,
+      "learning_rate": 3.29206527855937e-06,
+      "loss": 2.5973,
+      "step": 11150
+    },
+    {
+      "epoch": 2.8356016965246567,
+      "grad_norm": 2.399686813354492,
+      "learning_rate": 3.057587694616395e-06,
+      "loss": 2.5566,
+      "step": 11200
+    },
+    {
+      "epoch": 2.8356016965246567,
+      "eval_loss": 2.4338231086730957,
+      "eval_runtime": 40.4877,
+      "eval_samples_per_second": 43.371,
+      "eval_steps_per_second": 43.371,
+      "step": 11200
+    },
+    {
+      "epoch": 2.8482623282901818,
+      "grad_norm": 1.7053141593933105,
+      "learning_rate": 2.8231101106734197e-06,
+      "loss": 2.6224,
+      "step": 11250
+    },
+    {
+      "epoch": 2.860922960055707,
+      "grad_norm": 1.8215903043746948,
+      "learning_rate": 2.5886325267304445e-06,
+      "loss": 2.5108,
+      "step": 11300
+    },
+    {
+      "epoch": 2.873583591821232,
+      "grad_norm": 1.1648200750350952,
+      "learning_rate": 2.3541549427874697e-06,
+      "loss": 2.557,
+      "step": 11350
+    },
+    {
+      "epoch": 2.886244223586757,
+      "grad_norm": 1.5225868225097656,
+      "learning_rate": 2.1196773588444944e-06,
+      "loss": 2.6285,
+      "step": 11400
+    },
+    {
+      "epoch": 2.886244223586757,
+      "eval_loss": 2.4334514141082764,
+      "eval_runtime": 40.3985,
+      "eval_samples_per_second": 43.467,
+      "eval_steps_per_second": 43.467,
+      "step": 11400
+    },
+    {
+      "epoch": 2.8989048553522823,
+      "grad_norm": 1.4937622547149658,
+      "learning_rate": 1.8851997749015194e-06,
+      "loss": 2.481,
+      "step": 11450
+    },
+    {
+      "epoch": 2.911565487117807,
+      "grad_norm": 1.9169902801513672,
+      "learning_rate": 1.6507221909585446e-06,
+      "loss": 2.5412,
+      "step": 11500
+    },
+    {
+      "epoch": 2.9242261188833325,
+      "grad_norm": 1.6611114740371704,
+      "learning_rate": 1.4162446070155693e-06,
+      "loss": 2.5086,
+      "step": 11550
+    },
+    {
+      "epoch": 2.936886750648857,
+      "grad_norm": 1.3464007377624512,
+      "learning_rate": 1.1817670230725943e-06,
+      "loss": 2.6063,
+      "step": 11600
+    },
+    {
+      "epoch": 2.936886750648857,
+      "eval_loss": 2.4329476356506348,
+      "eval_runtime": 40.4334,
+      "eval_samples_per_second": 43.429,
+      "eval_steps_per_second": 43.429,
+      "step": 11600
+    },
+    {
+      "epoch": 2.9495473824143827,
+      "grad_norm": 1.453385829925537,
+      "learning_rate": 9.472894391296193e-07,
+      "loss": 2.5012,
+      "step": 11650
+    },
+    {
+      "epoch": 2.9622080141799074,
+      "grad_norm": 1.6921356916427612,
+      "learning_rate": 7.128118551866442e-07,
+      "loss": 2.5589,
+      "step": 11700
+    },
+    {
+      "epoch": 2.9748686459454325,
+      "grad_norm": 1.0562982559204102,
+      "learning_rate": 4.783342712436691e-07,
+      "loss": 2.6015,
+      "step": 11750
+    },
+    {
+      "epoch": 2.9875292777109577,
+      "grad_norm": 1.457960844039917,
+      "learning_rate": 2.4385668730069406e-07,
+      "loss": 2.6224,
+      "step": 11800
+    },
+    {
+      "epoch": 2.9875292777109577,
+      "eval_loss": 2.432849645614624,
+      "eval_runtime": 40.4518,
+      "eval_samples_per_second": 43.41,
+      "eval_steps_per_second": 43.41,
+      "step": 11800
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 11847,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6249293680214016.0,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-11847/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d46b7107c3dfd17ff0fb12b8cdcc256a49e8d4594358d4495904a5fcb069382
+size 5304

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,21 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d46b7107c3dfd17ff0fb12b8cdcc256a49e8d4594358d4495904a5fcb069382
+size 5304