Upload folder using huggingface_hub

README.md

@@ -0,0 +1,202 @@
+---
+base_model: gpt2-medium
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": {
+    "base_model_class": "GPT2LMHeadModel",
+    "parent_library": "transformers.models.gpt2.modeling_gpt2"
+  },
+  "base_model_name_or_path": "gpt2-medium",
+  "bias": "none",
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_attn"
+  ],
+  "task_type": null,
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc7aee4b9c42ebd62357b449167318942459034a053848582eb484db1a06992a
+size 25172088

checkpoint-3125/README.md

@@ -0,0 +1,202 @@
+---
+base_model: gpt2-medium
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

checkpoint-3125/adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": {
+    "base_model_class": "GPT2LMHeadModel",
+    "parent_library": "transformers.models.gpt2.modeling_gpt2"
+  },
+  "base_model_name_or_path": "gpt2-medium",
+  "bias": "none",
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_attn"
+  ],
+  "task_type": null,
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-3125/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc7aee4b9c42ebd62357b449167318942459034a053848582eb484db1a06992a
+size 25172088

checkpoint-3125/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fcc7cb6169aaa58f32fa131dc2e772496762aec1d5372a15c1fb827f0358f3cc
+size 50372538

checkpoint-3125/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5994634e90ce1b45d012a45568063be05fea876e791cd66b48a4efc924164b2
+size 14244

checkpoint-3125/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e42e57efa8b95889e87232fe6ea0805483ed6c8a2d35bb60230715b624ff5bda
+size 1064

checkpoint-3125/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-3125/tokenizer_config.json

@@ -0,0 +1,22 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-3125/trainer_state.json

@@ -0,0 +1,2217 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 3125,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0032,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 6.369426751592357e-06,
+      "loss": 22.4391,
+      "step": 10
+    },
+    {
+      "epoch": 0.0064,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 1.2738853503184714e-05,
+      "loss": 18.5406,
+      "step": 20
+    },
+    {
+      "epoch": 0.0096,
+      "grad_norm": 10.0,
+      "learning_rate": 1.910828025477707e-05,
+      "loss": 18.8755,
+      "step": 30
+    },
+    {
+      "epoch": 0.0128,
+      "grad_norm": 9.999998092651367,
+      "learning_rate": 2.5477707006369428e-05,
+      "loss": 18.0102,
+      "step": 40
+    },
+    {
+      "epoch": 0.016,
+      "grad_norm": 10.0,
+      "learning_rate": 3.184713375796178e-05,
+      "loss": 15.1105,
+      "step": 50
+    },
+    {
+      "epoch": 0.0192,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 3.821656050955414e-05,
+      "loss": 14.5174,
+      "step": 60
+    },
+    {
+      "epoch": 0.0224,
+      "grad_norm": 10.0,
+      "learning_rate": 4.45859872611465e-05,
+      "loss": 12.4194,
+      "step": 70
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 5.0955414012738855e-05,
+      "loss": 10.4174,
+      "step": 80
+    },
+    {
+      "epoch": 0.0288,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 5.732484076433121e-05,
+      "loss": 9.3693,
+      "step": 90
+    },
+    {
+      "epoch": 0.032,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 6.369426751592356e-05,
+      "loss": 8.6031,
+      "step": 100
+    },
+    {
+      "epoch": 0.0352,
+      "grad_norm": 10.0,
+      "learning_rate": 7.006369426751592e-05,
+      "loss": 6.5878,
+      "step": 110
+    },
+    {
+      "epoch": 0.0384,
+      "grad_norm": 10.0,
+      "learning_rate": 7.643312101910829e-05,
+      "loss": 6.7467,
+      "step": 120
+    },
+    {
+      "epoch": 0.0416,
+      "grad_norm": 10.0,
+      "learning_rate": 8.280254777070065e-05,
+      "loss": 5.6174,
+      "step": 130
+    },
+    {
+      "epoch": 0.0448,
+      "grad_norm": 10.0,
+      "learning_rate": 8.9171974522293e-05,
+      "loss": 5.1907,
+      "step": 140
+    },
+    {
+      "epoch": 0.048,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 9.554140127388536e-05,
+      "loss": 4.9405,
+      "step": 150
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 10.0,
+      "learning_rate": 9.989892183288411e-05,
+      "loss": 4.4356,
+      "step": 160
+    },
+    {
+      "epoch": 0.0544,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 9.956199460916442e-05,
+      "loss": 4.1208,
+      "step": 170
+    },
+    {
+      "epoch": 0.0576,
+      "grad_norm": 9.999998092651367,
+      "learning_rate": 9.922506738544474e-05,
+      "loss": 4.1031,
+      "step": 180
+    },
+    {
+      "epoch": 0.0608,
+      "grad_norm": 10.000000953674316,
+      "learning_rate": 9.888814016172507e-05,
+      "loss": 4.0036,
+      "step": 190
+    },
+    {
+      "epoch": 0.064,
+      "grad_norm": 10.0,
+      "learning_rate": 9.85512129380054e-05,
+      "loss": 3.4382,
+      "step": 200
+    },
+    {
+      "epoch": 0.0672,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 9.821428571428572e-05,
+      "loss": 3.4154,
+      "step": 210
+    },
+    {
+      "epoch": 0.0704,
+      "grad_norm": 9.999998092651367,
+      "learning_rate": 9.787735849056603e-05,
+      "loss": 3.1193,
+      "step": 220
+    },
+    {
+      "epoch": 0.0736,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 9.754043126684636e-05,
+      "loss": 3.2772,
+      "step": 230
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 9.720350404312669e-05,
+      "loss": 3.0657,
+      "step": 240
+    },
+    {
+      "epoch": 0.08,
+      "grad_norm": 8.761540412902832,
+      "learning_rate": 9.686657681940702e-05,
+      "loss": 2.972,
+      "step": 250
+    },
+    {
+      "epoch": 0.0832,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 9.652964959568734e-05,
+      "loss": 2.9124,
+      "step": 260
+    },
+    {
+      "epoch": 0.0864,
+      "grad_norm": 9.999998092651367,
+      "learning_rate": 9.619272237196765e-05,
+      "loss": 3.0919,
+      "step": 270
+    },
+    {
+      "epoch": 0.0896,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 9.585579514824798e-05,
+      "loss": 2.774,
+      "step": 280
+    },
+    {
+      "epoch": 0.0928,
+      "grad_norm": 9.565616607666016,
+      "learning_rate": 9.551886792452831e-05,
+      "loss": 2.5328,
+      "step": 290
+    },
+    {
+      "epoch": 0.096,
+      "grad_norm": 8.902998924255371,
+      "learning_rate": 9.518194070080863e-05,
+      "loss": 2.9905,
+      "step": 300
+    },
+    {
+      "epoch": 0.0992,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 9.484501347708896e-05,
+      "loss": 2.6495,
+      "step": 310
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 9.722219467163086,
+      "learning_rate": 9.450808625336927e-05,
+      "loss": 2.5586,
+      "step": 320
+    },
+    {
+      "epoch": 0.1056,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 9.41711590296496e-05,
+      "loss": 2.3115,
+      "step": 330
+    },
+    {
+      "epoch": 0.1088,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 9.383423180592993e-05,
+      "loss": 2.1889,
+      "step": 340
+    },
+    {
+      "epoch": 0.112,
+      "grad_norm": 8.74697494506836,
+      "learning_rate": 9.349730458221025e-05,
+      "loss": 2.253,
+      "step": 350
+    },
+    {
+      "epoch": 0.1152,
+      "grad_norm": 7.3374552726745605,
+      "learning_rate": 9.316037735849057e-05,
+      "loss": 2.0232,
+      "step": 360
+    },
+    {
+      "epoch": 0.1184,
+      "grad_norm": 9.001267433166504,
+      "learning_rate": 9.282345013477089e-05,
+      "loss": 1.9705,
+      "step": 370
+    },
+    {
+      "epoch": 0.1216,
+      "grad_norm": 9.275607109069824,
+      "learning_rate": 9.248652291105122e-05,
+      "loss": 1.9923,
+      "step": 380
+    },
+    {
+      "epoch": 0.1248,
+      "grad_norm": 9.999998092651367,
+      "learning_rate": 9.214959568733154e-05,
+      "loss": 1.7894,
+      "step": 390
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 10.0,
+      "learning_rate": 9.181266846361186e-05,
+      "loss": 1.8159,
+      "step": 400
+    },
+    {
+      "epoch": 0.1312,
+      "grad_norm": 7.648778915405273,
+      "learning_rate": 9.14757412398922e-05,
+      "loss": 1.7174,
+      "step": 410
+    },
+    {
+      "epoch": 0.1344,
+      "grad_norm": 6.330312728881836,
+      "learning_rate": 9.113881401617251e-05,
+      "loss": 1.5952,
+      "step": 420
+    },
+    {
+      "epoch": 0.1376,
+      "grad_norm": 8.466596603393555,
+      "learning_rate": 9.080188679245284e-05,
+      "loss": 1.6767,
+      "step": 430
+    },
+    {
+      "epoch": 0.1408,
+      "grad_norm": 6.782797336578369,
+      "learning_rate": 9.046495956873315e-05,
+      "loss": 1.4808,
+      "step": 440
+    },
+    {
+      "epoch": 0.144,
+      "grad_norm": 5.610926628112793,
+      "learning_rate": 9.012803234501348e-05,
+      "loss": 1.4708,
+      "step": 450
+    },
+    {
+      "epoch": 0.1472,
+      "grad_norm": 6.343318939208984,
+      "learning_rate": 8.979110512129381e-05,
+      "loss": 1.5098,
+      "step": 460
+    },
+    {
+      "epoch": 0.1504,
+      "grad_norm": 6.048929214477539,
+      "learning_rate": 8.945417789757413e-05,
+      "loss": 1.367,
+      "step": 470
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 5.7954630851745605,
+      "learning_rate": 8.911725067385444e-05,
+      "loss": 1.3129,
+      "step": 480
+    },
+    {
+      "epoch": 0.1568,
+      "grad_norm": 6.33234977722168,
+      "learning_rate": 8.878032345013477e-05,
+      "loss": 1.3068,
+      "step": 490
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 5.885000705718994,
+      "learning_rate": 8.84433962264151e-05,
+      "loss": 1.3866,
+      "step": 500
+    },
+    {
+      "epoch": 0.1632,
+      "grad_norm": 7.096286773681641,
+      "learning_rate": 8.810646900269543e-05,
+      "loss": 1.2557,
+      "step": 510
+    },
+    {
+      "epoch": 0.1664,
+      "grad_norm": 7.024704933166504,
+      "learning_rate": 8.776954177897575e-05,
+      "loss": 1.324,
+      "step": 520
+    },
+    {
+      "epoch": 0.1696,
+      "grad_norm": 6.996052265167236,
+      "learning_rate": 8.743261455525606e-05,
+      "loss": 1.1659,
+      "step": 530
+    },
+    {
+      "epoch": 0.1728,
+      "grad_norm": 7.000609397888184,
+      "learning_rate": 8.709568733153639e-05,
+      "loss": 1.2756,
+      "step": 540
+    },
+    {
+      "epoch": 0.176,
+      "grad_norm": 7.971796989440918,
+      "learning_rate": 8.675876010781672e-05,
+      "loss": 1.2388,
+      "step": 550
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 5.358087539672852,
+      "learning_rate": 8.642183288409704e-05,
+      "loss": 1.2079,
+      "step": 560
+    },
+    {
+      "epoch": 0.1824,
+      "grad_norm": 9.458792686462402,
+      "learning_rate": 8.608490566037735e-05,
+      "loss": 1.2221,
+      "step": 570
+    },
+    {
+      "epoch": 0.1856,
+      "grad_norm": 5.870639801025391,
+      "learning_rate": 8.574797843665768e-05,
+      "loss": 1.1891,
+      "step": 580
+    },
+    {
+      "epoch": 0.1888,
+      "grad_norm": 6.394337177276611,
+      "learning_rate": 8.541105121293801e-05,
+      "loss": 1.159,
+      "step": 590
+    },
+    {
+      "epoch": 0.192,
+      "grad_norm": 8.1649808883667,
+      "learning_rate": 8.507412398921834e-05,
+      "loss": 1.1628,
+      "step": 600
+    },
+    {
+      "epoch": 0.1952,
+      "grad_norm": 5.609042167663574,
+      "learning_rate": 8.473719676549866e-05,
+      "loss": 1.0305,
+      "step": 610
+    },
+    {
+      "epoch": 0.1984,
+      "grad_norm": 5.179653644561768,
+      "learning_rate": 8.440026954177897e-05,
+      "loss": 1.1044,
+      "step": 620
+    },
+    {
+      "epoch": 0.2016,
+      "grad_norm": 7.2446699142456055,
+      "learning_rate": 8.40633423180593e-05,
+      "loss": 1.023,
+      "step": 630
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 8.372641509433963e-05,
+      "loss": 1.0946,
+      "step": 640
+    },
+    {
+      "epoch": 0.208,
+      "grad_norm": 8.831192970275879,
+      "learning_rate": 8.338948787061996e-05,
+      "loss": 1.0212,
+      "step": 650
+    },
+    {
+      "epoch": 0.2112,
+      "grad_norm": 6.253108978271484,
+      "learning_rate": 8.305256064690027e-05,
+      "loss": 1.0434,
+      "step": 660
+    },
+    {
+      "epoch": 0.2144,
+      "grad_norm": 7.159657001495361,
+      "learning_rate": 8.271563342318059e-05,
+      "loss": 1.0539,
+      "step": 670
+    },
+    {
+      "epoch": 0.2176,
+      "grad_norm": 6.7119059562683105,
+      "learning_rate": 8.237870619946092e-05,
+      "loss": 0.9833,
+      "step": 680
+    },
+    {
+      "epoch": 0.2208,
+      "grad_norm": 6.3738555908203125,
+      "learning_rate": 8.204177897574125e-05,
+      "loss": 0.9824,
+      "step": 690
+    },
+    {
+      "epoch": 0.224,
+      "grad_norm": 5.861403465270996,
+      "learning_rate": 8.170485175202158e-05,
+      "loss": 0.9474,
+      "step": 700
+    },
+    {
+      "epoch": 0.2272,
+      "grad_norm": 5.485085487365723,
+      "learning_rate": 8.136792452830189e-05,
+      "loss": 0.9043,
+      "step": 710
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 6.042101860046387,
+      "learning_rate": 8.103099730458221e-05,
+      "loss": 0.9085,
+      "step": 720
+    },
+    {
+      "epoch": 0.2336,
+      "grad_norm": 4.15228796005249,
+      "learning_rate": 8.069407008086254e-05,
+      "loss": 0.9097,
+      "step": 730
+    },
+    {
+      "epoch": 0.2368,
+      "grad_norm": 6.994100093841553,
+      "learning_rate": 8.035714285714287e-05,
+      "loss": 0.9379,
+      "step": 740
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 5.987849235534668,
+      "learning_rate": 8.002021563342318e-05,
+      "loss": 0.8864,
+      "step": 750
+    },
+    {
+      "epoch": 0.2432,
+      "grad_norm": 5.018003463745117,
+      "learning_rate": 7.968328840970351e-05,
+      "loss": 0.8352,
+      "step": 760
+    },
+    {
+      "epoch": 0.2464,
+      "grad_norm": 5.481317520141602,
+      "learning_rate": 7.934636118598383e-05,
+      "loss": 0.8373,
+      "step": 770
+    },
+    {
+      "epoch": 0.2496,
+      "grad_norm": 5.776020526885986,
+      "learning_rate": 7.900943396226416e-05,
+      "loss": 0.8507,
+      "step": 780
+    },
+    {
+      "epoch": 0.2528,
+      "grad_norm": 9.179359436035156,
+      "learning_rate": 7.867250673854449e-05,
+      "loss": 0.9298,
+      "step": 790
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 6.685598850250244,
+      "learning_rate": 7.83355795148248e-05,
+      "loss": 0.921,
+      "step": 800
+    },
+    {
+      "epoch": 0.2592,
+      "grad_norm": 5.743672847747803,
+      "learning_rate": 7.799865229110512e-05,
+      "loss": 0.7999,
+      "step": 810
+    },
+    {
+      "epoch": 0.2624,
+      "grad_norm": 5.498839855194092,
+      "learning_rate": 7.766172506738545e-05,
+      "loss": 0.8054,
+      "step": 820
+    },
+    {
+      "epoch": 0.2656,
+      "grad_norm": 6.5647759437561035,
+      "learning_rate": 7.732479784366577e-05,
+      "loss": 0.7677,
+      "step": 830
+    },
+    {
+      "epoch": 0.2688,
+      "grad_norm": 8.668718338012695,
+      "learning_rate": 7.69878706199461e-05,
+      "loss": 0.8012,
+      "step": 840
+    },
+    {
+      "epoch": 0.272,
+      "grad_norm": 9.059419631958008,
+      "learning_rate": 7.665094339622642e-05,
+      "loss": 0.7835,
+      "step": 850
+    },
+    {
+      "epoch": 0.2752,
+      "grad_norm": 7.959586143493652,
+      "learning_rate": 7.631401617250674e-05,
+      "loss": 0.7771,
+      "step": 860
+    },
+    {
+      "epoch": 0.2784,
+      "grad_norm": 6.478748321533203,
+      "learning_rate": 7.597708894878706e-05,
+      "loss": 0.7368,
+      "step": 870
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 4.239030361175537,
+      "learning_rate": 7.56401617250674e-05,
+      "loss": 0.7089,
+      "step": 880
+    },
+    {
+      "epoch": 0.2848,
+      "grad_norm": 9.278738975524902,
+      "learning_rate": 7.530323450134771e-05,
+      "loss": 0.7317,
+      "step": 890
+    },
+    {
+      "epoch": 0.288,
+      "grad_norm": 6.447314262390137,
+      "learning_rate": 7.496630727762804e-05,
+      "loss": 0.722,
+      "step": 900
+    },
+    {
+      "epoch": 0.2912,
+      "grad_norm": 5.464788913726807,
+      "learning_rate": 7.462938005390835e-05,
+      "loss": 0.7324,
+      "step": 910
+    },
+    {
+      "epoch": 0.2944,
+      "grad_norm": 7.056868553161621,
+      "learning_rate": 7.429245283018868e-05,
+      "loss": 0.6865,
+      "step": 920
+    },
+    {
+      "epoch": 0.2976,
+      "grad_norm": 6.860642433166504,
+      "learning_rate": 7.395552560646901e-05,
+      "loss": 0.685,
+      "step": 930
+    },
+    {
+      "epoch": 0.3008,
+      "grad_norm": 5.985160827636719,
+      "learning_rate": 7.361859838274933e-05,
+      "loss": 0.642,
+      "step": 940
+    },
+    {
+      "epoch": 0.304,
+      "grad_norm": 4.997369766235352,
+      "learning_rate": 7.328167115902966e-05,
+      "loss": 0.6923,
+      "step": 950
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 4.959667205810547,
+      "learning_rate": 7.294474393530997e-05,
+      "loss": 0.5946,
+      "step": 960
+    },
+    {
+      "epoch": 0.3104,
+      "grad_norm": 3.164045810699463,
+      "learning_rate": 7.26078167115903e-05,
+      "loss": 0.6428,
+      "step": 970
+    },
+    {
+      "epoch": 0.3136,
+      "grad_norm": 6.073975086212158,
+      "learning_rate": 7.227088948787062e-05,
+      "loss": 0.6144,
+      "step": 980
+    },
+    {
+      "epoch": 0.3168,
+      "grad_norm": 5.7967658042907715,
+      "learning_rate": 7.193396226415095e-05,
+      "loss": 0.5612,
+      "step": 990
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 7.2569193840026855,
+      "learning_rate": 7.159703504043128e-05,
+      "loss": 0.6135,
+      "step": 1000
+    },
+    {
+      "epoch": 0.3232,
+      "grad_norm": 6.437429428100586,
+      "learning_rate": 7.126010781671159e-05,
+      "loss": 0.5547,
+      "step": 1010
+    },
+    {
+      "epoch": 0.3264,
+      "grad_norm": 7.040581703186035,
+      "learning_rate": 7.092318059299192e-05,
+      "loss": 0.5782,
+      "step": 1020
+    },
+    {
+      "epoch": 0.3296,
+      "grad_norm": 4.126226425170898,
+      "learning_rate": 7.058625336927224e-05,
+      "loss": 0.5826,
+      "step": 1030
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 4.131496429443359,
+      "learning_rate": 7.024932614555257e-05,
+      "loss": 0.5487,
+      "step": 1040
+    },
+    {
+      "epoch": 0.336,
+      "grad_norm": 5.120275020599365,
+      "learning_rate": 6.99123989218329e-05,
+      "loss": 0.6073,
+      "step": 1050
+    },
+    {
+      "epoch": 0.3392,
+      "grad_norm": 7.478310585021973,
+      "learning_rate": 6.957547169811321e-05,
+      "loss": 0.538,
+      "step": 1060
+    },
+    {
+      "epoch": 0.3424,
+      "grad_norm": 6.256986618041992,
+      "learning_rate": 6.923854447439353e-05,
+      "loss": 0.5305,
+      "step": 1070
+    },
+    {
+      "epoch": 0.3456,
+      "grad_norm": 5.667956829071045,
+      "learning_rate": 6.890161725067386e-05,
+      "loss": 0.6043,
+      "step": 1080
+    },
+    {
+      "epoch": 0.3488,
+      "grad_norm": 6.5927629470825195,
+      "learning_rate": 6.856469002695418e-05,
+      "loss": 0.5626,
+      "step": 1090
+    },
+    {
+      "epoch": 0.352,
+      "grad_norm": 3.64158034324646,
+      "learning_rate": 6.822776280323451e-05,
+      "loss": 0.4975,
+      "step": 1100
+    },
+    {
+      "epoch": 0.3552,
+      "grad_norm": 5.662642955780029,
+      "learning_rate": 6.789083557951483e-05,
+      "loss": 0.4688,
+      "step": 1110
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 5.656427383422852,
+      "learning_rate": 6.755390835579514e-05,
+      "loss": 0.5197,
+      "step": 1120
+    },
+    {
+      "epoch": 0.3616,
+      "grad_norm": 4.796627521514893,
+      "learning_rate": 6.721698113207547e-05,
+      "loss": 0.4557,
+      "step": 1130
+    },
+    {
+      "epoch": 0.3648,
+      "grad_norm": 3.5207388401031494,
+      "learning_rate": 6.68800539083558e-05,
+      "loss": 0.4797,
+      "step": 1140
+    },
+    {
+      "epoch": 0.368,
+      "grad_norm": 3.8315589427948,
+      "learning_rate": 6.654312668463612e-05,
+      "loss": 0.4477,
+      "step": 1150
+    },
+    {
+      "epoch": 0.3712,
+      "grad_norm": 5.227057456970215,
+      "learning_rate": 6.620619946091643e-05,
+      "loss": 0.4506,
+      "step": 1160
+    },
+    {
+      "epoch": 0.3744,
+      "grad_norm": 6.755670547485352,
+      "learning_rate": 6.586927223719676e-05,
+      "loss": 0.4408,
+      "step": 1170
+    },
+    {
+      "epoch": 0.3776,
+      "grad_norm": 4.697142124176025,
+      "learning_rate": 6.553234501347709e-05,
+      "loss": 0.4378,
+      "step": 1180
+    },
+    {
+      "epoch": 0.3808,
+      "grad_norm": 6.8359694480896,
+      "learning_rate": 6.519541778975742e-05,
+      "loss": 0.4559,
+      "step": 1190
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 4.0120954513549805,
+      "learning_rate": 6.485849056603774e-05,
+      "loss": 0.3792,
+      "step": 1200
+    },
+    {
+      "epoch": 0.3872,
+      "grad_norm": 4.8486223220825195,
+      "learning_rate": 6.452156334231805e-05,
+      "loss": 0.4224,
+      "step": 1210
+    },
+    {
+      "epoch": 0.3904,
+      "grad_norm": 4.273190975189209,
+      "learning_rate": 6.418463611859838e-05,
+      "loss": 0.4128,
+      "step": 1220
+    },
+    {
+      "epoch": 0.3936,
+      "grad_norm": 5.036397457122803,
+      "learning_rate": 6.384770889487871e-05,
+      "loss": 0.4054,
+      "step": 1230
+    },
+    {
+      "epoch": 0.3968,
+      "grad_norm": 4.676331520080566,
+      "learning_rate": 6.351078167115904e-05,
+      "loss": 0.4142,
+      "step": 1240
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 5.700189590454102,
+      "learning_rate": 6.317385444743936e-05,
+      "loss": 0.3809,
+      "step": 1250
+    },
+    {
+      "epoch": 0.4032,
+      "grad_norm": 3.157870292663574,
+      "learning_rate": 6.283692722371967e-05,
+      "loss": 0.3646,
+      "step": 1260
+    },
+    {
+      "epoch": 0.4064,
+      "grad_norm": 4.610825061798096,
+      "learning_rate": 6.25e-05,
+      "loss": 0.3865,
+      "step": 1270
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 5.093286991119385,
+      "learning_rate": 6.216307277628033e-05,
+      "loss": 0.3755,
+      "step": 1280
+    },
+    {
+      "epoch": 0.4128,
+      "grad_norm": 6.79491662979126,
+      "learning_rate": 6.182614555256066e-05,
+      "loss": 0.4263,
+      "step": 1290
+    },
+    {
+      "epoch": 0.416,
+      "grad_norm": 4.933012962341309,
+      "learning_rate": 6.148921832884098e-05,
+      "loss": 0.3855,
+      "step": 1300
+    },
+    {
+      "epoch": 0.4192,
+      "grad_norm": 2.9703903198242188,
+      "learning_rate": 6.115229110512129e-05,
+      "loss": 0.3651,
+      "step": 1310
+    },
+    {
+      "epoch": 0.4224,
+      "grad_norm": 2.5094480514526367,
+      "learning_rate": 6.081536388140162e-05,
+      "loss": 0.3714,
+      "step": 1320
+    },
+    {
+      "epoch": 0.4256,
+      "grad_norm": 7.232475757598877,
+      "learning_rate": 6.047843665768195e-05,
+      "loss": 0.3566,
+      "step": 1330
+    },
+    {
+      "epoch": 0.4288,
+      "grad_norm": 3.515270233154297,
+      "learning_rate": 6.0141509433962265e-05,
+      "loss": 0.3918,
+      "step": 1340
+    },
+    {
+      "epoch": 0.432,
+      "grad_norm": 4.976472854614258,
+      "learning_rate": 5.980458221024259e-05,
+      "loss": 0.3426,
+      "step": 1350
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 3.9805610179901123,
+      "learning_rate": 5.9467654986522916e-05,
+      "loss": 0.3565,
+      "step": 1360
+    },
+    {
+      "epoch": 0.4384,
+      "grad_norm": 4.896690845489502,
+      "learning_rate": 5.913072776280324e-05,
+      "loss": 0.321,
+      "step": 1370
+    },
+    {
+      "epoch": 0.4416,
+      "grad_norm": 4.799086570739746,
+      "learning_rate": 5.879380053908357e-05,
+      "loss": 0.3153,
+      "step": 1380
+    },
+    {
+      "epoch": 0.4448,
+      "grad_norm": 6.445730686187744,
+      "learning_rate": 5.8456873315363884e-05,
+      "loss": 0.3124,
+      "step": 1390
+    },
+    {
+      "epoch": 0.448,
+      "grad_norm": 3.805938720703125,
+      "learning_rate": 5.8119946091644206e-05,
+      "loss": 0.2992,
+      "step": 1400
+    },
+    {
+      "epoch": 0.4512,
+      "grad_norm": 4.300934314727783,
+      "learning_rate": 5.7783018867924535e-05,
+      "loss": 0.3244,
+      "step": 1410
+    },
+    {
+      "epoch": 0.4544,
+      "grad_norm": 5.322298526763916,
+      "learning_rate": 5.744609164420486e-05,
+      "loss": 0.2993,
+      "step": 1420
+    },
+    {
+      "epoch": 0.4576,
+      "grad_norm": 3.0863025188446045,
+      "learning_rate": 5.710916442048517e-05,
+      "loss": 0.3228,
+      "step": 1430
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 2.8616793155670166,
+      "learning_rate": 5.6772237196765496e-05,
+      "loss": 0.2901,
+      "step": 1440
+    },
+    {
+      "epoch": 0.464,
+      "grad_norm": 2.0354461669921875,
+      "learning_rate": 5.6435309973045825e-05,
+      "loss": 0.2827,
+      "step": 1450
+    },
+    {
+      "epoch": 0.4672,
+      "grad_norm": 3.377239465713501,
+      "learning_rate": 5.609838274932615e-05,
+      "loss": 0.2676,
+      "step": 1460
+    },
+    {
+      "epoch": 0.4704,
+      "grad_norm": 5.67920446395874,
+      "learning_rate": 5.5761455525606476e-05,
+      "loss": 0.2909,
+      "step": 1470
+    },
+    {
+      "epoch": 0.4736,
+      "grad_norm": 3.444645881652832,
+      "learning_rate": 5.542452830188679e-05,
+      "loss": 0.3126,
+      "step": 1480
+    },
+    {
+      "epoch": 0.4768,
+      "grad_norm": 3.0336413383483887,
+      "learning_rate": 5.5087601078167114e-05,
+      "loss": 0.2749,
+      "step": 1490
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 3.9594080448150635,
+      "learning_rate": 5.4750673854447444e-05,
+      "loss": 0.2601,
+      "step": 1500
+    },
+    {
+      "epoch": 0.4832,
+      "grad_norm": 4.596884250640869,
+      "learning_rate": 5.4413746630727766e-05,
+      "loss": 0.2802,
+      "step": 1510
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 4.85414981842041,
+      "learning_rate": 5.407681940700808e-05,
+      "loss": 0.2885,
+      "step": 1520
+    },
+    {
+      "epoch": 0.4896,
+      "grad_norm": 4.045989036560059,
+      "learning_rate": 5.373989218328841e-05,
+      "loss": 0.2758,
+      "step": 1530
+    },
+    {
+      "epoch": 0.4928,
+      "grad_norm": 4.910004615783691,
+      "learning_rate": 5.340296495956873e-05,
+      "loss": 0.2662,
+      "step": 1540
+    },
+    {
+      "epoch": 0.496,
+      "grad_norm": 2.3728251457214355,
+      "learning_rate": 5.306603773584906e-05,
+      "loss": 0.2358,
+      "step": 1550
+    },
+    {
+      "epoch": 0.4992,
+      "grad_norm": 4.005071640014648,
+      "learning_rate": 5.2729110512129385e-05,
+      "loss": 0.2588,
+      "step": 1560
+    },
+    {
+      "epoch": 0.5024,
+      "grad_norm": 2.8403639793395996,
+      "learning_rate": 5.23921832884097e-05,
+      "loss": 0.2581,
+      "step": 1570
+    },
+    {
+      "epoch": 0.5056,
+      "grad_norm": 3.1361353397369385,
+      "learning_rate": 5.205525606469003e-05,
+      "loss": 0.2571,
+      "step": 1580
+    },
+    {
+      "epoch": 0.5088,
+      "grad_norm": 4.602989196777344,
+      "learning_rate": 5.171832884097035e-05,
+      "loss": 0.2336,
+      "step": 1590
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 2.989924907684326,
+      "learning_rate": 5.138140161725068e-05,
+      "loss": 0.2522,
+      "step": 1600
+    },
+    {
+      "epoch": 0.5152,
+      "grad_norm": 2.5031306743621826,
+      "learning_rate": 5.1044474393531e-05,
+      "loss": 0.2687,
+      "step": 1610
+    },
+    {
+      "epoch": 0.5184,
+      "grad_norm": 3.078021287918091,
+      "learning_rate": 5.070754716981132e-05,
+      "loss": 0.247,
+      "step": 1620
+    },
+    {
+      "epoch": 0.5216,
+      "grad_norm": 2.928231954574585,
+      "learning_rate": 5.037061994609165e-05,
+      "loss": 0.2142,
+      "step": 1630
+    },
+    {
+      "epoch": 0.5248,
+      "grad_norm": 2.902031183242798,
+      "learning_rate": 5.003369272237197e-05,
+      "loss": 0.2244,
+      "step": 1640
+    },
+    {
+      "epoch": 0.528,
+      "grad_norm": 3.3071529865264893,
+      "learning_rate": 4.969676549865229e-05,
+      "loss": 0.2215,
+      "step": 1650
+    },
+    {
+      "epoch": 0.5312,
+      "grad_norm": 4.961091041564941,
+      "learning_rate": 4.9359838274932616e-05,
+      "loss": 0.2194,
+      "step": 1660
+    },
+    {
+      "epoch": 0.5344,
+      "grad_norm": 1.9838968515396118,
+      "learning_rate": 4.902291105121294e-05,
+      "loss": 0.2351,
+      "step": 1670
+    },
+    {
+      "epoch": 0.5376,
+      "grad_norm": 2.13077712059021,
+      "learning_rate": 4.868598382749327e-05,
+      "loss": 0.2223,
+      "step": 1680
+    },
+    {
+      "epoch": 0.5408,
+      "grad_norm": 2.2034871578216553,
+      "learning_rate": 4.834905660377358e-05,
+      "loss": 0.1995,
+      "step": 1690
+    },
+    {
+      "epoch": 0.544,
+      "grad_norm": 2.9130163192749023,
+      "learning_rate": 4.801212938005391e-05,
+      "loss": 0.21,
+      "step": 1700
+    },
+    {
+      "epoch": 0.5472,
+      "grad_norm": 3.775465488433838,
+      "learning_rate": 4.7675202156334234e-05,
+      "loss": 0.2116,
+      "step": 1710
+    },
+    {
+      "epoch": 0.5504,
+      "grad_norm": 2.462954521179199,
+      "learning_rate": 4.733827493261456e-05,
+      "loss": 0.2039,
+      "step": 1720
+    },
+    {
+      "epoch": 0.5536,
+      "grad_norm": 2.234292984008789,
+      "learning_rate": 4.7001347708894886e-05,
+      "loss": 0.2043,
+      "step": 1730
+    },
+    {
+      "epoch": 0.5568,
+      "grad_norm": 4.2636003494262695,
+      "learning_rate": 4.66644204851752e-05,
+      "loss": 0.2069,
+      "step": 1740
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 2.1068809032440186,
+      "learning_rate": 4.632749326145553e-05,
+      "loss": 0.2034,
+      "step": 1750
+    },
+    {
+      "epoch": 0.5632,
+      "grad_norm": 1.9375625848770142,
+      "learning_rate": 4.5990566037735846e-05,
+      "loss": 0.2117,
+      "step": 1760
+    },
+    {
+      "epoch": 0.5664,
+      "grad_norm": 2.2950830459594727,
+      "learning_rate": 4.5653638814016176e-05,
+      "loss": 0.192,
+      "step": 1770
+    },
+    {
+      "epoch": 0.5696,
+      "grad_norm": 2.402663469314575,
+      "learning_rate": 4.53167115902965e-05,
+      "loss": 0.1947,
+      "step": 1780
+    },
+    {
+      "epoch": 0.5728,
+      "grad_norm": 2.897888422012329,
+      "learning_rate": 4.497978436657682e-05,
+      "loss": 0.2201,
+      "step": 1790
+    },
+    {
+      "epoch": 0.576,
+      "grad_norm": 2.6517560482025146,
+      "learning_rate": 4.464285714285715e-05,
+      "loss": 0.2113,
+      "step": 1800
+    },
+    {
+      "epoch": 0.5792,
+      "grad_norm": 1.8639112710952759,
+      "learning_rate": 4.4305929919137465e-05,
+      "loss": 0.2017,
+      "step": 1810
+    },
+    {
+      "epoch": 0.5824,
+      "grad_norm": 2.103375196456909,
+      "learning_rate": 4.3969002695417794e-05,
+      "loss": 0.181,
+      "step": 1820
+    },
+    {
+      "epoch": 0.5856,
+      "grad_norm": 3.1469459533691406,
+      "learning_rate": 4.363207547169812e-05,
+      "loss": 0.1982,
+      "step": 1830
+    },
+    {
+      "epoch": 0.5888,
+      "grad_norm": 2.1390271186828613,
+      "learning_rate": 4.329514824797844e-05,
+      "loss": 0.1896,
+      "step": 1840
+    },
+    {
+      "epoch": 0.592,
+      "grad_norm": 3.2703566551208496,
+      "learning_rate": 4.295822102425876e-05,
+      "loss": 0.1892,
+      "step": 1850
+    },
+    {
+      "epoch": 0.5952,
+      "grad_norm": 1.4746158123016357,
+      "learning_rate": 4.2621293800539084e-05,
+      "loss": 0.1712,
+      "step": 1860
+    },
+    {
+      "epoch": 0.5984,
+      "grad_norm": 2.221888303756714,
+      "learning_rate": 4.2284366576819406e-05,
+      "loss": 0.1759,
+      "step": 1870
+    },
+    {
+      "epoch": 0.6016,
+      "grad_norm": 3.779874801635742,
+      "learning_rate": 4.1947439353099736e-05,
+      "loss": 0.1992,
+      "step": 1880
+    },
+    {
+      "epoch": 0.6048,
+      "grad_norm": 2.6584346294403076,
+      "learning_rate": 4.161051212938006e-05,
+      "loss": 0.1763,
+      "step": 1890
+    },
+    {
+      "epoch": 0.608,
+      "grad_norm": 3.278395175933838,
+      "learning_rate": 4.127358490566038e-05,
+      "loss": 0.1881,
+      "step": 1900
+    },
+    {
+      "epoch": 0.6112,
+      "grad_norm": 2.540827512741089,
+      "learning_rate": 4.09366576819407e-05,
+      "loss": 0.1906,
+      "step": 1910
+    },
+    {
+      "epoch": 0.6144,
+      "grad_norm": 3.8679630756378174,
+      "learning_rate": 4.0599730458221025e-05,
+      "loss": 0.181,
+      "step": 1920
+    },
+    {
+      "epoch": 0.6176,
+      "grad_norm": 1.833381175994873,
+      "learning_rate": 4.026280323450135e-05,
+      "loss": 0.1702,
+      "step": 1930
+    },
+    {
+      "epoch": 0.6208,
+      "grad_norm": 2.688124895095825,
+      "learning_rate": 3.992587601078167e-05,
+      "loss": 0.1733,
+      "step": 1940
+    },
+    {
+      "epoch": 0.624,
+      "grad_norm": 2.145625591278076,
+      "learning_rate": 3.9588948787062e-05,
+      "loss": 0.1734,
+      "step": 1950
+    },
+    {
+      "epoch": 0.6272,
+      "grad_norm": 3.425027370452881,
+      "learning_rate": 3.9252021563342315e-05,
+      "loss": 0.1648,
+      "step": 1960
+    },
+    {
+      "epoch": 0.6304,
+      "grad_norm": 2.4815447330474854,
+      "learning_rate": 3.8915094339622644e-05,
+      "loss": 0.1713,
+      "step": 1970
+    },
+    {
+      "epoch": 0.6336,
+      "grad_norm": 4.090925216674805,
+      "learning_rate": 3.8578167115902966e-05,
+      "loss": 0.1899,
+      "step": 1980
+    },
+    {
+      "epoch": 0.6368,
+      "grad_norm": 2.073986768722534,
+      "learning_rate": 3.824123989218329e-05,
+      "loss": 0.1631,
+      "step": 1990
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 2.5611414909362793,
+      "learning_rate": 3.790431266846362e-05,
+      "loss": 0.1801,
+      "step": 2000
+    },
+    {
+      "epoch": 0.6432,
+      "grad_norm": 2.3403127193450928,
+      "learning_rate": 3.7567385444743934e-05,
+      "loss": 0.1903,
+      "step": 2010
+    },
+    {
+      "epoch": 0.6464,
+      "grad_norm": 1.810889482498169,
+      "learning_rate": 3.723045822102426e-05,
+      "loss": 0.1573,
+      "step": 2020
+    },
+    {
+      "epoch": 0.6496,
+      "grad_norm": 2.551084280014038,
+      "learning_rate": 3.689353099730458e-05,
+      "loss": 0.1673,
+      "step": 2030
+    },
+    {
+      "epoch": 0.6528,
+      "grad_norm": 1.5169024467468262,
+      "learning_rate": 3.655660377358491e-05,
+      "loss": 0.175,
+      "step": 2040
+    },
+    {
+      "epoch": 0.656,
+      "grad_norm": 2.4477052688598633,
+      "learning_rate": 3.621967654986524e-05,
+      "loss": 0.1734,
+      "step": 2050
+    },
+    {
+      "epoch": 0.6592,
+      "grad_norm": 2.7971584796905518,
+      "learning_rate": 3.588274932614555e-05,
+      "loss": 0.1743,
+      "step": 2060
+    },
+    {
+      "epoch": 0.6624,
+      "grad_norm": 1.5752153396606445,
+      "learning_rate": 3.554582210242588e-05,
+      "loss": 0.1558,
+      "step": 2070
+    },
+    {
+      "epoch": 0.6656,
+      "grad_norm": 2.5388197898864746,
+      "learning_rate": 3.52088948787062e-05,
+      "loss": 0.1702,
+      "step": 2080
+    },
+    {
+      "epoch": 0.6688,
+      "grad_norm": 3.4686882495880127,
+      "learning_rate": 3.4871967654986526e-05,
+      "loss": 0.1485,
+      "step": 2090
+    },
+    {
+      "epoch": 0.672,
+      "grad_norm": 1.6380963325500488,
+      "learning_rate": 3.453504043126685e-05,
+      "loss": 0.15,
+      "step": 2100
+    },
+    {
+      "epoch": 0.6752,
+      "grad_norm": 1.6580733060836792,
+      "learning_rate": 3.419811320754717e-05,
+      "loss": 0.1811,
+      "step": 2110
+    },
+    {
+      "epoch": 0.6784,
+      "grad_norm": 3.220553398132324,
+      "learning_rate": 3.3861185983827494e-05,
+      "loss": 0.1471,
+      "step": 2120
+    },
+    {
+      "epoch": 0.6816,
+      "grad_norm": 3.0005507469177246,
+      "learning_rate": 3.3524258760107816e-05,
+      "loss": 0.1735,
+      "step": 2130
+    },
+    {
+      "epoch": 0.6848,
+      "grad_norm": 3.4063844680786133,
+      "learning_rate": 3.3187331536388145e-05,
+      "loss": 0.1616,
+      "step": 2140
+    },
+    {
+      "epoch": 0.688,
+      "grad_norm": 2.0916380882263184,
+      "learning_rate": 3.285040431266847e-05,
+      "loss": 0.1561,
+      "step": 2150
+    },
+    {
+      "epoch": 0.6912,
+      "grad_norm": 2.4332773685455322,
+      "learning_rate": 3.251347708894879e-05,
+      "loss": 0.1496,
+      "step": 2160
+    },
+    {
+      "epoch": 0.6944,
+      "grad_norm": 1.630477786064148,
+      "learning_rate": 3.217654986522911e-05,
+      "loss": 0.1593,
+      "step": 2170
+    },
+    {
+      "epoch": 0.6976,
+      "grad_norm": 1.7820992469787598,
+      "learning_rate": 3.1839622641509435e-05,
+      "loss": 0.1552,
+      "step": 2180
+    },
+    {
+      "epoch": 0.7008,
+      "grad_norm": 2.1285452842712402,
+      "learning_rate": 3.150269541778976e-05,
+      "loss": 0.1567,
+      "step": 2190
+    },
+    {
+      "epoch": 0.704,
+      "grad_norm": 1.402932047843933,
+      "learning_rate": 3.1165768194070086e-05,
+      "loss": 0.1391,
+      "step": 2200
+    },
+    {
+      "epoch": 0.7072,
+      "grad_norm": 2.5307276248931885,
+      "learning_rate": 3.08288409703504e-05,
+      "loss": 0.1784,
+      "step": 2210
+    },
+    {
+      "epoch": 0.7104,
+      "grad_norm": 2.1182377338409424,
+      "learning_rate": 3.0491913746630728e-05,
+      "loss": 0.1447,
+      "step": 2220
+    },
+    {
+      "epoch": 0.7136,
+      "grad_norm": 2.1142852306365967,
+      "learning_rate": 3.0154986522911054e-05,
+      "loss": 0.1488,
+      "step": 2230
+    },
+    {
+      "epoch": 0.7168,
+      "grad_norm": 1.8930586576461792,
+      "learning_rate": 2.9818059299191376e-05,
+      "loss": 0.1382,
+      "step": 2240
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 2.6080002784729004,
+      "learning_rate": 2.9481132075471702e-05,
+      "loss": 0.1331,
+      "step": 2250
+    },
+    {
+      "epoch": 0.7232,
+      "grad_norm": 1.5683133602142334,
+      "learning_rate": 2.914420485175202e-05,
+      "loss": 0.1362,
+      "step": 2260
+    },
+    {
+      "epoch": 0.7264,
+      "grad_norm": 2.0423901081085205,
+      "learning_rate": 2.8807277628032347e-05,
+      "loss": 0.1417,
+      "step": 2270
+    },
+    {
+      "epoch": 0.7296,
+      "grad_norm": 1.4694466590881348,
+      "learning_rate": 2.847035040431267e-05,
+      "loss": 0.1226,
+      "step": 2280
+    },
+    {
+      "epoch": 0.7328,
+      "grad_norm": 2.6974313259124756,
+      "learning_rate": 2.8133423180592995e-05,
+      "loss": 0.1255,
+      "step": 2290
+    },
+    {
+      "epoch": 0.736,
+      "grad_norm": 1.8697987794876099,
+      "learning_rate": 2.7796495956873314e-05,
+      "loss": 0.1403,
+      "step": 2300
+    },
+    {
+      "epoch": 0.7392,
+      "grad_norm": 2.2099053859710693,
+      "learning_rate": 2.745956873315364e-05,
+      "loss": 0.1283,
+      "step": 2310
+    },
+    {
+      "epoch": 0.7424,
+      "grad_norm": 1.6549280881881714,
+      "learning_rate": 2.7122641509433965e-05,
+      "loss": 0.1483,
+      "step": 2320
+    },
+    {
+      "epoch": 0.7456,
+      "grad_norm": 1.6001911163330078,
+      "learning_rate": 2.6785714285714288e-05,
+      "loss": 0.1391,
+      "step": 2330
+    },
+    {
+      "epoch": 0.7488,
+      "grad_norm": 2.202669620513916,
+      "learning_rate": 2.6448787061994614e-05,
+      "loss": 0.1527,
+      "step": 2340
+    },
+    {
+      "epoch": 0.752,
+      "grad_norm": 2.218289375305176,
+      "learning_rate": 2.6111859838274933e-05,
+      "loss": 0.1398,
+      "step": 2350
+    },
+    {
+      "epoch": 0.7552,
+      "grad_norm": 1.7072354555130005,
+      "learning_rate": 2.577493261455526e-05,
+      "loss": 0.1407,
+      "step": 2360
+    },
+    {
+      "epoch": 0.7584,
+      "grad_norm": 1.6652235984802246,
+      "learning_rate": 2.5438005390835577e-05,
+      "loss": 0.1538,
+      "step": 2370
+    },
+    {
+      "epoch": 0.7616,
+      "grad_norm": 1.6194990873336792,
+      "learning_rate": 2.5101078167115903e-05,
+      "loss": 0.132,
+      "step": 2380
+    },
+    {
+      "epoch": 0.7648,
+      "grad_norm": 1.8026024103164673,
+      "learning_rate": 2.476415094339623e-05,
+      "loss": 0.1393,
+      "step": 2390
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 2.7763803005218506,
+      "learning_rate": 2.442722371967655e-05,
+      "loss": 0.1372,
+      "step": 2400
+    },
+    {
+      "epoch": 0.7712,
+      "grad_norm": 1.5770050287246704,
+      "learning_rate": 2.4090296495956874e-05,
+      "loss": 0.1286,
+      "step": 2410
+    },
+    {
+      "epoch": 0.7744,
+      "grad_norm": 2.233755350112915,
+      "learning_rate": 2.3753369272237196e-05,
+      "loss": 0.129,
+      "step": 2420
+    },
+    {
+      "epoch": 0.7776,
+      "grad_norm": 2.8266842365264893,
+      "learning_rate": 2.341644204851752e-05,
+      "loss": 0.1319,
+      "step": 2430
+    },
+    {
+      "epoch": 0.7808,
+      "grad_norm": 2.37268328666687,
+      "learning_rate": 2.3079514824797844e-05,
+      "loss": 0.1282,
+      "step": 2440
+    },
+    {
+      "epoch": 0.784,
+      "grad_norm": 2.108520269393921,
+      "learning_rate": 2.274258760107817e-05,
+      "loss": 0.1258,
+      "step": 2450
+    },
+    {
+      "epoch": 0.7872,
+      "grad_norm": 2.3609049320220947,
+      "learning_rate": 2.2405660377358493e-05,
+      "loss": 0.1365,
+      "step": 2460
+    },
+    {
+      "epoch": 0.7904,
+      "grad_norm": 1.553999423980713,
+      "learning_rate": 2.2068733153638815e-05,
+      "loss": 0.1257,
+      "step": 2470
+    },
+    {
+      "epoch": 0.7936,
+      "grad_norm": 2.2637341022491455,
+      "learning_rate": 2.1731805929919137e-05,
+      "loss": 0.135,
+      "step": 2480
+    },
+    {
+      "epoch": 0.7968,
+      "grad_norm": 2.061565637588501,
+      "learning_rate": 2.1394878706199463e-05,
+      "loss": 0.1305,
+      "step": 2490
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 1.9381234645843506,
+      "learning_rate": 2.1057951482479785e-05,
+      "loss": 0.1232,
+      "step": 2500
+    },
+    {
+      "epoch": 0.8032,
+      "grad_norm": 2.815253496170044,
+      "learning_rate": 2.0721024258760108e-05,
+      "loss": 0.1301,
+      "step": 2510
+    },
+    {
+      "epoch": 0.8064,
+      "grad_norm": 1.929908037185669,
+      "learning_rate": 2.038409703504043e-05,
+      "loss": 0.1229,
+      "step": 2520
+    },
+    {
+      "epoch": 0.8096,
+      "grad_norm": 1.764770269393921,
+      "learning_rate": 2.0047169811320756e-05,
+      "loss": 0.1308,
+      "step": 2530
+    },
+    {
+      "epoch": 0.8128,
+      "grad_norm": 2.756413698196411,
+      "learning_rate": 1.971024258760108e-05,
+      "loss": 0.1274,
+      "step": 2540
+    },
+    {
+      "epoch": 0.816,
+      "grad_norm": 2.02465558052063,
+      "learning_rate": 1.9373315363881404e-05,
+      "loss": 0.1305,
+      "step": 2550
+    },
+    {
+      "epoch": 0.8192,
+      "grad_norm": 2.436875581741333,
+      "learning_rate": 1.9036388140161727e-05,
+      "loss": 0.1241,
+      "step": 2560
+    },
+    {
+      "epoch": 0.8224,
+      "grad_norm": 1.831134557723999,
+      "learning_rate": 1.869946091644205e-05,
+      "loss": 0.1385,
+      "step": 2570
+    },
+    {
+      "epoch": 0.8256,
+      "grad_norm": 1.4205396175384521,
+      "learning_rate": 1.836253369272237e-05,
+      "loss": 0.1237,
+      "step": 2580
+    },
+    {
+      "epoch": 0.8288,
+      "grad_norm": 2.1751766204833984,
+      "learning_rate": 1.8025606469002694e-05,
+      "loss": 0.1246,
+      "step": 2590
+    },
+    {
+      "epoch": 0.832,
+      "grad_norm": 1.4494165182113647,
+      "learning_rate": 1.768867924528302e-05,
+      "loss": 0.1367,
+      "step": 2600
+    },
+    {
+      "epoch": 0.8352,
+      "grad_norm": 1.5519049167633057,
+      "learning_rate": 1.7351752021563345e-05,
+      "loss": 0.1215,
+      "step": 2610
+    },
+    {
+      "epoch": 0.8384,
+      "grad_norm": 1.6617779731750488,
+      "learning_rate": 1.7014824797843668e-05,
+      "loss": 0.1213,
+      "step": 2620
+    },
+    {
+      "epoch": 0.8416,
+      "grad_norm": 2.4521913528442383,
+      "learning_rate": 1.667789757412399e-05,
+      "loss": 0.124,
+      "step": 2630
+    },
+    {
+      "epoch": 0.8448,
+      "grad_norm": 2.1117968559265137,
+      "learning_rate": 1.6340970350404313e-05,
+      "loss": 0.1361,
+      "step": 2640
+    },
+    {
+      "epoch": 0.848,
+      "grad_norm": 1.941954255104065,
+      "learning_rate": 1.600404312668464e-05,
+      "loss": 0.1183,
+      "step": 2650
+    },
+    {
+      "epoch": 0.8512,
+      "grad_norm": 2.1985981464385986,
+      "learning_rate": 1.566711590296496e-05,
+      "loss": 0.1228,
+      "step": 2660
+    },
+    {
+      "epoch": 0.8544,
+      "grad_norm": 1.612240195274353,
+      "learning_rate": 1.5330188679245283e-05,
+      "loss": 0.131,
+      "step": 2670
+    },
+    {
+      "epoch": 0.8576,
+      "grad_norm": 1.4507054090499878,
+      "learning_rate": 1.4993261455525606e-05,
+      "loss": 0.1178,
+      "step": 2680
+    },
+    {
+      "epoch": 0.8608,
+      "grad_norm": 1.6707614660263062,
+      "learning_rate": 1.465633423180593e-05,
+      "loss": 0.1214,
+      "step": 2690
+    },
+    {
+      "epoch": 0.864,
+      "grad_norm": 1.8366183042526245,
+      "learning_rate": 1.4319407008086256e-05,
+      "loss": 0.1264,
+      "step": 2700
+    },
+    {
+      "epoch": 0.8672,
+      "grad_norm": 1.4520529508590698,
+      "learning_rate": 1.3982479784366578e-05,
+      "loss": 0.1249,
+      "step": 2710
+    },
+    {
+      "epoch": 0.8704,
+      "grad_norm": 1.988840937614441,
+      "learning_rate": 1.3645552560646902e-05,
+      "loss": 0.1181,
+      "step": 2720
+    },
+    {
+      "epoch": 0.8736,
+      "grad_norm": 1.2967309951782227,
+      "learning_rate": 1.3308625336927224e-05,
+      "loss": 0.1187,
+      "step": 2730
+    },
+    {
+      "epoch": 0.8768,
+      "grad_norm": 1.501754641532898,
+      "learning_rate": 1.2971698113207547e-05,
+      "loss": 0.1148,
+      "step": 2740
+    },
+    {
+      "epoch": 0.88,
+      "grad_norm": 1.7325700521469116,
+      "learning_rate": 1.2634770889487871e-05,
+      "loss": 0.1316,
+      "step": 2750
+    },
+    {
+      "epoch": 0.8832,
+      "grad_norm": 1.819923758506775,
+      "learning_rate": 1.2297843665768195e-05,
+      "loss": 0.1894,
+      "step": 2760
+    },
+    {
+      "epoch": 0.8864,
+      "grad_norm": 2.5763540267944336,
+      "learning_rate": 1.1960916442048519e-05,
+      "loss": 0.1177,
+      "step": 2770
+    },
+    {
+      "epoch": 0.8896,
+      "grad_norm": 2.4021575450897217,
+      "learning_rate": 1.1623989218328842e-05,
+      "loss": 0.1166,
+      "step": 2780
+    },
+    {
+      "epoch": 0.8928,
+      "grad_norm": 2.2571208477020264,
+      "learning_rate": 1.1287061994609164e-05,
+      "loss": 0.116,
+      "step": 2790
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 2.1551597118377686,
+      "learning_rate": 1.0950134770889488e-05,
+      "loss": 0.1203,
+      "step": 2800
+    },
+    {
+      "epoch": 0.8992,
+      "grad_norm": 1.2261362075805664,
+      "learning_rate": 1.0613207547169812e-05,
+      "loss": 0.1281,
+      "step": 2810
+    },
+    {
+      "epoch": 0.9024,
+      "grad_norm": 2.0020546913146973,
+      "learning_rate": 1.0276280323450135e-05,
+      "loss": 0.1069,
+      "step": 2820
+    },
+    {
+      "epoch": 0.9056,
+      "grad_norm": 1.861535668373108,
+      "learning_rate": 9.939353099730459e-06,
+      "loss": 0.1092,
+      "step": 2830
+    },
+    {
+      "epoch": 0.9088,
+      "grad_norm": 1.7456763982772827,
+      "learning_rate": 9.602425876010781e-06,
+      "loss": 0.1243,
+      "step": 2840
+    },
+    {
+      "epoch": 0.912,
+      "grad_norm": 2.4167239665985107,
+      "learning_rate": 9.265498652291107e-06,
+      "loss": 0.1134,
+      "step": 2850
+    },
+    {
+      "epoch": 0.9152,
+      "grad_norm": 1.368345022201538,
+      "learning_rate": 8.92857142857143e-06,
+      "loss": 0.1081,
+      "step": 2860
+    },
+    {
+      "epoch": 0.9184,
+      "grad_norm": 1.4125988483428955,
+      "learning_rate": 8.591644204851752e-06,
+      "loss": 0.1067,
+      "step": 2870
+    },
+    {
+      "epoch": 0.9216,
+      "grad_norm": 1.7639960050582886,
+      "learning_rate": 8.254716981132076e-06,
+      "loss": 0.1235,
+      "step": 2880
+    },
+    {
+      "epoch": 0.9248,
+      "grad_norm": 1.6002197265625,
+      "learning_rate": 7.9177897574124e-06,
+      "loss": 0.1105,
+      "step": 2890
+    },
+    {
+      "epoch": 0.928,
+      "grad_norm": 1.2249375581741333,
+      "learning_rate": 7.580862533692723e-06,
+      "loss": 0.1228,
+      "step": 2900
+    },
+    {
+      "epoch": 0.9312,
+      "grad_norm": 2.2637360095977783,
+      "learning_rate": 7.243935309973046e-06,
+      "loss": 0.1197,
+      "step": 2910
+    },
+    {
+      "epoch": 0.9344,
+      "grad_norm": 2.7617554664611816,
+      "learning_rate": 6.90700808625337e-06,
+      "loss": 0.1176,
+      "step": 2920
+    },
+    {
+      "epoch": 0.9376,
+      "grad_norm": 1.5859897136688232,
+      "learning_rate": 6.570080862533692e-06,
+      "loss": 0.114,
+      "step": 2930
+    },
+    {
+      "epoch": 0.9408,
+      "grad_norm": 1.2503385543823242,
+      "learning_rate": 6.233153638814016e-06,
+      "loss": 0.1136,
+      "step": 2940
+    },
+    {
+      "epoch": 0.944,
+      "grad_norm": 1.396317720413208,
+      "learning_rate": 5.89622641509434e-06,
+      "loss": 0.1168,
+      "step": 2950
+    },
+    {
+      "epoch": 0.9472,
+      "grad_norm": 2.3643596172332764,
+      "learning_rate": 5.5592991913746634e-06,
+      "loss": 0.1174,
+      "step": 2960
+    },
+    {
+      "epoch": 0.9504,
+      "grad_norm": 1.6137357950210571,
+      "learning_rate": 5.222371967654987e-06,
+      "loss": 0.1299,
+      "step": 2970
+    },
+    {
+      "epoch": 0.9536,
+      "grad_norm": 1.7156883478164673,
+      "learning_rate": 4.88544474393531e-06,
+      "loss": 0.1137,
+      "step": 2980
+    },
+    {
+      "epoch": 0.9568,
+      "grad_norm": 1.6474695205688477,
+      "learning_rate": 4.548517520215634e-06,
+      "loss": 0.1153,
+      "step": 2990
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 1.7834662199020386,
+      "learning_rate": 4.211590296495957e-06,
+      "loss": 0.1169,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9632,
+      "grad_norm": 2.008061647415161,
+      "learning_rate": 3.8746630727762805e-06,
+      "loss": 0.1066,
+      "step": 3010
+    },
+    {
+      "epoch": 0.9664,
+      "grad_norm": 1.6721856594085693,
+      "learning_rate": 3.5377358490566038e-06,
+      "loss": 0.1222,
+      "step": 3020
+    },
+    {
+      "epoch": 0.9696,
+      "grad_norm": 1.7396196126937866,
+      "learning_rate": 3.200808625336928e-06,
+      "loss": 0.1173,
+      "step": 3030
+    },
+    {
+      "epoch": 0.9728,
+      "grad_norm": 2.110220432281494,
+      "learning_rate": 2.8638814016172507e-06,
+      "loss": 0.1202,
+      "step": 3040
+    },
+    {
+      "epoch": 0.976,
+      "grad_norm": 1.8802578449249268,
+      "learning_rate": 2.5269541778975744e-06,
+      "loss": 0.1177,
+      "step": 3050
+    },
+    {
+      "epoch": 0.9792,
+      "grad_norm": 2.8264000415802,
+      "learning_rate": 2.1900269541778976e-06,
+      "loss": 0.1215,
+      "step": 3060
+    },
+    {
+      "epoch": 0.9824,
+      "grad_norm": 1.3315201997756958,
+      "learning_rate": 1.853099730458221e-06,
+      "loss": 0.1123,
+      "step": 3070
+    },
+    {
+      "epoch": 0.9856,
+      "grad_norm": 1.5331279039382935,
+      "learning_rate": 1.5161725067385445e-06,
+      "loss": 0.1185,
+      "step": 3080
+    },
+    {
+      "epoch": 0.9888,
+      "grad_norm": 1.5397155284881592,
+      "learning_rate": 1.179245283018868e-06,
+      "loss": 0.1191,
+      "step": 3090
+    },
+    {
+      "epoch": 0.992,
+      "grad_norm": 1.5110286474227905,
+      "learning_rate": 8.423180592991913e-07,
+      "loss": 0.1085,
+      "step": 3100
+    },
+    {
+      "epoch": 0.9952,
+      "grad_norm": 1.888731598854065,
+      "learning_rate": 5.053908355795148e-07,
+      "loss": 0.1234,
+      "step": 3110
+    },
+    {
+      "epoch": 0.9984,
+      "grad_norm": 2.0192277431488037,
+      "learning_rate": 1.684636118598383e-07,
+      "loss": 0.11,
+      "step": 3120
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 3125,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 32,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-3125/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e9329772e53563fcdb638af8c623193bcd649c78e663bd5579313b162d75ece3
+size 5304

config.json

@@ -0,0 +1,40 @@
+{
+  "_name_or_path": "gpt2-medium",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 1024,
+  "n_head": 16,
+  "n_inner": null,
+  "n_layer": 24,
+  "n_positions": 1024,
+  "n_special": 0,
+  "predict_special_tokens": true,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
+  "transformers_version": "4.45.2",
+  "use_cache": true,
+  "vocab_size": 50257
+}

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,22 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

training_logs.csv

@@ -0,0 +1,314 @@
+step,training_loss,grad_norm
+10,22.4391,9.999999046325684
+20,18.5406,9.999999046325684
+30,18.8755,10.0
+40,18.0102,9.999998092651367
+50,15.1105,10.0
+60,14.5174,9.999999046325684
+70,12.4194,10.0
+80,10.4174,9.999999046325684
+90,9.3693,9.999999046325684
+100,8.6031,9.999999046325684
+110,6.5878,10.0
+120,6.7467,10.0
+130,5.6174,10.0
+140,5.1907,10.0
+150,4.9405,9.999999046325684
+160,4.4356,10.0
+170,4.1208,9.999999046325684
+180,4.1031,9.999998092651367
+190,4.0036,10.000000953674316
+200,3.4382,10.0
+210,3.4154,9.999999046325684
+220,3.1193,9.999998092651367
+230,3.2772,9.999999046325684
+240,3.0657,9.999999046325684
+250,2.972,8.761540412902832
+260,2.9124,9.999999046325684
+270,3.0919,9.999998092651367
+280,2.774,9.999999046325684
+290,2.5328,9.565616607666016
+300,2.9905,8.902998924255371
+310,2.6495,9.999999046325684
+320,2.5586,9.722219467163086
+330,2.3115,9.999999046325684
+340,2.1889,9.999999046325684
+350,2.253,8.74697494506836
+360,2.0232,7.3374552726745605
+370,1.9705,9.001267433166504
+380,1.9923,9.275607109069824
+390,1.7894,9.999998092651367
+400,1.8159,10.0
+410,1.7174,7.648778915405273
+420,1.5952,6.330312728881836
+430,1.6767,8.466596603393555
+440,1.4808,6.782797336578369
+450,1.4708,5.610926628112793
+460,1.5098,6.343318939208984
+470,1.367,6.048929214477539
+480,1.3129,5.7954630851745605
+490,1.3068,6.33234977722168
+500,1.3866,5.885000705718994
+510,1.2557,7.096286773681641
+520,1.324,7.024704933166504
+530,1.1659,6.996052265167236
+540,1.2756,7.000609397888184
+550,1.2388,7.971796989440918
+560,1.2079,5.358087539672852
+570,1.2221,9.458792686462402
+580,1.1891,5.870639801025391
+590,1.159,6.394337177276611
+600,1.1628,8.1649808883667
+610,1.0305,5.609042167663574
+620,1.1044,5.179653644561768
+630,1.023,7.2446699142456055
+640,1.0946,9.999999046325684
+650,1.0212,8.831192970275879
+660,1.0434,6.253108978271484
+670,1.0539,7.159657001495361
+680,0.9833,6.7119059562683105
+690,0.9824,6.3738555908203125
+700,0.9474,5.861403465270996
+710,0.9043,5.485085487365723
+720,0.9085,6.042101860046387
+730,0.9097,4.15228796005249
+740,0.9379,6.994100093841553
+750,0.8864,5.987849235534668
+760,0.8352,5.018003463745117
+770,0.8373,5.481317520141602
+780,0.8507,5.776020526885986
+790,0.9298,9.179359436035156
+800,0.921,6.685598850250244
+810,0.7999,5.743672847747803
+820,0.8054,5.498839855194092
+830,0.7677,6.5647759437561035
+840,0.8012,8.668718338012695
+850,0.7835,9.059419631958008
+860,0.7771,7.959586143493652
+870,0.7368,6.478748321533203
+880,0.7089,4.239030361175537
+890,0.7317,9.278738975524902
+900,0.722,6.447314262390137
+910,0.7324,5.464788913726807
+920,0.6865,7.056868553161621
+930,0.685,6.860642433166504
+940,0.642,5.985160827636719
+950,0.6923,4.997369766235352
+960,0.5946,4.959667205810547
+970,0.6428,3.164045810699463
+980,0.6144,6.073975086212158
+990,0.5612,5.7967658042907715
+1000,0.6135,7.2569193840026855
+1010,0.5547,6.437429428100586
+1020,0.5782,7.040581703186035
+1030,0.5826,4.126226425170898
+1040,0.5487,4.131496429443359
+1050,0.6073,5.120275020599365
+1060,0.538,7.478310585021973
+1070,0.5305,6.256986618041992
+1080,0.6043,5.667956829071045
+1090,0.5626,6.5927629470825195
+1100,0.4975,3.64158034324646
+1110,0.4688,5.662642955780029
+1120,0.5197,5.656427383422852
+1130,0.4557,4.796627521514893
+1140,0.4797,3.5207388401031494
+1150,0.4477,3.8315589427948
+1160,0.4506,5.227057456970215
+1170,0.4408,6.755670547485352
+1180,0.4378,4.697142124176025
+1190,0.4559,6.8359694480896
+1200,0.3792,4.0120954513549805
+1210,0.4224,4.8486223220825195
+1220,0.4128,4.273190975189209
+1230,0.4054,5.036397457122803
+1240,0.4142,4.676331520080566
+1250,0.3809,5.700189590454102
+1260,0.3646,3.157870292663574
+1270,0.3865,4.610825061798096
+1280,0.3755,5.093286991119385
+1290,0.4263,6.79491662979126
+1300,0.3855,4.933012962341309
+1310,0.3651,2.9703903198242188
+1320,0.3714,2.5094480514526367
+1330,0.3566,7.232475757598877
+1340,0.3918,3.515270233154297
+1350,0.3426,4.976472854614258
+1360,0.3565,3.9805610179901123
+1370,0.321,4.896690845489502
+1380,0.3153,4.799086570739746
+1390,0.3124,6.445730686187744
+1400,0.2992,3.805938720703125
+1410,0.3244,4.300934314727783
+1420,0.2993,5.322298526763916
+1430,0.3228,3.0863025188446045
+1440,0.2901,2.8616793155670166
+1450,0.2827,2.0354461669921875
+1460,0.2676,3.377239465713501
+1470,0.2909,5.67920446395874
+1480,0.3126,3.444645881652832
+1490,0.2749,3.0336413383483887
+1500,0.2601,3.9594080448150635
+1510,0.2802,4.596884250640869
+1520,0.2885,4.85414981842041
+1530,0.2758,4.045989036560059
+1540,0.2662,4.910004615783691
+1550,0.2358,2.3728251457214355
+1560,0.2588,4.005071640014648
+1570,0.2581,2.8403639793395996
+1580,0.2571,3.1361353397369385
+1590,0.2336,4.602989196777344
+1600,0.2522,2.989924907684326
+1610,0.2687,2.5031306743621826
+1620,0.247,3.078021287918091
+1630,0.2142,2.928231954574585
+1640,0.2244,2.902031183242798
+1650,0.2215,3.3071529865264893
+1660,0.2194,4.961091041564941
+1670,0.2351,1.9838968515396118
+1680,0.2223,2.13077712059021
+1690,0.1995,2.2034871578216553
+1700,0.21,2.9130163192749023
+1710,0.2116,3.775465488433838
+1720,0.2039,2.462954521179199
+1730,0.2043,2.234292984008789
+1740,0.2069,4.2636003494262695
+1750,0.2034,2.1068809032440186
+1760,0.2117,1.9375625848770142
+1770,0.192,2.2950830459594727
+1780,0.1947,2.402663469314575
+1790,0.2201,2.897888422012329
+1800,0.2113,2.6517560482025146
+1810,0.2017,1.8639112710952759
+1820,0.181,2.103375196456909
+1830,0.1982,3.1469459533691406
+1840,0.1896,2.1390271186828613
+1850,0.1892,3.2703566551208496
+1860,0.1712,1.4746158123016357
+1870,0.1759,2.221888303756714
+1880,0.1992,3.779874801635742
+1890,0.1763,2.6584346294403076
+1900,0.1881,3.278395175933838
+1910,0.1906,2.540827512741089
+1920,0.181,3.8679630756378174
+1930,0.1702,1.833381175994873
+1940,0.1733,2.688124895095825
+1950,0.1734,2.145625591278076
+1960,0.1648,3.425027370452881
+1970,0.1713,2.4815447330474854
+1980,0.1899,4.090925216674805
+1990,0.1631,2.073986768722534
+2000,0.1801,2.5611414909362793
+2010,0.1903,2.3403127193450928
+2020,0.1573,1.810889482498169
+2030,0.1673,2.551084280014038
+2040,0.175,1.5169024467468262
+2050,0.1734,2.4477052688598633
+2060,0.1743,2.7971584796905518
+2070,0.1558,1.5752153396606445
+2080,0.1702,2.5388197898864746
+2090,0.1485,3.4686882495880127
+2100,0.15,1.6380963325500488
+2110,0.1811,1.6580733060836792
+2120,0.1471,3.220553398132324
+2130,0.1735,3.0005507469177246
+2140,0.1616,3.4063844680786133
+2150,0.1561,2.0916380882263184
+2160,0.1496,2.4332773685455322
+2170,0.1593,1.630477786064148
+2180,0.1552,1.7820992469787598
+2190,0.1567,2.1285452842712402
+2200,0.1391,1.402932047843933
+2210,0.1784,2.5307276248931885
+2220,0.1447,2.1182377338409424
+2230,0.1488,2.1142852306365967
+2240,0.1382,1.8930586576461792
+2250,0.1331,2.6080002784729004
+2260,0.1362,1.5683133602142334
+2270,0.1417,2.0423901081085205
+2280,0.1226,1.4694466590881348
+2290,0.1255,2.6974313259124756
+2300,0.1403,1.8697987794876099
+2310,0.1283,2.2099053859710693
+2320,0.1483,1.6549280881881714
+2330,0.1391,1.6001911163330078
+2340,0.1527,2.202669620513916
+2350,0.1398,2.218289375305176
+2360,0.1407,1.7072354555130005
+2370,0.1538,1.6652235984802246
+2380,0.132,1.6194990873336792
+2390,0.1393,1.8026024103164673
+2400,0.1372,2.7763803005218506
+2410,0.1286,1.5770050287246704
+2420,0.129,2.233755350112915
+2430,0.1319,2.8266842365264893
+2440,0.1282,2.37268328666687
+2450,0.1258,2.108520269393921
+2460,0.1365,2.3609049320220947
+2470,0.1257,1.553999423980713
+2480,0.135,2.2637341022491455
+2490,0.1305,2.061565637588501
+2500,0.1232,1.9381234645843506
+2510,0.1301,2.815253496170044
+2520,0.1229,1.929908037185669
+2530,0.1308,1.764770269393921
+2540,0.1274,2.756413698196411
+2550,0.1305,2.02465558052063
+2560,0.1241,2.436875581741333
+2570,0.1385,1.831134557723999
+2580,0.1237,1.4205396175384521
+2590,0.1246,2.1751766204833984
+2600,0.1367,1.4494165182113647
+2610,0.1215,1.5519049167633057
+2620,0.1213,1.6617779731750488
+2630,0.124,2.4521913528442383
+2640,0.1361,2.1117968559265137
+2650,0.1183,1.941954255104065
+2660,0.1228,2.1985981464385986
+2670,0.131,1.612240195274353
+2680,0.1178,1.4507054090499878
+2690,0.1214,1.6707614660263062
+2700,0.1264,1.8366183042526245
+2710,0.1249,1.4520529508590698
+2720,0.1181,1.988840937614441
+2730,0.1187,1.2967309951782227
+2740,0.1148,1.501754641532898
+2750,0.1316,1.7325700521469116
+2760,0.1894,1.819923758506775
+2770,0.1177,2.5763540267944336
+2780,0.1166,2.4021575450897217
+2790,0.116,2.2571208477020264
+2800,0.1203,2.1551597118377686
+2810,0.1281,1.2261362075805664
+2820,0.1069,2.0020546913146973
+2830,0.1092,1.861535668373108
+2840,0.1243,1.7456763982772827
+2850,0.1134,2.4167239665985107
+2860,0.1081,1.368345022201538
+2870,0.1067,1.4125988483428955
+2880,0.1235,1.7639960050582886
+2890,0.1105,1.6002197265625
+2900,0.1228,1.2249375581741333
+2910,0.1197,2.2637360095977783
+2920,0.1176,2.7617554664611816
+2930,0.114,1.5859897136688232
+2940,0.1136,1.2503385543823242
+2950,0.1168,1.396317720413208
+2960,0.1174,2.3643596172332764
+2970,0.1299,1.6137357950210571
+2980,0.1137,1.7156883478164673
+2990,0.1153,1.6474695205688477
+3000,0.1169,1.7834662199020386
+3010,0.1066,2.008061647415161
+3020,0.1222,1.6721856594085693
+3030,0.1173,1.7396196126937866
+3040,0.1202,2.110220432281494
+3050,0.1177,1.8802578449249268
+3060,0.1215,2.8264000415802
+3070,0.1123,1.3315201997756958
+3080,0.1185,1.5331279039382935
+3090,0.1191,1.5397155284881592
+3100,0.1085,1.5110286474227905
+3110,0.1234,1.888731598854065
+3120,0.11,2.0192277431488037
+3125,nan,nan