Initial commit.

README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: mistralai/Mistral-7B-v0.3
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

adapter_config.json

@@ -0,0 +1,35 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "../ckpts/Mistral-7B-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "up_proj",
+    "v_proj",
+    "lm_head",
+    "gate_proj",
+    "k_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b7e86b85bd4fd4ddabb7a34a8773f7f88b2b163326384dd3e5afc554e02a65ce
+size 353562640

checkpoint-100/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ../ckpts/Mistral-7B-v0.3
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

checkpoint-100/adapter_config.json

@@ -0,0 +1,35 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "../ckpts/Mistral-7B-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "up_proj",
+    "v_proj",
+    "lm_head",
+    "gate_proj",
+    "k_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-100/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7fb3e8a039de3dcbd93cc1fde0b17a0fa3892b213894cc1f8afa73e2f1e15841
+size 353562640

checkpoint-100/trainer_state.json

@@ -0,0 +1,733 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.8533333333333334,
+  "eval_steps": 500,
+  "global_step": 100,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.008533333333333334,
+      "grad_norm": 250.03163081148352,
+      "learning_rate": 0.0,
+      "loss": 9.7972,
+      "step": 1
+    },
+    {
+      "epoch": 0.017066666666666667,
+      "grad_norm": 248.51781720582932,
+      "learning_rate": 3.010299956639811e-07,
+      "loss": 9.6851,
+      "step": 2
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 242.7004728698525,
+      "learning_rate": 4.771212547196623e-07,
+      "loss": 9.7332,
+      "step": 3
+    },
+    {
+      "epoch": 0.034133333333333335,
+      "grad_norm": 244.87426620614914,
+      "learning_rate": 6.020599913279622e-07,
+      "loss": 9.6243,
+      "step": 4
+    },
+    {
+      "epoch": 0.042666666666666665,
+      "grad_norm": 243.51473421797294,
+      "learning_rate": 6.989700043360186e-07,
+      "loss": 9.6145,
+      "step": 5
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 237.0939456470789,
+      "learning_rate": 7.781512503836435e-07,
+      "loss": 9.5558,
+      "step": 6
+    },
+    {
+      "epoch": 0.05973333333333333,
+      "grad_norm": 246.42426416396174,
+      "learning_rate": 8.450980400142567e-07,
+      "loss": 9.5691,
+      "step": 7
+    },
+    {
+      "epoch": 0.06826666666666667,
+      "grad_norm": 234.25935480253506,
+      "learning_rate": 9.030899869919433e-07,
+      "loss": 9.4209,
+      "step": 8
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 239.2983885236171,
+      "learning_rate": 9.542425094393247e-07,
+      "loss": 9.4733,
+      "step": 9
+    },
+    {
+      "epoch": 0.08533333333333333,
+      "grad_norm": 236.6105466399374,
+      "learning_rate": 9.999999999999997e-07,
+      "loss": 9.5095,
+      "step": 10
+    },
+    {
+      "epoch": 0.09386666666666667,
+      "grad_norm": 231.46864663491684,
+      "learning_rate": 1.0413926851582248e-06,
+      "loss": 9.1118,
+      "step": 11
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 231.50741585044312,
+      "learning_rate": 1.0791812460476246e-06,
+      "loss": 9.0775,
+      "step": 12
+    },
+    {
+      "epoch": 0.11093333333333333,
+      "grad_norm": 232.88580221859812,
+      "learning_rate": 1.1139433523068364e-06,
+      "loss": 8.7966,
+      "step": 13
+    },
+    {
+      "epoch": 0.11946666666666667,
+      "grad_norm": 227.17208510290166,
+      "learning_rate": 1.1461280356782378e-06,
+      "loss": 8.6432,
+      "step": 14
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 222.61277865106936,
+      "learning_rate": 1.176091259055681e-06,
+      "loss": 8.6176,
+      "step": 15
+    },
+    {
+      "epoch": 0.13653333333333334,
+      "grad_norm": 216.5108794899231,
+      "learning_rate": 1.2041199826559244e-06,
+      "loss": 8.1218,
+      "step": 16
+    },
+    {
+      "epoch": 0.14506666666666668,
+      "grad_norm": 219.20955299667028,
+      "learning_rate": 1.230448921378274e-06,
+      "loss": 8.0343,
+      "step": 17
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 199.72085988949678,
+      "learning_rate": 1.2552725051033058e-06,
+      "loss": 7.964,
+      "step": 18
+    },
+    {
+      "epoch": 0.16213333333333332,
+      "grad_norm": 200.24268479147997,
+      "learning_rate": 1.2787536009528286e-06,
+      "loss": 7.647,
+      "step": 19
+    },
+    {
+      "epoch": 0.17066666666666666,
+      "grad_norm": 186.85104361496084,
+      "learning_rate": 1.301029995663981e-06,
+      "loss": 7.3774,
+      "step": 20
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 182.71120056663193,
+      "learning_rate": 1.322219294733919e-06,
+      "loss": 7.1637,
+      "step": 21
+    },
+    {
+      "epoch": 0.18773333333333334,
+      "grad_norm": 182.76003520668846,
+      "learning_rate": 1.3424226808222062e-06,
+      "loss": 7.012,
+      "step": 22
+    },
+    {
+      "epoch": 0.19626666666666667,
+      "grad_norm": 168.48191088919606,
+      "learning_rate": 1.3617278360175927e-06,
+      "loss": 6.6768,
+      "step": 23
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 156.229917459413,
+      "learning_rate": 1.3802112417116059e-06,
+      "loss": 6.7099,
+      "step": 24
+    },
+    {
+      "epoch": 0.21333333333333335,
+      "grad_norm": 154.37179905588326,
+      "learning_rate": 1.3979400086720373e-06,
+      "loss": 6.4922,
+      "step": 25
+    },
+    {
+      "epoch": 0.22186666666666666,
+      "grad_norm": 143.71528792851163,
+      "learning_rate": 1.4149733479708177e-06,
+      "loss": 6.1601,
+      "step": 26
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 146.34965888668822,
+      "learning_rate": 1.431363764158987e-06,
+      "loss": 5.8541,
+      "step": 27
+    },
+    {
+      "epoch": 0.23893333333333333,
+      "grad_norm": 121.92596290244133,
+      "learning_rate": 1.4471580313422189e-06,
+      "loss": 5.4416,
+      "step": 28
+    },
+    {
+      "epoch": 0.24746666666666667,
+      "grad_norm": 120.75132633847133,
+      "learning_rate": 1.4623979978989559e-06,
+      "loss": 5.1664,
+      "step": 29
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 127.72268390687302,
+      "learning_rate": 1.477121254719662e-06,
+      "loss": 4.9839,
+      "step": 30
+    },
+    {
+      "epoch": 0.26453333333333334,
+      "grad_norm": 101.32005196251875,
+      "learning_rate": 1.4913616938342723e-06,
+      "loss": 5.1121,
+      "step": 31
+    },
+    {
+      "epoch": 0.2730666666666667,
+      "grad_norm": 90.5009306970445,
+      "learning_rate": 1.5051499783199057e-06,
+      "loss": 4.3866,
+      "step": 32
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 84.35234360166734,
+      "learning_rate": 1.5185139398778872e-06,
+      "loss": 4.4437,
+      "step": 33
+    },
+    {
+      "epoch": 0.29013333333333335,
+      "grad_norm": 75.2060158778455,
+      "learning_rate": 1.5314789170422548e-06,
+      "loss": 4.3798,
+      "step": 34
+    },
+    {
+      "epoch": 0.2986666666666667,
+      "grad_norm": 73.14821012527409,
+      "learning_rate": 1.544068044350275e-06,
+      "loss": 4.1696,
+      "step": 35
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 65.68340407068212,
+      "learning_rate": 1.556302500767287e-06,
+      "loss": 3.8687,
+      "step": 36
+    },
+    {
+      "epoch": 0.3157333333333333,
+      "grad_norm": 63.72172703320077,
+      "learning_rate": 1.5682017240669948e-06,
+      "loss": 3.9212,
+      "step": 37
+    },
+    {
+      "epoch": 0.32426666666666665,
+      "grad_norm": 55.96549796875415,
+      "learning_rate": 1.57978359661681e-06,
+      "loss": 3.7807,
+      "step": 38
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 50.34103179735071,
+      "learning_rate": 1.5910646070264987e-06,
+      "loss": 3.6901,
+      "step": 39
+    },
+    {
+      "epoch": 0.3413333333333333,
+      "grad_norm": 53.230164751324274,
+      "learning_rate": 1.602059991327962e-06,
+      "loss": 3.6057,
+      "step": 40
+    },
+    {
+      "epoch": 0.34986666666666666,
+      "grad_norm": 42.48618083004681,
+      "learning_rate": 1.6127838567197353e-06,
+      "loss": 3.5198,
+      "step": 41
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 39.512047584209995,
+      "learning_rate": 1.6232492903979003e-06,
+      "loss": 3.4435,
+      "step": 42
+    },
+    {
+      "epoch": 0.36693333333333333,
+      "grad_norm": 37.88712129419359,
+      "learning_rate": 1.633468455579586e-06,
+      "loss": 3.2061,
+      "step": 43
+    },
+    {
+      "epoch": 0.37546666666666667,
+      "grad_norm": 35.24892126286012,
+      "learning_rate": 1.643452676486187e-06,
+      "loss": 3.2579,
+      "step": 44
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 33.04491165858123,
+      "learning_rate": 1.6532125137753431e-06,
+      "loss": 2.8525,
+      "step": 45
+    },
+    {
+      "epoch": 0.39253333333333335,
+      "grad_norm": 31.165406009520897,
+      "learning_rate": 1.6627578316815738e-06,
+      "loss": 3.1049,
+      "step": 46
+    },
+    {
+      "epoch": 0.4010666666666667,
+      "grad_norm": 28.391582735290466,
+      "learning_rate": 1.672097857935717e-06,
+      "loss": 2.6253,
+      "step": 47
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 25.829470766134442,
+      "learning_rate": 1.6812412373755868e-06,
+      "loss": 2.9859,
+      "step": 48
+    },
+    {
+      "epoch": 0.41813333333333336,
+      "grad_norm": 24.083581978153447,
+      "learning_rate": 1.6901960800285134e-06,
+      "loss": 2.6692,
+      "step": 49
+    },
+    {
+      "epoch": 0.4266666666666667,
+      "grad_norm": 22.867521136957954,
+      "learning_rate": 1.6989700043360184e-06,
+      "loss": 2.8821,
+      "step": 50
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 19.900767003905905,
+      "learning_rate": 1.707570176097936e-06,
+      "loss": 2.4392,
+      "step": 51
+    },
+    {
+      "epoch": 0.4437333333333333,
+      "grad_norm": 18.380866742540494,
+      "learning_rate": 1.716003343634799e-06,
+      "loss": 2.6001,
+      "step": 52
+    },
+    {
+      "epoch": 0.45226666666666665,
+      "grad_norm": 18.893434932402364,
+      "learning_rate": 1.7242758696007888e-06,
+      "loss": 2.4671,
+      "step": 53
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 17.676419618997368,
+      "learning_rate": 1.7323937598229684e-06,
+      "loss": 2.6264,
+      "step": 54
+    },
+    {
+      "epoch": 0.4693333333333333,
+      "grad_norm": 16.505413670063586,
+      "learning_rate": 1.7403626894942437e-06,
+      "loss": 2.5316,
+      "step": 55
+    },
+    {
+      "epoch": 0.47786666666666666,
+      "grad_norm": 16.56193892770173,
+      "learning_rate": 1.7481880270062002e-06,
+      "loss": 2.4163,
+      "step": 56
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 14.824844560242497,
+      "learning_rate": 1.7558748556724912e-06,
+      "loss": 2.2749,
+      "step": 57
+    },
+    {
+      "epoch": 0.49493333333333334,
+      "grad_norm": 16.122860315584806,
+      "learning_rate": 1.7634279935629368e-06,
+      "loss": 2.3755,
+      "step": 58
+    },
+    {
+      "epoch": 0.5034666666666666,
+      "grad_norm": 14.99798519590293,
+      "learning_rate": 1.7708520116421439e-06,
+      "loss": 2.2518,
+      "step": 59
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 14.833498207347544,
+      "learning_rate": 1.7781512503836432e-06,
+      "loss": 2.2296,
+      "step": 60
+    },
+    {
+      "epoch": 0.5205333333333333,
+      "grad_norm": 14.084920820041408,
+      "learning_rate": 1.7853298350107666e-06,
+      "loss": 1.9893,
+      "step": 61
+    },
+    {
+      "epoch": 0.5290666666666667,
+      "grad_norm": 14.495022149491797,
+      "learning_rate": 1.7923916894982536e-06,
+      "loss": 1.9802,
+      "step": 62
+    },
+    {
+      "epoch": 0.5376,
+      "grad_norm": 15.49634090425538,
+      "learning_rate": 1.7993405494535814e-06,
+      "loss": 1.8676,
+      "step": 63
+    },
+    {
+      "epoch": 0.5461333333333334,
+      "grad_norm": 16.434037108501176,
+      "learning_rate": 1.8061799739838866e-06,
+      "loss": 1.9443,
+      "step": 64
+    },
+    {
+      "epoch": 0.5546666666666666,
+      "grad_norm": 17.17498656692979,
+      "learning_rate": 1.8129133566428552e-06,
+      "loss": 1.9935,
+      "step": 65
+    },
+    {
+      "epoch": 0.5632,
+      "grad_norm": 18.914832117241684,
+      "learning_rate": 1.8195439355418683e-06,
+      "loss": 1.8664,
+      "step": 66
+    },
+    {
+      "epoch": 0.5717333333333333,
+      "grad_norm": 18.78089008208585,
+      "learning_rate": 1.826074802700826e-06,
+      "loss": 1.7597,
+      "step": 67
+    },
+    {
+      "epoch": 0.5802666666666667,
+      "grad_norm": 18.908370877942197,
+      "learning_rate": 1.8325089127062361e-06,
+      "loss": 1.5813,
+      "step": 68
+    },
+    {
+      "epoch": 0.5888,
+      "grad_norm": 22.558658346768986,
+      "learning_rate": 1.8388490907372552e-06,
+      "loss": 1.598,
+      "step": 69
+    },
+    {
+      "epoch": 0.5973333333333334,
+      "grad_norm": 25.269744707032434,
+      "learning_rate": 1.8450980400142566e-06,
+      "loss": 1.642,
+      "step": 70
+    },
+    {
+      "epoch": 0.6058666666666667,
+      "grad_norm": 29.404854136304497,
+      "learning_rate": 1.851258348719075e-06,
+      "loss": 1.5158,
+      "step": 71
+    },
+    {
+      "epoch": 0.6144,
+      "grad_norm": 28.123079361503738,
+      "learning_rate": 1.857332496431268e-06,
+      "loss": 1.316,
+      "step": 72
+    },
+    {
+      "epoch": 0.6229333333333333,
+      "grad_norm": 33.051693280540675,
+      "learning_rate": 1.8633228601204554e-06,
+      "loss": 1.329,
+      "step": 73
+    },
+    {
+      "epoch": 0.6314666666666666,
+      "grad_norm": 34.63716787552787,
+      "learning_rate": 1.8692317197309759e-06,
+      "loss": 1.2845,
+      "step": 74
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 39.96989376051133,
+      "learning_rate": 1.8750612633916996e-06,
+      "loss": 1.3346,
+      "step": 75
+    },
+    {
+      "epoch": 0.6485333333333333,
+      "grad_norm": 33.72574540792705,
+      "learning_rate": 1.880813592280791e-06,
+      "loss": 1.1742,
+      "step": 76
+    },
+    {
+      "epoch": 0.6570666666666667,
+      "grad_norm": 26.56407853765471,
+      "learning_rate": 1.8864907251724815e-06,
+      "loss": 1.0382,
+      "step": 77
+    },
+    {
+      "epoch": 0.6656,
+      "grad_norm": 17.783020253212666,
+      "learning_rate": 1.89209460269048e-06,
+      "loss": 1.0619,
+      "step": 78
+    },
+    {
+      "epoch": 0.6741333333333334,
+      "grad_norm": 12.089890448213527,
+      "learning_rate": 1.897627091290441e-06,
+      "loss": 1.0258,
+      "step": 79
+    },
+    {
+      "epoch": 0.6826666666666666,
+      "grad_norm": 20.871344149138956,
+      "learning_rate": 1.903089986991943e-06,
+      "loss": 1.1339,
+      "step": 80
+    },
+    {
+      "epoch": 0.6912,
+      "grad_norm": 33.37131665653342,
+      "learning_rate": 1.9084850188786494e-06,
+      "loss": 1.1282,
+      "step": 81
+    },
+    {
+      "epoch": 0.6997333333333333,
+      "grad_norm": 10.57984322098236,
+      "learning_rate": 1.9138138523837166e-06,
+      "loss": 0.9893,
+      "step": 82
+    },
+    {
+      "epoch": 0.7082666666666667,
+      "grad_norm": 85.446420757762,
+      "learning_rate": 1.919078092376074e-06,
+      "loss": 0.9865,
+      "step": 83
+    },
+    {
+      "epoch": 0.7168,
+      "grad_norm": 107.5638346082223,
+      "learning_rate": 1.9242792860618812e-06,
+      "loss": 1.027,
+      "step": 84
+    },
+    {
+      "epoch": 0.7253333333333334,
+      "grad_norm": 101.08530683443811,
+      "learning_rate": 1.9294189257142923e-06,
+      "loss": 1.0515,
+      "step": 85
+    },
+    {
+      "epoch": 0.7338666666666667,
+      "grad_norm": 49.462738500814936,
+      "learning_rate": 1.934498451243567e-06,
+      "loss": 1.0222,
+      "step": 86
+    },
+    {
+      "epoch": 0.7424,
+      "grad_norm": 18.272590764371305,
+      "learning_rate": 1.939519252618618e-06,
+      "loss": 0.9933,
+      "step": 87
+    },
+    {
+      "epoch": 0.7509333333333333,
+      "grad_norm": 25.55179138472976,
+      "learning_rate": 1.9444826721501684e-06,
+      "loss": 0.8641,
+      "step": 88
+    },
+    {
+      "epoch": 0.7594666666666666,
+      "grad_norm": 25.348564922936834,
+      "learning_rate": 1.949390006644912e-06,
+      "loss": 0.9519,
+      "step": 89
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 11.011938725520993,
+      "learning_rate": 1.9542425094393244e-06,
+      "loss": 0.8895,
+      "step": 90
+    },
+    {
+      "epoch": 0.7765333333333333,
+      "grad_norm": 19.925323690467913,
+      "learning_rate": 1.9590413923210933e-06,
+      "loss": 0.8825,
+      "step": 91
+    },
+    {
+      "epoch": 0.7850666666666667,
+      "grad_norm": 24.37047609260986,
+      "learning_rate": 1.963787827345555e-06,
+      "loss": 0.8658,
+      "step": 92
+    },
+    {
+      "epoch": 0.7936,
+      "grad_norm": 22.06573030419392,
+      "learning_rate": 1.968482948553935e-06,
+      "loss": 0.9117,
+      "step": 93
+    },
+    {
+      "epoch": 0.8021333333333334,
+      "grad_norm": 12.409310297339818,
+      "learning_rate": 1.9731278535996984e-06,
+      "loss": 0.8574,
+      "step": 94
+    },
+    {
+      "epoch": 0.8106666666666666,
+      "grad_norm": 16.67272340712659,
+      "learning_rate": 1.9777236052888472e-06,
+      "loss": 0.7485,
+      "step": 95
+    },
+    {
+      "epoch": 0.8192,
+      "grad_norm": 18.424007742054844,
+      "learning_rate": 1.982271233039568e-06,
+      "loss": 0.8916,
+      "step": 96
+    },
+    {
+      "epoch": 0.8277333333333333,
+      "grad_norm": 13.372669052507455,
+      "learning_rate": 1.9867717342662444e-06,
+      "loss": 0.823,
+      "step": 97
+    },
+    {
+      "epoch": 0.8362666666666667,
+      "grad_norm": 20.67631223635667,
+      "learning_rate": 1.9912260756924947e-06,
+      "loss": 0.8447,
+      "step": 98
+    },
+    {
+      "epoch": 0.8448,
+      "grad_norm": 22.08691438633688,
+      "learning_rate": 1.9956351945975495e-06,
+      "loss": 0.7767,
+      "step": 99
+    },
+    {
+      "epoch": 0.8533333333333334,
+      "grad_norm": 12.061092630617507,
+      "learning_rate": 1.9999999999999995e-06,
+      "loss": 0.7029,
+      "step": 100
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 201,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 20,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.5667723770171228e+18,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-100/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e37c6d1ee3e42c776d697d1638e2504df8ccee33d99629ecefbef2fbb06988e3
+size 6840

checkpoint-120/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ../ckpts/Mistral-7B-v0.3
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

checkpoint-120/adapter_config.json

@@ -0,0 +1,35 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "../ckpts/Mistral-7B-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "up_proj",
+    "v_proj",
+    "lm_head",
+    "gate_proj",
+    "k_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-120/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f62cc54b756bcf0504e839249ec0607f04a20e6e21d0398cd30d7b6ebfe5ecdc
+size 353562640

checkpoint-120/trainer_state.json

@@ -0,0 +1,873 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.024,
+  "eval_steps": 500,
+  "global_step": 120,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.008533333333333334,
+      "grad_norm": 250.03163081148352,
+      "learning_rate": 0.0,
+      "loss": 9.7972,
+      "step": 1
+    },
+    {
+      "epoch": 0.017066666666666667,
+      "grad_norm": 248.51781720582932,
+      "learning_rate": 3.010299956639811e-07,
+      "loss": 9.6851,
+      "step": 2
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 242.7004728698525,
+      "learning_rate": 4.771212547196623e-07,
+      "loss": 9.7332,
+      "step": 3
+    },
+    {
+      "epoch": 0.034133333333333335,
+      "grad_norm": 244.87426620614914,
+      "learning_rate": 6.020599913279622e-07,
+      "loss": 9.6243,
+      "step": 4
+    },
+    {
+      "epoch": 0.042666666666666665,
+      "grad_norm": 243.51473421797294,
+      "learning_rate": 6.989700043360186e-07,
+      "loss": 9.6145,
+      "step": 5
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 237.0939456470789,
+      "learning_rate": 7.781512503836435e-07,
+      "loss": 9.5558,
+      "step": 6
+    },
+    {
+      "epoch": 0.05973333333333333,
+      "grad_norm": 246.42426416396174,
+      "learning_rate": 8.450980400142567e-07,
+      "loss": 9.5691,
+      "step": 7
+    },
+    {
+      "epoch": 0.06826666666666667,
+      "grad_norm": 234.25935480253506,
+      "learning_rate": 9.030899869919433e-07,
+      "loss": 9.4209,
+      "step": 8
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 239.2983885236171,
+      "learning_rate": 9.542425094393247e-07,
+      "loss": 9.4733,
+      "step": 9
+    },
+    {
+      "epoch": 0.08533333333333333,
+      "grad_norm": 236.6105466399374,
+      "learning_rate": 9.999999999999997e-07,
+      "loss": 9.5095,
+      "step": 10
+    },
+    {
+      "epoch": 0.09386666666666667,
+      "grad_norm": 231.46864663491684,
+      "learning_rate": 1.0413926851582248e-06,
+      "loss": 9.1118,
+      "step": 11
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 231.50741585044312,
+      "learning_rate": 1.0791812460476246e-06,
+      "loss": 9.0775,
+      "step": 12
+    },
+    {
+      "epoch": 0.11093333333333333,
+      "grad_norm": 232.88580221859812,
+      "learning_rate": 1.1139433523068364e-06,
+      "loss": 8.7966,
+      "step": 13
+    },
+    {
+      "epoch": 0.11946666666666667,
+      "grad_norm": 227.17208510290166,
+      "learning_rate": 1.1461280356782378e-06,
+      "loss": 8.6432,
+      "step": 14
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 222.61277865106936,
+      "learning_rate": 1.176091259055681e-06,
+      "loss": 8.6176,
+      "step": 15
+    },
+    {
+      "epoch": 0.13653333333333334,
+      "grad_norm": 216.5108794899231,
+      "learning_rate": 1.2041199826559244e-06,
+      "loss": 8.1218,
+      "step": 16
+    },
+    {
+      "epoch": 0.14506666666666668,
+      "grad_norm": 219.20955299667028,
+      "learning_rate": 1.230448921378274e-06,
+      "loss": 8.0343,
+      "step": 17
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 199.72085988949678,
+      "learning_rate": 1.2552725051033058e-06,
+      "loss": 7.964,
+      "step": 18
+    },
+    {
+      "epoch": 0.16213333333333332,
+      "grad_norm": 200.24268479147997,
+      "learning_rate": 1.2787536009528286e-06,
+      "loss": 7.647,
+      "step": 19
+    },
+    {
+      "epoch": 0.17066666666666666,
+      "grad_norm": 186.85104361496084,
+      "learning_rate": 1.301029995663981e-06,
+      "loss": 7.3774,
+      "step": 20
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 182.71120056663193,
+      "learning_rate": 1.322219294733919e-06,
+      "loss": 7.1637,
+      "step": 21
+    },
+    {
+      "epoch": 0.18773333333333334,
+      "grad_norm": 182.76003520668846,
+      "learning_rate": 1.3424226808222062e-06,
+      "loss": 7.012,
+      "step": 22
+    },
+    {
+      "epoch": 0.19626666666666667,
+      "grad_norm": 168.48191088919606,
+      "learning_rate": 1.3617278360175927e-06,
+      "loss": 6.6768,
+      "step": 23
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 156.229917459413,
+      "learning_rate": 1.3802112417116059e-06,
+      "loss": 6.7099,
+      "step": 24
+    },
+    {
+      "epoch": 0.21333333333333335,
+      "grad_norm": 154.37179905588326,
+      "learning_rate": 1.3979400086720373e-06,
+      "loss": 6.4922,
+      "step": 25
+    },
+    {
+      "epoch": 0.22186666666666666,
+      "grad_norm": 143.71528792851163,
+      "learning_rate": 1.4149733479708177e-06,
+      "loss": 6.1601,
+      "step": 26
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 146.34965888668822,
+      "learning_rate": 1.431363764158987e-06,
+      "loss": 5.8541,
+      "step": 27
+    },
+    {
+      "epoch": 0.23893333333333333,
+      "grad_norm": 121.92596290244133,
+      "learning_rate": 1.4471580313422189e-06,
+      "loss": 5.4416,
+      "step": 28
+    },
+    {
+      "epoch": 0.24746666666666667,
+      "grad_norm": 120.75132633847133,
+      "learning_rate": 1.4623979978989559e-06,
+      "loss": 5.1664,
+      "step": 29
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 127.72268390687302,
+      "learning_rate": 1.477121254719662e-06,
+      "loss": 4.9839,
+      "step": 30
+    },
+    {
+      "epoch": 0.26453333333333334,
+      "grad_norm": 101.32005196251875,
+      "learning_rate": 1.4913616938342723e-06,
+      "loss": 5.1121,
+      "step": 31
+    },
+    {
+      "epoch": 0.2730666666666667,
+      "grad_norm": 90.5009306970445,
+      "learning_rate": 1.5051499783199057e-06,
+      "loss": 4.3866,
+      "step": 32
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 84.35234360166734,
+      "learning_rate": 1.5185139398778872e-06,
+      "loss": 4.4437,
+      "step": 33
+    },
+    {
+      "epoch": 0.29013333333333335,
+      "grad_norm": 75.2060158778455,
+      "learning_rate": 1.5314789170422548e-06,
+      "loss": 4.3798,
+      "step": 34
+    },
+    {
+      "epoch": 0.2986666666666667,
+      "grad_norm": 73.14821012527409,
+      "learning_rate": 1.544068044350275e-06,
+      "loss": 4.1696,
+      "step": 35
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 65.68340407068212,
+      "learning_rate": 1.556302500767287e-06,
+      "loss": 3.8687,
+      "step": 36
+    },
+    {
+      "epoch": 0.3157333333333333,
+      "grad_norm": 63.72172703320077,
+      "learning_rate": 1.5682017240669948e-06,
+      "loss": 3.9212,
+      "step": 37
+    },
+    {
+      "epoch": 0.32426666666666665,
+      "grad_norm": 55.96549796875415,
+      "learning_rate": 1.57978359661681e-06,
+      "loss": 3.7807,
+      "step": 38
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 50.34103179735071,
+      "learning_rate": 1.5910646070264987e-06,
+      "loss": 3.6901,
+      "step": 39
+    },
+    {
+      "epoch": 0.3413333333333333,
+      "grad_norm": 53.230164751324274,
+      "learning_rate": 1.602059991327962e-06,
+      "loss": 3.6057,
+      "step": 40
+    },
+    {
+      "epoch": 0.34986666666666666,
+      "grad_norm": 42.48618083004681,
+      "learning_rate": 1.6127838567197353e-06,
+      "loss": 3.5198,
+      "step": 41
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 39.512047584209995,
+      "learning_rate": 1.6232492903979003e-06,
+      "loss": 3.4435,
+      "step": 42
+    },
+    {
+      "epoch": 0.36693333333333333,
+      "grad_norm": 37.88712129419359,
+      "learning_rate": 1.633468455579586e-06,
+      "loss": 3.2061,
+      "step": 43
+    },
+    {
+      "epoch": 0.37546666666666667,
+      "grad_norm": 35.24892126286012,
+      "learning_rate": 1.643452676486187e-06,
+      "loss": 3.2579,
+      "step": 44
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 33.04491165858123,
+      "learning_rate": 1.6532125137753431e-06,
+      "loss": 2.8525,
+      "step": 45
+    },
+    {
+      "epoch": 0.39253333333333335,
+      "grad_norm": 31.165406009520897,
+      "learning_rate": 1.6627578316815738e-06,
+      "loss": 3.1049,
+      "step": 46
+    },
+    {
+      "epoch": 0.4010666666666667,
+      "grad_norm": 28.391582735290466,
+      "learning_rate": 1.672097857935717e-06,
+      "loss": 2.6253,
+      "step": 47
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 25.829470766134442,
+      "learning_rate": 1.6812412373755868e-06,
+      "loss": 2.9859,
+      "step": 48
+    },
+    {
+      "epoch": 0.41813333333333336,
+      "grad_norm": 24.083581978153447,
+      "learning_rate": 1.6901960800285134e-06,
+      "loss": 2.6692,
+      "step": 49
+    },
+    {
+      "epoch": 0.4266666666666667,
+      "grad_norm": 22.867521136957954,
+      "learning_rate": 1.6989700043360184e-06,
+      "loss": 2.8821,
+      "step": 50
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 19.900767003905905,
+      "learning_rate": 1.707570176097936e-06,
+      "loss": 2.4392,
+      "step": 51
+    },
+    {
+      "epoch": 0.4437333333333333,
+      "grad_norm": 18.380866742540494,
+      "learning_rate": 1.716003343634799e-06,
+      "loss": 2.6001,
+      "step": 52
+    },
+    {
+      "epoch": 0.45226666666666665,
+      "grad_norm": 18.893434932402364,
+      "learning_rate": 1.7242758696007888e-06,
+      "loss": 2.4671,
+      "step": 53
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 17.676419618997368,
+      "learning_rate": 1.7323937598229684e-06,
+      "loss": 2.6264,
+      "step": 54
+    },
+    {
+      "epoch": 0.4693333333333333,
+      "grad_norm": 16.505413670063586,
+      "learning_rate": 1.7403626894942437e-06,
+      "loss": 2.5316,
+      "step": 55
+    },
+    {
+      "epoch": 0.47786666666666666,
+      "grad_norm": 16.56193892770173,
+      "learning_rate": 1.7481880270062002e-06,
+      "loss": 2.4163,
+      "step": 56
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 14.824844560242497,
+      "learning_rate": 1.7558748556724912e-06,
+      "loss": 2.2749,
+      "step": 57
+    },
+    {
+      "epoch": 0.49493333333333334,
+      "grad_norm": 16.122860315584806,
+      "learning_rate": 1.7634279935629368e-06,
+      "loss": 2.3755,
+      "step": 58
+    },
+    {
+      "epoch": 0.5034666666666666,
+      "grad_norm": 14.99798519590293,
+      "learning_rate": 1.7708520116421439e-06,
+      "loss": 2.2518,
+      "step": 59
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 14.833498207347544,
+      "learning_rate": 1.7781512503836432e-06,
+      "loss": 2.2296,
+      "step": 60
+    },
+    {
+      "epoch": 0.5205333333333333,
+      "grad_norm": 14.084920820041408,
+      "learning_rate": 1.7853298350107666e-06,
+      "loss": 1.9893,
+      "step": 61
+    },
+    {
+      "epoch": 0.5290666666666667,
+      "grad_norm": 14.495022149491797,
+      "learning_rate": 1.7923916894982536e-06,
+      "loss": 1.9802,
+      "step": 62
+    },
+    {
+      "epoch": 0.5376,
+      "grad_norm": 15.49634090425538,
+      "learning_rate": 1.7993405494535814e-06,
+      "loss": 1.8676,
+      "step": 63
+    },
+    {
+      "epoch": 0.5461333333333334,
+      "grad_norm": 16.434037108501176,
+      "learning_rate": 1.8061799739838866e-06,
+      "loss": 1.9443,
+      "step": 64
+    },
+    {
+      "epoch": 0.5546666666666666,
+      "grad_norm": 17.17498656692979,
+      "learning_rate": 1.8129133566428552e-06,
+      "loss": 1.9935,
+      "step": 65
+    },
+    {
+      "epoch": 0.5632,
+      "grad_norm": 18.914832117241684,
+      "learning_rate": 1.8195439355418683e-06,
+      "loss": 1.8664,
+      "step": 66
+    },
+    {
+      "epoch": 0.5717333333333333,
+      "grad_norm": 18.78089008208585,
+      "learning_rate": 1.826074802700826e-06,
+      "loss": 1.7597,
+      "step": 67
+    },
+    {
+      "epoch": 0.5802666666666667,
+      "grad_norm": 18.908370877942197,
+      "learning_rate": 1.8325089127062361e-06,
+      "loss": 1.5813,
+      "step": 68
+    },
+    {
+      "epoch": 0.5888,
+      "grad_norm": 22.558658346768986,
+      "learning_rate": 1.8388490907372552e-06,
+      "loss": 1.598,
+      "step": 69
+    },
+    {
+      "epoch": 0.5973333333333334,
+      "grad_norm": 25.269744707032434,
+      "learning_rate": 1.8450980400142566e-06,
+      "loss": 1.642,
+      "step": 70
+    },
+    {
+      "epoch": 0.6058666666666667,
+      "grad_norm": 29.404854136304497,
+      "learning_rate": 1.851258348719075e-06,
+      "loss": 1.5158,
+      "step": 71
+    },
+    {
+      "epoch": 0.6144,
+      "grad_norm": 28.123079361503738,
+      "learning_rate": 1.857332496431268e-06,
+      "loss": 1.316,
+      "step": 72
+    },
+    {
+      "epoch": 0.6229333333333333,
+      "grad_norm": 33.051693280540675,
+      "learning_rate": 1.8633228601204554e-06,
+      "loss": 1.329,
+      "step": 73
+    },
+    {
+      "epoch": 0.6314666666666666,
+      "grad_norm": 34.63716787552787,
+      "learning_rate": 1.8692317197309759e-06,
+      "loss": 1.2845,
+      "step": 74
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 39.96989376051133,
+      "learning_rate": 1.8750612633916996e-06,
+      "loss": 1.3346,
+      "step": 75
+    },
+    {
+      "epoch": 0.6485333333333333,
+      "grad_norm": 33.72574540792705,
+      "learning_rate": 1.880813592280791e-06,
+      "loss": 1.1742,
+      "step": 76
+    },
+    {
+      "epoch": 0.6570666666666667,
+      "grad_norm": 26.56407853765471,
+      "learning_rate": 1.8864907251724815e-06,
+      "loss": 1.0382,
+      "step": 77
+    },
+    {
+      "epoch": 0.6656,
+      "grad_norm": 17.783020253212666,
+      "learning_rate": 1.89209460269048e-06,
+      "loss": 1.0619,
+      "step": 78
+    },
+    {
+      "epoch": 0.6741333333333334,
+      "grad_norm": 12.089890448213527,
+      "learning_rate": 1.897627091290441e-06,
+      "loss": 1.0258,
+      "step": 79
+    },
+    {
+      "epoch": 0.6826666666666666,
+      "grad_norm": 20.871344149138956,
+      "learning_rate": 1.903089986991943e-06,
+      "loss": 1.1339,
+      "step": 80
+    },
+    {
+      "epoch": 0.6912,
+      "grad_norm": 33.37131665653342,
+      "learning_rate": 1.9084850188786494e-06,
+      "loss": 1.1282,
+      "step": 81
+    },
+    {
+      "epoch": 0.6997333333333333,
+      "grad_norm": 10.57984322098236,
+      "learning_rate": 1.9138138523837166e-06,
+      "loss": 0.9893,
+      "step": 82
+    },
+    {
+      "epoch": 0.7082666666666667,
+      "grad_norm": 85.446420757762,
+      "learning_rate": 1.919078092376074e-06,
+      "loss": 0.9865,
+      "step": 83
+    },
+    {
+      "epoch": 0.7168,
+      "grad_norm": 107.5638346082223,
+      "learning_rate": 1.9242792860618812e-06,
+      "loss": 1.027,
+      "step": 84
+    },
+    {
+      "epoch": 0.7253333333333334,
+      "grad_norm": 101.08530683443811,
+      "learning_rate": 1.9294189257142923e-06,
+      "loss": 1.0515,
+      "step": 85
+    },
+    {
+      "epoch": 0.7338666666666667,
+      "grad_norm": 49.462738500814936,
+      "learning_rate": 1.934498451243567e-06,
+      "loss": 1.0222,
+      "step": 86
+    },
+    {
+      "epoch": 0.7424,
+      "grad_norm": 18.272590764371305,
+      "learning_rate": 1.939519252618618e-06,
+      "loss": 0.9933,
+      "step": 87
+    },
+    {
+      "epoch": 0.7509333333333333,
+      "grad_norm": 25.55179138472976,
+      "learning_rate": 1.9444826721501684e-06,
+      "loss": 0.8641,
+      "step": 88
+    },
+    {
+      "epoch": 0.7594666666666666,
+      "grad_norm": 25.348564922936834,
+      "learning_rate": 1.949390006644912e-06,
+      "loss": 0.9519,
+      "step": 89
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 11.011938725520993,
+      "learning_rate": 1.9542425094393244e-06,
+      "loss": 0.8895,
+      "step": 90
+    },
+    {
+      "epoch": 0.7765333333333333,
+      "grad_norm": 19.925323690467913,
+      "learning_rate": 1.9590413923210933e-06,
+      "loss": 0.8825,
+      "step": 91
+    },
+    {
+      "epoch": 0.7850666666666667,
+      "grad_norm": 24.37047609260986,
+      "learning_rate": 1.963787827345555e-06,
+      "loss": 0.8658,
+      "step": 92
+    },
+    {
+      "epoch": 0.7936,
+      "grad_norm": 22.06573030419392,
+      "learning_rate": 1.968482948553935e-06,
+      "loss": 0.9117,
+      "step": 93
+    },
+    {
+      "epoch": 0.8021333333333334,
+      "grad_norm": 12.409310297339818,
+      "learning_rate": 1.9731278535996984e-06,
+      "loss": 0.8574,
+      "step": 94
+    },
+    {
+      "epoch": 0.8106666666666666,
+      "grad_norm": 16.67272340712659,
+      "learning_rate": 1.9777236052888472e-06,
+      "loss": 0.7485,
+      "step": 95
+    },
+    {
+      "epoch": 0.8192,
+      "grad_norm": 18.424007742054844,
+      "learning_rate": 1.982271233039568e-06,
+      "loss": 0.8916,
+      "step": 96
+    },
+    {
+      "epoch": 0.8277333333333333,
+      "grad_norm": 13.372669052507455,
+      "learning_rate": 1.9867717342662444e-06,
+      "loss": 0.823,
+      "step": 97
+    },
+    {
+      "epoch": 0.8362666666666667,
+      "grad_norm": 20.67631223635667,
+      "learning_rate": 1.9912260756924947e-06,
+      "loss": 0.8447,
+      "step": 98
+    },
+    {
+      "epoch": 0.8448,
+      "grad_norm": 22.08691438633688,
+      "learning_rate": 1.9956351945975495e-06,
+      "loss": 0.7767,
+      "step": 99
+    },
+    {
+      "epoch": 0.8533333333333334,
+      "grad_norm": 12.061092630617507,
+      "learning_rate": 1.9999999999999995e-06,
+      "loss": 0.7029,
+      "step": 100
+    },
+    {
+      "epoch": 0.8618666666666667,
+      "grad_norm": 13.563356715273038,
+      "learning_rate": 2e-06,
+      "loss": 0.6925,
+      "step": 101
+    },
+    {
+      "epoch": 0.8704,
+      "grad_norm": 15.841269394844565,
+      "learning_rate": 2e-06,
+      "loss": 0.5944,
+      "step": 102
+    },
+    {
+      "epoch": 0.8789333333333333,
+      "grad_norm": 14.854530419188935,
+      "learning_rate": 2e-06,
+      "loss": 0.6077,
+      "step": 103
+    },
+    {
+      "epoch": 0.8874666666666666,
+      "grad_norm": 12.791409733249388,
+      "learning_rate": 2e-06,
+      "loss": 0.543,
+      "step": 104
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 14.362179553886866,
+      "learning_rate": 2e-06,
+      "loss": 0.5541,
+      "step": 105
+    },
+    {
+      "epoch": 0.9045333333333333,
+      "grad_norm": 12.49157003340309,
+      "learning_rate": 2e-06,
+      "loss": 0.5295,
+      "step": 106
+    },
+    {
+      "epoch": 0.9130666666666667,
+      "grad_norm": 13.208695431789076,
+      "learning_rate": 2e-06,
+      "loss": 0.4524,
+      "step": 107
+    },
+    {
+      "epoch": 0.9216,
+      "grad_norm": 15.463267948258817,
+      "learning_rate": 2e-06,
+      "loss": 0.4159,
+      "step": 108
+    },
+    {
+      "epoch": 0.9301333333333334,
+      "grad_norm": 12.140164988871907,
+      "learning_rate": 2e-06,
+      "loss": 0.4014,
+      "step": 109
+    },
+    {
+      "epoch": 0.9386666666666666,
+      "grad_norm": 11.905397079235023,
+      "learning_rate": 2e-06,
+      "loss": 0.3618,
+      "step": 110
+    },
+    {
+      "epoch": 0.9472,
+      "grad_norm": 9.415569770415516,
+      "learning_rate": 2e-06,
+      "loss": 0.3057,
+      "step": 111
+    },
+    {
+      "epoch": 0.9557333333333333,
+      "grad_norm": 8.352190984303535,
+      "learning_rate": 2e-06,
+      "loss": 0.3047,
+      "step": 112
+    },
+    {
+      "epoch": 0.9642666666666667,
+      "grad_norm": 7.302691540894704,
+      "learning_rate": 2e-06,
+      "loss": 0.2524,
+      "step": 113
+    },
+    {
+      "epoch": 0.9728,
+      "grad_norm": 5.9832616024269045,
+      "learning_rate": 2e-06,
+      "loss": 0.2518,
+      "step": 114
+    },
+    {
+      "epoch": 0.9813333333333333,
+      "grad_norm": 4.358646236198701,
+      "learning_rate": 2e-06,
+      "loss": 0.2175,
+      "step": 115
+    },
+    {
+      "epoch": 0.9898666666666667,
+      "grad_norm": 4.760656349386409,
+      "learning_rate": 2e-06,
+      "loss": 0.2447,
+      "step": 116
+    },
+    {
+      "epoch": 0.9984,
+      "grad_norm": 3.7039196594291646,
+      "learning_rate": 2e-06,
+      "loss": 0.2033,
+      "step": 117
+    },
+    {
+      "epoch": 1.0069333333333332,
+      "grad_norm": 3.8277716025893387,
+      "learning_rate": 2e-06,
+      "loss": 0.1974,
+      "step": 118
+    },
+    {
+      "epoch": 1.0154666666666667,
+      "grad_norm": 5.741701523334058,
+      "learning_rate": 2e-06,
+      "loss": 0.2018,
+      "step": 119
+    },
+    {
+      "epoch": 1.024,
+      "grad_norm": 4.6412716809913706,
+      "learning_rate": 2e-06,
+      "loss": 0.1875,
+      "step": 120
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 201,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 20,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.88307590710349e+18,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-120/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e37c6d1ee3e42c776d697d1638e2504df8ccee33d99629ecefbef2fbb06988e3
+size 6840

checkpoint-140/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ../ckpts/Mistral-7B-v0.3
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

checkpoint-140/adapter_config.json

@@ -0,0 +1,35 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "../ckpts/Mistral-7B-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "up_proj",
+    "v_proj",
+    "lm_head",
+    "gate_proj",
+    "k_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-140/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1549ac3d375c8ad862edbfb0d25438851905a1c51b310050806b24719ecdf8f
+size 353562640

checkpoint-140/trainer_state.json

@@ -0,0 +1,1013 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.1946666666666665,
+  "eval_steps": 500,
+  "global_step": 140,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.008533333333333334,
+      "grad_norm": 250.03163081148352,
+      "learning_rate": 0.0,
+      "loss": 9.7972,
+      "step": 1
+    },
+    {
+      "epoch": 0.017066666666666667,
+      "grad_norm": 248.51781720582932,
+      "learning_rate": 3.010299956639811e-07,
+      "loss": 9.6851,
+      "step": 2
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 242.7004728698525,
+      "learning_rate": 4.771212547196623e-07,
+      "loss": 9.7332,
+      "step": 3
+    },
+    {
+      "epoch": 0.034133333333333335,
+      "grad_norm": 244.87426620614914,
+      "learning_rate": 6.020599913279622e-07,
+      "loss": 9.6243,
+      "step": 4
+    },
+    {
+      "epoch": 0.042666666666666665,
+      "grad_norm": 243.51473421797294,
+      "learning_rate": 6.989700043360186e-07,
+      "loss": 9.6145,
+      "step": 5
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 237.0939456470789,
+      "learning_rate": 7.781512503836435e-07,
+      "loss": 9.5558,
+      "step": 6
+    },
+    {
+      "epoch": 0.05973333333333333,
+      "grad_norm": 246.42426416396174,
+      "learning_rate": 8.450980400142567e-07,
+      "loss": 9.5691,
+      "step": 7
+    },
+    {
+      "epoch": 0.06826666666666667,
+      "grad_norm": 234.25935480253506,
+      "learning_rate": 9.030899869919433e-07,
+      "loss": 9.4209,
+      "step": 8
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 239.2983885236171,
+      "learning_rate": 9.542425094393247e-07,
+      "loss": 9.4733,
+      "step": 9
+    },
+    {
+      "epoch": 0.08533333333333333,
+      "grad_norm": 236.6105466399374,
+      "learning_rate": 9.999999999999997e-07,
+      "loss": 9.5095,
+      "step": 10
+    },
+    {
+      "epoch": 0.09386666666666667,
+      "grad_norm": 231.46864663491684,
+      "learning_rate": 1.0413926851582248e-06,
+      "loss": 9.1118,
+      "step": 11
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 231.50741585044312,
+      "learning_rate": 1.0791812460476246e-06,
+      "loss": 9.0775,
+      "step": 12
+    },
+    {
+      "epoch": 0.11093333333333333,
+      "grad_norm": 232.88580221859812,
+      "learning_rate": 1.1139433523068364e-06,
+      "loss": 8.7966,
+      "step": 13
+    },
+    {
+      "epoch": 0.11946666666666667,
+      "grad_norm": 227.17208510290166,
+      "learning_rate": 1.1461280356782378e-06,
+      "loss": 8.6432,
+      "step": 14
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 222.61277865106936,
+      "learning_rate": 1.176091259055681e-06,
+      "loss": 8.6176,
+      "step": 15
+    },
+    {
+      "epoch": 0.13653333333333334,
+      "grad_norm": 216.5108794899231,
+      "learning_rate": 1.2041199826559244e-06,
+      "loss": 8.1218,
+      "step": 16
+    },
+    {
+      "epoch": 0.14506666666666668,
+      "grad_norm": 219.20955299667028,
+      "learning_rate": 1.230448921378274e-06,
+      "loss": 8.0343,
+      "step": 17
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 199.72085988949678,
+      "learning_rate": 1.2552725051033058e-06,
+      "loss": 7.964,
+      "step": 18
+    },
+    {
+      "epoch": 0.16213333333333332,
+      "grad_norm": 200.24268479147997,
+      "learning_rate": 1.2787536009528286e-06,
+      "loss": 7.647,
+      "step": 19
+    },
+    {
+      "epoch": 0.17066666666666666,
+      "grad_norm": 186.85104361496084,
+      "learning_rate": 1.301029995663981e-06,
+      "loss": 7.3774,
+      "step": 20
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 182.71120056663193,
+      "learning_rate": 1.322219294733919e-06,
+      "loss": 7.1637,
+      "step": 21
+    },
+    {
+      "epoch": 0.18773333333333334,
+      "grad_norm": 182.76003520668846,
+      "learning_rate": 1.3424226808222062e-06,
+      "loss": 7.012,
+      "step": 22
+    },
+    {
+      "epoch": 0.19626666666666667,
+      "grad_norm": 168.48191088919606,
+      "learning_rate": 1.3617278360175927e-06,
+      "loss": 6.6768,
+      "step": 23
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 156.229917459413,
+      "learning_rate": 1.3802112417116059e-06,
+      "loss": 6.7099,
+      "step": 24
+    },
+    {
+      "epoch": 0.21333333333333335,
+      "grad_norm": 154.37179905588326,
+      "learning_rate": 1.3979400086720373e-06,
+      "loss": 6.4922,
+      "step": 25
+    },
+    {
+      "epoch": 0.22186666666666666,
+      "grad_norm": 143.71528792851163,
+      "learning_rate": 1.4149733479708177e-06,
+      "loss": 6.1601,
+      "step": 26
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 146.34965888668822,
+      "learning_rate": 1.431363764158987e-06,
+      "loss": 5.8541,
+      "step": 27
+    },
+    {
+      "epoch": 0.23893333333333333,
+      "grad_norm": 121.92596290244133,
+      "learning_rate": 1.4471580313422189e-06,
+      "loss": 5.4416,
+      "step": 28
+    },
+    {
+      "epoch": 0.24746666666666667,
+      "grad_norm": 120.75132633847133,
+      "learning_rate": 1.4623979978989559e-06,
+      "loss": 5.1664,
+      "step": 29
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 127.72268390687302,
+      "learning_rate": 1.477121254719662e-06,
+      "loss": 4.9839,
+      "step": 30
+    },
+    {
+      "epoch": 0.26453333333333334,
+      "grad_norm": 101.32005196251875,
+      "learning_rate": 1.4913616938342723e-06,
+      "loss": 5.1121,
+      "step": 31
+    },
+    {
+      "epoch": 0.2730666666666667,
+      "grad_norm": 90.5009306970445,
+      "learning_rate": 1.5051499783199057e-06,
+      "loss": 4.3866,
+      "step": 32
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 84.35234360166734,
+      "learning_rate": 1.5185139398778872e-06,
+      "loss": 4.4437,
+      "step": 33
+    },
+    {
+      "epoch": 0.29013333333333335,
+      "grad_norm": 75.2060158778455,
+      "learning_rate": 1.5314789170422548e-06,
+      "loss": 4.3798,
+      "step": 34
+    },
+    {
+      "epoch": 0.2986666666666667,
+      "grad_norm": 73.14821012527409,
+      "learning_rate": 1.544068044350275e-06,
+      "loss": 4.1696,
+      "step": 35
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 65.68340407068212,
+      "learning_rate": 1.556302500767287e-06,
+      "loss": 3.8687,
+      "step": 36
+    },
+    {
+      "epoch": 0.3157333333333333,
+      "grad_norm": 63.72172703320077,
+      "learning_rate": 1.5682017240669948e-06,
+      "loss": 3.9212,
+      "step": 37
+    },
+    {
+      "epoch": 0.32426666666666665,
+      "grad_norm": 55.96549796875415,
+      "learning_rate": 1.57978359661681e-06,
+      "loss": 3.7807,
+      "step": 38
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 50.34103179735071,
+      "learning_rate": 1.5910646070264987e-06,
+      "loss": 3.6901,
+      "step": 39
+    },
+    {
+      "epoch": 0.3413333333333333,
+      "grad_norm": 53.230164751324274,
+      "learning_rate": 1.602059991327962e-06,
+      "loss": 3.6057,
+      "step": 40
+    },
+    {
+      "epoch": 0.34986666666666666,
+      "grad_norm": 42.48618083004681,
+      "learning_rate": 1.6127838567197353e-06,
+      "loss": 3.5198,
+      "step": 41
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 39.512047584209995,
+      "learning_rate": 1.6232492903979003e-06,
+      "loss": 3.4435,
+      "step": 42
+    },
+    {
+      "epoch": 0.36693333333333333,
+      "grad_norm": 37.88712129419359,
+      "learning_rate": 1.633468455579586e-06,
+      "loss": 3.2061,
+      "step": 43
+    },
+    {
+      "epoch": 0.37546666666666667,
+      "grad_norm": 35.24892126286012,
+      "learning_rate": 1.643452676486187e-06,
+      "loss": 3.2579,
+      "step": 44
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 33.04491165858123,
+      "learning_rate": 1.6532125137753431e-06,
+      "loss": 2.8525,
+      "step": 45
+    },
+    {
+      "epoch": 0.39253333333333335,
+      "grad_norm": 31.165406009520897,
+      "learning_rate": 1.6627578316815738e-06,
+      "loss": 3.1049,
+      "step": 46
+    },
+    {
+      "epoch": 0.4010666666666667,
+      "grad_norm": 28.391582735290466,
+      "learning_rate": 1.672097857935717e-06,
+      "loss": 2.6253,
+      "step": 47
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 25.829470766134442,
+      "learning_rate": 1.6812412373755868e-06,
+      "loss": 2.9859,
+      "step": 48
+    },
+    {
+      "epoch": 0.41813333333333336,
+      "grad_norm": 24.083581978153447,
+      "learning_rate": 1.6901960800285134e-06,
+      "loss": 2.6692,
+      "step": 49
+    },
+    {
+      "epoch": 0.4266666666666667,
+      "grad_norm": 22.867521136957954,
+      "learning_rate": 1.6989700043360184e-06,
+      "loss": 2.8821,
+      "step": 50
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 19.900767003905905,
+      "learning_rate": 1.707570176097936e-06,
+      "loss": 2.4392,
+      "step": 51
+    },
+    {
+      "epoch": 0.4437333333333333,
+      "grad_norm": 18.380866742540494,
+      "learning_rate": 1.716003343634799e-06,
+      "loss": 2.6001,
+      "step": 52
+    },
+    {
+      "epoch": 0.45226666666666665,
+      "grad_norm": 18.893434932402364,
+      "learning_rate": 1.7242758696007888e-06,
+      "loss": 2.4671,
+      "step": 53
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 17.676419618997368,
+      "learning_rate": 1.7323937598229684e-06,
+      "loss": 2.6264,
+      "step": 54
+    },
+    {
+      "epoch": 0.4693333333333333,
+      "grad_norm": 16.505413670063586,
+      "learning_rate": 1.7403626894942437e-06,
+      "loss": 2.5316,
+      "step": 55
+    },
+    {
+      "epoch": 0.47786666666666666,
+      "grad_norm": 16.56193892770173,
+      "learning_rate": 1.7481880270062002e-06,
+      "loss": 2.4163,
+      "step": 56
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 14.824844560242497,
+      "learning_rate": 1.7558748556724912e-06,
+      "loss": 2.2749,
+      "step": 57
+    },
+    {
+      "epoch": 0.49493333333333334,
+      "grad_norm": 16.122860315584806,
+      "learning_rate": 1.7634279935629368e-06,
+      "loss": 2.3755,
+      "step": 58
+    },
+    {
+      "epoch": 0.5034666666666666,
+      "grad_norm": 14.99798519590293,
+      "learning_rate": 1.7708520116421439e-06,
+      "loss": 2.2518,
+      "step": 59
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 14.833498207347544,
+      "learning_rate": 1.7781512503836432e-06,
+      "loss": 2.2296,
+      "step": 60
+    },
+    {
+      "epoch": 0.5205333333333333,
+      "grad_norm": 14.084920820041408,
+      "learning_rate": 1.7853298350107666e-06,
+      "loss": 1.9893,
+      "step": 61
+    },
+    {
+      "epoch": 0.5290666666666667,
+      "grad_norm": 14.495022149491797,
+      "learning_rate": 1.7923916894982536e-06,
+      "loss": 1.9802,
+      "step": 62
+    },
+    {
+      "epoch": 0.5376,
+      "grad_norm": 15.49634090425538,
+      "learning_rate": 1.7993405494535814e-06,
+      "loss": 1.8676,
+      "step": 63
+    },
+    {
+      "epoch": 0.5461333333333334,
+      "grad_norm": 16.434037108501176,
+      "learning_rate": 1.8061799739838866e-06,
+      "loss": 1.9443,
+      "step": 64
+    },
+    {
+      "epoch": 0.5546666666666666,
+      "grad_norm": 17.17498656692979,
+      "learning_rate": 1.8129133566428552e-06,
+      "loss": 1.9935,
+      "step": 65
+    },
+    {
+      "epoch": 0.5632,
+      "grad_norm": 18.914832117241684,
+      "learning_rate": 1.8195439355418683e-06,
+      "loss": 1.8664,
+      "step": 66
+    },
+    {
+      "epoch": 0.5717333333333333,
+      "grad_norm": 18.78089008208585,
+      "learning_rate": 1.826074802700826e-06,
+      "loss": 1.7597,
+      "step": 67
+    },
+    {
+      "epoch": 0.5802666666666667,
+      "grad_norm": 18.908370877942197,
+      "learning_rate": 1.8325089127062361e-06,
+      "loss": 1.5813,
+      "step": 68
+    },
+    {
+      "epoch": 0.5888,
+      "grad_norm": 22.558658346768986,
+      "learning_rate": 1.8388490907372552e-06,
+      "loss": 1.598,
+      "step": 69
+    },
+    {
+      "epoch": 0.5973333333333334,
+      "grad_norm": 25.269744707032434,
+      "learning_rate": 1.8450980400142566e-06,
+      "loss": 1.642,
+      "step": 70
+    },
+    {
+      "epoch": 0.6058666666666667,
+      "grad_norm": 29.404854136304497,
+      "learning_rate": 1.851258348719075e-06,
+      "loss": 1.5158,
+      "step": 71
+    },
+    {
+      "epoch": 0.6144,
+      "grad_norm": 28.123079361503738,
+      "learning_rate": 1.857332496431268e-06,
+      "loss": 1.316,
+      "step": 72
+    },
+    {
+      "epoch": 0.6229333333333333,
+      "grad_norm": 33.051693280540675,
+      "learning_rate": 1.8633228601204554e-06,
+      "loss": 1.329,
+      "step": 73
+    },
+    {
+      "epoch": 0.6314666666666666,
+      "grad_norm": 34.63716787552787,
+      "learning_rate": 1.8692317197309759e-06,
+      "loss": 1.2845,
+      "step": 74
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 39.96989376051133,
+      "learning_rate": 1.8750612633916996e-06,
+      "loss": 1.3346,
+      "step": 75
+    },
+    {
+      "epoch": 0.6485333333333333,
+      "grad_norm": 33.72574540792705,
+      "learning_rate": 1.880813592280791e-06,
+      "loss": 1.1742,
+      "step": 76
+    },
+    {
+      "epoch": 0.6570666666666667,
+      "grad_norm": 26.56407853765471,
+      "learning_rate": 1.8864907251724815e-06,
+      "loss": 1.0382,
+      "step": 77
+    },
+    {
+      "epoch": 0.6656,
+      "grad_norm": 17.783020253212666,
+      "learning_rate": 1.89209460269048e-06,
+      "loss": 1.0619,
+      "step": 78
+    },
+    {
+      "epoch": 0.6741333333333334,
+      "grad_norm": 12.089890448213527,
+      "learning_rate": 1.897627091290441e-06,
+      "loss": 1.0258,
+      "step": 79
+    },
+    {
+      "epoch": 0.6826666666666666,
+      "grad_norm": 20.871344149138956,
+      "learning_rate": 1.903089986991943e-06,
+      "loss": 1.1339,
+      "step": 80
+    },
+    {
+      "epoch": 0.6912,
+      "grad_norm": 33.37131665653342,
+      "learning_rate": 1.9084850188786494e-06,
+      "loss": 1.1282,
+      "step": 81
+    },
+    {
+      "epoch": 0.6997333333333333,
+      "grad_norm": 10.57984322098236,
+      "learning_rate": 1.9138138523837166e-06,
+      "loss": 0.9893,
+      "step": 82
+    },
+    {
+      "epoch": 0.7082666666666667,
+      "grad_norm": 85.446420757762,
+      "learning_rate": 1.919078092376074e-06,
+      "loss": 0.9865,
+      "step": 83
+    },
+    {
+      "epoch": 0.7168,
+      "grad_norm": 107.5638346082223,
+      "learning_rate": 1.9242792860618812e-06,
+      "loss": 1.027,
+      "step": 84
+    },
+    {
+      "epoch": 0.7253333333333334,
+      "grad_norm": 101.08530683443811,
+      "learning_rate": 1.9294189257142923e-06,
+      "loss": 1.0515,
+      "step": 85
+    },
+    {
+      "epoch": 0.7338666666666667,
+      "grad_norm": 49.462738500814936,
+      "learning_rate": 1.934498451243567e-06,
+      "loss": 1.0222,
+      "step": 86
+    },
+    {
+      "epoch": 0.7424,
+      "grad_norm": 18.272590764371305,
+      "learning_rate": 1.939519252618618e-06,
+      "loss": 0.9933,
+      "step": 87
+    },
+    {
+      "epoch": 0.7509333333333333,
+      "grad_norm": 25.55179138472976,
+      "learning_rate": 1.9444826721501684e-06,
+      "loss": 0.8641,
+      "step": 88
+    },
+    {
+      "epoch": 0.7594666666666666,
+      "grad_norm": 25.348564922936834,
+      "learning_rate": 1.949390006644912e-06,
+      "loss": 0.9519,
+      "step": 89
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 11.011938725520993,
+      "learning_rate": 1.9542425094393244e-06,
+      "loss": 0.8895,
+      "step": 90
+    },
+    {
+      "epoch": 0.7765333333333333,
+      "grad_norm": 19.925323690467913,
+      "learning_rate": 1.9590413923210933e-06,
+      "loss": 0.8825,
+      "step": 91
+    },
+    {
+      "epoch": 0.7850666666666667,
+      "grad_norm": 24.37047609260986,
+      "learning_rate": 1.963787827345555e-06,
+      "loss": 0.8658,
+      "step": 92
+    },
+    {
+      "epoch": 0.7936,
+      "grad_norm": 22.06573030419392,
+      "learning_rate": 1.968482948553935e-06,
+      "loss": 0.9117,
+      "step": 93
+    },
+    {
+      "epoch": 0.8021333333333334,
+      "grad_norm": 12.409310297339818,
+      "learning_rate": 1.9731278535996984e-06,
+      "loss": 0.8574,
+      "step": 94
+    },
+    {
+      "epoch": 0.8106666666666666,
+      "grad_norm": 16.67272340712659,
+      "learning_rate": 1.9777236052888472e-06,
+      "loss": 0.7485,
+      "step": 95
+    },
+    {
+      "epoch": 0.8192,
+      "grad_norm": 18.424007742054844,
+      "learning_rate": 1.982271233039568e-06,
+      "loss": 0.8916,
+      "step": 96
+    },
+    {
+      "epoch": 0.8277333333333333,
+      "grad_norm": 13.372669052507455,
+      "learning_rate": 1.9867717342662444e-06,
+      "loss": 0.823,
+      "step": 97
+    },
+    {
+      "epoch": 0.8362666666666667,
+      "grad_norm": 20.67631223635667,
+      "learning_rate": 1.9912260756924947e-06,
+      "loss": 0.8447,
+      "step": 98
+    },
+    {
+      "epoch": 0.8448,
+      "grad_norm": 22.08691438633688,
+      "learning_rate": 1.9956351945975495e-06,
+      "loss": 0.7767,
+      "step": 99
+    },
+    {
+      "epoch": 0.8533333333333334,
+      "grad_norm": 12.061092630617507,
+      "learning_rate": 1.9999999999999995e-06,
+      "loss": 0.7029,
+      "step": 100
+    },
+    {
+      "epoch": 0.8618666666666667,
+      "grad_norm": 13.563356715273038,
+      "learning_rate": 2e-06,
+      "loss": 0.6925,
+      "step": 101
+    },
+    {
+      "epoch": 0.8704,
+      "grad_norm": 15.841269394844565,
+      "learning_rate": 2e-06,
+      "loss": 0.5944,
+      "step": 102
+    },
+    {
+      "epoch": 0.8789333333333333,
+      "grad_norm": 14.854530419188935,
+      "learning_rate": 2e-06,
+      "loss": 0.6077,
+      "step": 103
+    },
+    {
+      "epoch": 0.8874666666666666,
+      "grad_norm": 12.791409733249388,
+      "learning_rate": 2e-06,
+      "loss": 0.543,
+      "step": 104
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 14.362179553886866,
+      "learning_rate": 2e-06,
+      "loss": 0.5541,
+      "step": 105
+    },
+    {
+      "epoch": 0.9045333333333333,
+      "grad_norm": 12.49157003340309,
+      "learning_rate": 2e-06,
+      "loss": 0.5295,
+      "step": 106
+    },
+    {
+      "epoch": 0.9130666666666667,
+      "grad_norm": 13.208695431789076,
+      "learning_rate": 2e-06,
+      "loss": 0.4524,
+      "step": 107
+    },
+    {
+      "epoch": 0.9216,
+      "grad_norm": 15.463267948258817,
+      "learning_rate": 2e-06,
+      "loss": 0.4159,
+      "step": 108
+    },
+    {
+      "epoch": 0.9301333333333334,
+      "grad_norm": 12.140164988871907,
+      "learning_rate": 2e-06,
+      "loss": 0.4014,
+      "step": 109
+    },
+    {
+      "epoch": 0.9386666666666666,
+      "grad_norm": 11.905397079235023,
+      "learning_rate": 2e-06,
+      "loss": 0.3618,
+      "step": 110
+    },
+    {
+      "epoch": 0.9472,
+      "grad_norm": 9.415569770415516,
+      "learning_rate": 2e-06,
+      "loss": 0.3057,
+      "step": 111
+    },
+    {
+      "epoch": 0.9557333333333333,
+      "grad_norm": 8.352190984303535,
+      "learning_rate": 2e-06,
+      "loss": 0.3047,
+      "step": 112
+    },
+    {
+      "epoch": 0.9642666666666667,
+      "grad_norm": 7.302691540894704,
+      "learning_rate": 2e-06,
+      "loss": 0.2524,
+      "step": 113
+    },
+    {
+      "epoch": 0.9728,
+      "grad_norm": 5.9832616024269045,
+      "learning_rate": 2e-06,
+      "loss": 0.2518,
+      "step": 114
+    },
+    {
+      "epoch": 0.9813333333333333,
+      "grad_norm": 4.358646236198701,
+      "learning_rate": 2e-06,
+      "loss": 0.2175,
+      "step": 115
+    },
+    {
+      "epoch": 0.9898666666666667,
+      "grad_norm": 4.760656349386409,
+      "learning_rate": 2e-06,
+      "loss": 0.2447,
+      "step": 116
+    },
+    {
+      "epoch": 0.9984,
+      "grad_norm": 3.7039196594291646,
+      "learning_rate": 2e-06,
+      "loss": 0.2033,
+      "step": 117
+    },
+    {
+      "epoch": 1.0069333333333332,
+      "grad_norm": 3.8277716025893387,
+      "learning_rate": 2e-06,
+      "loss": 0.1974,
+      "step": 118
+    },
+    {
+      "epoch": 1.0154666666666667,
+      "grad_norm": 5.741701523334058,
+      "learning_rate": 2e-06,
+      "loss": 0.2018,
+      "step": 119
+    },
+    {
+      "epoch": 1.024,
+      "grad_norm": 4.6412716809913706,
+      "learning_rate": 2e-06,
+      "loss": 0.1875,
+      "step": 120
+    },
+    {
+      "epoch": 1.0325333333333333,
+      "grad_norm": 6.425208050517247,
+      "learning_rate": 2e-06,
+      "loss": 0.2435,
+      "step": 121
+    },
+    {
+      "epoch": 1.0410666666666666,
+      "grad_norm": 6.635511300825933,
+      "learning_rate": 2e-06,
+      "loss": 0.2273,
+      "step": 122
+    },
+    {
+      "epoch": 1.0496,
+      "grad_norm": 23.988717288110806,
+      "learning_rate": 2e-06,
+      "loss": 0.1956,
+      "step": 123
+    },
+    {
+      "epoch": 1.0581333333333334,
+      "grad_norm": 21.03839887669501,
+      "learning_rate": 2e-06,
+      "loss": 0.2077,
+      "step": 124
+    },
+    {
+      "epoch": 1.0666666666666667,
+      "grad_norm": 15.128721173506475,
+      "learning_rate": 2e-06,
+      "loss": 0.1945,
+      "step": 125
+    },
+    {
+      "epoch": 1.0752,
+      "grad_norm": 8.391874252322712,
+      "learning_rate": 2e-06,
+      "loss": 0.1946,
+      "step": 126
+    },
+    {
+      "epoch": 1.0837333333333334,
+      "grad_norm": 10.540931846931757,
+      "learning_rate": 2e-06,
+      "loss": 0.1618,
+      "step": 127
+    },
+    {
+      "epoch": 1.0922666666666667,
+      "grad_norm": 8.206233599981232,
+      "learning_rate": 2e-06,
+      "loss": 0.1827,
+      "step": 128
+    },
+    {
+      "epoch": 1.1008,
+      "grad_norm": 6.272810243100556,
+      "learning_rate": 2e-06,
+      "loss": 0.1814,
+      "step": 129
+    },
+    {
+      "epoch": 1.1093333333333333,
+      "grad_norm": 9.002629637670674,
+      "learning_rate": 2e-06,
+      "loss": 0.1972,
+      "step": 130
+    },
+    {
+      "epoch": 1.1178666666666666,
+      "grad_norm": 8.389486509120731,
+      "learning_rate": 2e-06,
+      "loss": 0.1893,
+      "step": 131
+    },
+    {
+      "epoch": 1.1264,
+      "grad_norm": 2.376071688138911,
+      "learning_rate": 2e-06,
+      "loss": 0.1823,
+      "step": 132
+    },
+    {
+      "epoch": 1.1349333333333333,
+      "grad_norm": 6.541491787591643,
+      "learning_rate": 2e-06,
+      "loss": 0.1509,
+      "step": 133
+    },
+    {
+      "epoch": 1.1434666666666666,
+      "grad_norm": 10.002331652608282,
+      "learning_rate": 2e-06,
+      "loss": 0.1558,
+      "step": 134
+    },
+    {
+      "epoch": 1.152,
+      "grad_norm": 12.229245412509105,
+      "learning_rate": 2e-06,
+      "loss": 0.1466,
+      "step": 135
+    },
+    {
+      "epoch": 1.1605333333333334,
+      "grad_norm": 3.5907402372003747,
+      "learning_rate": 2e-06,
+      "loss": 0.169,
+      "step": 136
+    },
+    {
+      "epoch": 1.1690666666666667,
+      "grad_norm": 3.9213952002791634,
+      "learning_rate": 2e-06,
+      "loss": 0.1652,
+      "step": 137
+    },
+    {
+      "epoch": 1.1776,
+      "grad_norm": 2.624221504666517,
+      "learning_rate": 2e-06,
+      "loss": 0.1739,
+      "step": 138
+    },
+    {
+      "epoch": 1.1861333333333333,
+      "grad_norm": 4.260287456216391,
+      "learning_rate": 2e-06,
+      "loss": 0.1383,
+      "step": 139
+    },
+    {
+      "epoch": 1.1946666666666665,
+      "grad_norm": 4.1658524035506765,
+      "learning_rate": 2e-06,
+      "loss": 0.1483,
+      "step": 140
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 201,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 20,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.1954078401290568e+18,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-140/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e37c6d1ee3e42c776d697d1638e2504df8ccee33d99629ecefbef2fbb06988e3
+size 6840

checkpoint-160/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ../ckpts/Mistral-7B-v0.3
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

checkpoint-160/adapter_config.json

@@ -0,0 +1,35 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "../ckpts/Mistral-7B-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "up_proj",
+    "v_proj",
+    "lm_head",
+    "gate_proj",
+    "k_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-160/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f009596ec48705041d940f72de8bd1f8acd5808375724a5b2f50ff173f7ceae2
+size 353562640

checkpoint-160/trainer_state.json

@@ -0,0 +1,1153 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.3653333333333333,
+  "eval_steps": 500,
+  "global_step": 160,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.008533333333333334,
+      "grad_norm": 250.03163081148352,
+      "learning_rate": 0.0,
+      "loss": 9.7972,
+      "step": 1
+    },
+    {
+      "epoch": 0.017066666666666667,
+      "grad_norm": 248.51781720582932,
+      "learning_rate": 3.010299956639811e-07,
+      "loss": 9.6851,
+      "step": 2
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 242.7004728698525,
+      "learning_rate": 4.771212547196623e-07,
+      "loss": 9.7332,
+      "step": 3
+    },
+    {
+      "epoch": 0.034133333333333335,
+      "grad_norm": 244.87426620614914,
+      "learning_rate": 6.020599913279622e-07,
+      "loss": 9.6243,
+      "step": 4
+    },
+    {
+      "epoch": 0.042666666666666665,
+      "grad_norm": 243.51473421797294,
+      "learning_rate": 6.989700043360186e-07,
+      "loss": 9.6145,
+      "step": 5
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 237.0939456470789,
+      "learning_rate": 7.781512503836435e-07,
+      "loss": 9.5558,
+      "step": 6
+    },
+    {
+      "epoch": 0.05973333333333333,
+      "grad_norm": 246.42426416396174,
+      "learning_rate": 8.450980400142567e-07,
+      "loss": 9.5691,
+      "step": 7
+    },
+    {
+      "epoch": 0.06826666666666667,
+      "grad_norm": 234.25935480253506,
+      "learning_rate": 9.030899869919433e-07,
+      "loss": 9.4209,
+      "step": 8
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 239.2983885236171,
+      "learning_rate": 9.542425094393247e-07,
+      "loss": 9.4733,
+      "step": 9
+    },
+    {
+      "epoch": 0.08533333333333333,
+      "grad_norm": 236.6105466399374,
+      "learning_rate": 9.999999999999997e-07,
+      "loss": 9.5095,
+      "step": 10
+    },
+    {
+      "epoch": 0.09386666666666667,
+      "grad_norm": 231.46864663491684,
+      "learning_rate": 1.0413926851582248e-06,
+      "loss": 9.1118,
+      "step": 11
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 231.50741585044312,
+      "learning_rate": 1.0791812460476246e-06,
+      "loss": 9.0775,
+      "step": 12
+    },
+    {
+      "epoch": 0.11093333333333333,
+      "grad_norm": 232.88580221859812,
+      "learning_rate": 1.1139433523068364e-06,
+      "loss": 8.7966,
+      "step": 13
+    },
+    {
+      "epoch": 0.11946666666666667,
+      "grad_norm": 227.17208510290166,
+      "learning_rate": 1.1461280356782378e-06,
+      "loss": 8.6432,
+      "step": 14
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 222.61277865106936,
+      "learning_rate": 1.176091259055681e-06,
+      "loss": 8.6176,
+      "step": 15
+    },
+    {
+      "epoch": 0.13653333333333334,
+      "grad_norm": 216.5108794899231,
+      "learning_rate": 1.2041199826559244e-06,
+      "loss": 8.1218,
+      "step": 16
+    },
+    {
+      "epoch": 0.14506666666666668,
+      "grad_norm": 219.20955299667028,
+      "learning_rate": 1.230448921378274e-06,
+      "loss": 8.0343,
+      "step": 17
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 199.72085988949678,
+      "learning_rate": 1.2552725051033058e-06,
+      "loss": 7.964,
+      "step": 18
+    },
+    {
+      "epoch": 0.16213333333333332,
+      "grad_norm": 200.24268479147997,
+      "learning_rate": 1.2787536009528286e-06,
+      "loss": 7.647,
+      "step": 19
+    },
+    {
+      "epoch": 0.17066666666666666,
+      "grad_norm": 186.85104361496084,
+      "learning_rate": 1.301029995663981e-06,
+      "loss": 7.3774,
+      "step": 20
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 182.71120056663193,
+      "learning_rate": 1.322219294733919e-06,
+      "loss": 7.1637,
+      "step": 21
+    },
+    {
+      "epoch": 0.18773333333333334,
+      "grad_norm": 182.76003520668846,
+      "learning_rate": 1.3424226808222062e-06,
+      "loss": 7.012,
+      "step": 22
+    },
+    {
+      "epoch": 0.19626666666666667,
+      "grad_norm": 168.48191088919606,
+      "learning_rate": 1.3617278360175927e-06,
+      "loss": 6.6768,
+      "step": 23
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 156.229917459413,
+      "learning_rate": 1.3802112417116059e-06,
+      "loss": 6.7099,
+      "step": 24
+    },
+    {
+      "epoch": 0.21333333333333335,
+      "grad_norm": 154.37179905588326,
+      "learning_rate": 1.3979400086720373e-06,
+      "loss": 6.4922,
+      "step": 25
+    },
+    {
+      "epoch": 0.22186666666666666,
+      "grad_norm": 143.71528792851163,
+      "learning_rate": 1.4149733479708177e-06,
+      "loss": 6.1601,
+      "step": 26
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 146.34965888668822,
+      "learning_rate": 1.431363764158987e-06,
+      "loss": 5.8541,
+      "step": 27
+    },
+    {
+      "epoch": 0.23893333333333333,
+      "grad_norm": 121.92596290244133,
+      "learning_rate": 1.4471580313422189e-06,
+      "loss": 5.4416,
+      "step": 28
+    },
+    {
+      "epoch": 0.24746666666666667,
+      "grad_norm": 120.75132633847133,
+      "learning_rate": 1.4623979978989559e-06,
+      "loss": 5.1664,
+      "step": 29
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 127.72268390687302,
+      "learning_rate": 1.477121254719662e-06,
+      "loss": 4.9839,
+      "step": 30
+    },
+    {
+      "epoch": 0.26453333333333334,
+      "grad_norm": 101.32005196251875,
+      "learning_rate": 1.4913616938342723e-06,
+      "loss": 5.1121,
+      "step": 31
+    },
+    {
+      "epoch": 0.2730666666666667,
+      "grad_norm": 90.5009306970445,
+      "learning_rate": 1.5051499783199057e-06,
+      "loss": 4.3866,
+      "step": 32
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 84.35234360166734,
+      "learning_rate": 1.5185139398778872e-06,
+      "loss": 4.4437,
+      "step": 33
+    },
+    {
+      "epoch": 0.29013333333333335,
+      "grad_norm": 75.2060158778455,
+      "learning_rate": 1.5314789170422548e-06,
+      "loss": 4.3798,
+      "step": 34
+    },
+    {
+      "epoch": 0.2986666666666667,
+      "grad_norm": 73.14821012527409,
+      "learning_rate": 1.544068044350275e-06,
+      "loss": 4.1696,
+      "step": 35
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 65.68340407068212,
+      "learning_rate": 1.556302500767287e-06,
+      "loss": 3.8687,
+      "step": 36
+    },
+    {
+      "epoch": 0.3157333333333333,
+      "grad_norm": 63.72172703320077,
+      "learning_rate": 1.5682017240669948e-06,
+      "loss": 3.9212,
+      "step": 37
+    },
+    {
+      "epoch": 0.32426666666666665,
+      "grad_norm": 55.96549796875415,
+      "learning_rate": 1.57978359661681e-06,
+      "loss": 3.7807,
+      "step": 38
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 50.34103179735071,
+      "learning_rate": 1.5910646070264987e-06,
+      "loss": 3.6901,
+      "step": 39
+    },
+    {
+      "epoch": 0.3413333333333333,
+      "grad_norm": 53.230164751324274,
+      "learning_rate": 1.602059991327962e-06,
+      "loss": 3.6057,
+      "step": 40
+    },
+    {
+      "epoch": 0.34986666666666666,
+      "grad_norm": 42.48618083004681,
+      "learning_rate": 1.6127838567197353e-06,
+      "loss": 3.5198,
+      "step": 41
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 39.512047584209995,
+      "learning_rate": 1.6232492903979003e-06,
+      "loss": 3.4435,
+      "step": 42
+    },
+    {
+      "epoch": 0.36693333333333333,
+      "grad_norm": 37.88712129419359,
+      "learning_rate": 1.633468455579586e-06,
+      "loss": 3.2061,
+      "step": 43
+    },
+    {
+      "epoch": 0.37546666666666667,
+      "grad_norm": 35.24892126286012,
+      "learning_rate": 1.643452676486187e-06,
+      "loss": 3.2579,
+      "step": 44
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 33.04491165858123,
+      "learning_rate": 1.6532125137753431e-06,
+      "loss": 2.8525,
+      "step": 45
+    },
+    {
+      "epoch": 0.39253333333333335,
+      "grad_norm": 31.165406009520897,
+      "learning_rate": 1.6627578316815738e-06,
+      "loss": 3.1049,
+      "step": 46
+    },
+    {
+      "epoch": 0.4010666666666667,
+      "grad_norm": 28.391582735290466,
+      "learning_rate": 1.672097857935717e-06,
+      "loss": 2.6253,
+      "step": 47
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 25.829470766134442,
+      "learning_rate": 1.6812412373755868e-06,
+      "loss": 2.9859,
+      "step": 48
+    },
+    {
+      "epoch": 0.41813333333333336,
+      "grad_norm": 24.083581978153447,
+      "learning_rate": 1.6901960800285134e-06,
+      "loss": 2.6692,
+      "step": 49
+    },
+    {
+      "epoch": 0.4266666666666667,
+      "grad_norm": 22.867521136957954,
+      "learning_rate": 1.6989700043360184e-06,
+      "loss": 2.8821,
+      "step": 50
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 19.900767003905905,
+      "learning_rate": 1.707570176097936e-06,
+      "loss": 2.4392,
+      "step": 51
+    },
+    {
+      "epoch": 0.4437333333333333,
+      "grad_norm": 18.380866742540494,
+      "learning_rate": 1.716003343634799e-06,
+      "loss": 2.6001,
+      "step": 52
+    },
+    {
+      "epoch": 0.45226666666666665,
+      "grad_norm": 18.893434932402364,
+      "learning_rate": 1.7242758696007888e-06,
+      "loss": 2.4671,
+      "step": 53
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 17.676419618997368,
+      "learning_rate": 1.7323937598229684e-06,
+      "loss": 2.6264,
+      "step": 54
+    },
+    {
+      "epoch": 0.4693333333333333,
+      "grad_norm": 16.505413670063586,
+      "learning_rate": 1.7403626894942437e-06,
+      "loss": 2.5316,
+      "step": 55
+    },
+    {
+      "epoch": 0.47786666666666666,
+      "grad_norm": 16.56193892770173,
+      "learning_rate": 1.7481880270062002e-06,
+      "loss": 2.4163,
+      "step": 56
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 14.824844560242497,
+      "learning_rate": 1.7558748556724912e-06,
+      "loss": 2.2749,
+      "step": 57
+    },
+    {
+      "epoch": 0.49493333333333334,
+      "grad_norm": 16.122860315584806,
+      "learning_rate": 1.7634279935629368e-06,
+      "loss": 2.3755,
+      "step": 58
+    },
+    {
+      "epoch": 0.5034666666666666,
+      "grad_norm": 14.99798519590293,
+      "learning_rate": 1.7708520116421439e-06,
+      "loss": 2.2518,
+      "step": 59
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 14.833498207347544,
+      "learning_rate": 1.7781512503836432e-06,
+      "loss": 2.2296,
+      "step": 60
+    },
+    {
+      "epoch": 0.5205333333333333,
+      "grad_norm": 14.084920820041408,
+      "learning_rate": 1.7853298350107666e-06,
+      "loss": 1.9893,
+      "step": 61
+    },
+    {
+      "epoch": 0.5290666666666667,
+      "grad_norm": 14.495022149491797,
+      "learning_rate": 1.7923916894982536e-06,
+      "loss": 1.9802,
+      "step": 62
+    },
+    {
+      "epoch": 0.5376,
+      "grad_norm": 15.49634090425538,
+      "learning_rate": 1.7993405494535814e-06,
+      "loss": 1.8676,
+      "step": 63
+    },
+    {
+      "epoch": 0.5461333333333334,
+      "grad_norm": 16.434037108501176,
+      "learning_rate": 1.8061799739838866e-06,
+      "loss": 1.9443,
+      "step": 64
+    },
+    {
+      "epoch": 0.5546666666666666,
+      "grad_norm": 17.17498656692979,
+      "learning_rate": 1.8129133566428552e-06,
+      "loss": 1.9935,
+      "step": 65
+    },
+    {
+      "epoch": 0.5632,
+      "grad_norm": 18.914832117241684,
+      "learning_rate": 1.8195439355418683e-06,
+      "loss": 1.8664,
+      "step": 66
+    },
+    {
+      "epoch": 0.5717333333333333,
+      "grad_norm": 18.78089008208585,
+      "learning_rate": 1.826074802700826e-06,
+      "loss": 1.7597,
+      "step": 67
+    },
+    {
+      "epoch": 0.5802666666666667,
+      "grad_norm": 18.908370877942197,
+      "learning_rate": 1.8325089127062361e-06,
+      "loss": 1.5813,
+      "step": 68
+    },
+    {
+      "epoch": 0.5888,
+      "grad_norm": 22.558658346768986,
+      "learning_rate": 1.8388490907372552e-06,
+      "loss": 1.598,
+      "step": 69
+    },
+    {
+      "epoch": 0.5973333333333334,
+      "grad_norm": 25.269744707032434,
+      "learning_rate": 1.8450980400142566e-06,
+      "loss": 1.642,
+      "step": 70
+    },
+    {
+      "epoch": 0.6058666666666667,
+      "grad_norm": 29.404854136304497,
+      "learning_rate": 1.851258348719075e-06,
+      "loss": 1.5158,
+      "step": 71
+    },
+    {
+      "epoch": 0.6144,
+      "grad_norm": 28.123079361503738,
+      "learning_rate": 1.857332496431268e-06,
+      "loss": 1.316,
+      "step": 72
+    },
+    {
+      "epoch": 0.6229333333333333,
+      "grad_norm": 33.051693280540675,
+      "learning_rate": 1.8633228601204554e-06,
+      "loss": 1.329,
+      "step": 73
+    },
+    {
+      "epoch": 0.6314666666666666,
+      "grad_norm": 34.63716787552787,
+      "learning_rate": 1.8692317197309759e-06,
+      "loss": 1.2845,
+      "step": 74
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 39.96989376051133,
+      "learning_rate": 1.8750612633916996e-06,
+      "loss": 1.3346,
+      "step": 75
+    },
+    {
+      "epoch": 0.6485333333333333,
+      "grad_norm": 33.72574540792705,
+      "learning_rate": 1.880813592280791e-06,
+      "loss": 1.1742,
+      "step": 76
+    },
+    {
+      "epoch": 0.6570666666666667,
+      "grad_norm": 26.56407853765471,
+      "learning_rate": 1.8864907251724815e-06,
+      "loss": 1.0382,
+      "step": 77
+    },
+    {
+      "epoch": 0.6656,
+      "grad_norm": 17.783020253212666,
+      "learning_rate": 1.89209460269048e-06,
+      "loss": 1.0619,
+      "step": 78
+    },
+    {
+      "epoch": 0.6741333333333334,
+      "grad_norm": 12.089890448213527,
+      "learning_rate": 1.897627091290441e-06,
+      "loss": 1.0258,
+      "step": 79
+    },
+    {
+      "epoch": 0.6826666666666666,
+      "grad_norm": 20.871344149138956,
+      "learning_rate": 1.903089986991943e-06,
+      "loss": 1.1339,
+      "step": 80
+    },
+    {
+      "epoch": 0.6912,
+      "grad_norm": 33.37131665653342,
+      "learning_rate": 1.9084850188786494e-06,
+      "loss": 1.1282,
+      "step": 81
+    },
+    {
+      "epoch": 0.6997333333333333,
+      "grad_norm": 10.57984322098236,
+      "learning_rate": 1.9138138523837166e-06,
+      "loss": 0.9893,
+      "step": 82
+    },
+    {
+      "epoch": 0.7082666666666667,
+      "grad_norm": 85.446420757762,
+      "learning_rate": 1.919078092376074e-06,
+      "loss": 0.9865,
+      "step": 83
+    },
+    {
+      "epoch": 0.7168,
+      "grad_norm": 107.5638346082223,
+      "learning_rate": 1.9242792860618812e-06,
+      "loss": 1.027,
+      "step": 84
+    },
+    {
+      "epoch": 0.7253333333333334,
+      "grad_norm": 101.08530683443811,
+      "learning_rate": 1.9294189257142923e-06,
+      "loss": 1.0515,
+      "step": 85
+    },
+    {
+      "epoch": 0.7338666666666667,
+      "grad_norm": 49.462738500814936,
+      "learning_rate": 1.934498451243567e-06,
+      "loss": 1.0222,
+      "step": 86
+    },
+    {
+      "epoch": 0.7424,
+      "grad_norm": 18.272590764371305,
+      "learning_rate": 1.939519252618618e-06,
+      "loss": 0.9933,
+      "step": 87
+    },
+    {
+      "epoch": 0.7509333333333333,
+      "grad_norm": 25.55179138472976,
+      "learning_rate": 1.9444826721501684e-06,
+      "loss": 0.8641,
+      "step": 88
+    },
+    {
+      "epoch": 0.7594666666666666,
+      "grad_norm": 25.348564922936834,
+      "learning_rate": 1.949390006644912e-06,
+      "loss": 0.9519,
+      "step": 89
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 11.011938725520993,
+      "learning_rate": 1.9542425094393244e-06,
+      "loss": 0.8895,
+      "step": 90
+    },
+    {
+      "epoch": 0.7765333333333333,
+      "grad_norm": 19.925323690467913,
+      "learning_rate": 1.9590413923210933e-06,
+      "loss": 0.8825,
+      "step": 91
+    },
+    {
+      "epoch": 0.7850666666666667,
+      "grad_norm": 24.37047609260986,
+      "learning_rate": 1.963787827345555e-06,
+      "loss": 0.8658,
+      "step": 92
+    },
+    {
+      "epoch": 0.7936,
+      "grad_norm": 22.06573030419392,
+      "learning_rate": 1.968482948553935e-06,
+      "loss": 0.9117,
+      "step": 93
+    },
+    {
+      "epoch": 0.8021333333333334,
+      "grad_norm": 12.409310297339818,
+      "learning_rate": 1.9731278535996984e-06,
+      "loss": 0.8574,
+      "step": 94
+    },
+    {
+      "epoch": 0.8106666666666666,
+      "grad_norm": 16.67272340712659,
+      "learning_rate": 1.9777236052888472e-06,
+      "loss": 0.7485,
+      "step": 95
+    },
+    {
+      "epoch": 0.8192,
+      "grad_norm": 18.424007742054844,
+      "learning_rate": 1.982271233039568e-06,
+      "loss": 0.8916,
+      "step": 96
+    },
+    {
+      "epoch": 0.8277333333333333,
+      "grad_norm": 13.372669052507455,
+      "learning_rate": 1.9867717342662444e-06,
+      "loss": 0.823,
+      "step": 97
+    },
+    {
+      "epoch": 0.8362666666666667,
+      "grad_norm": 20.67631223635667,
+      "learning_rate": 1.9912260756924947e-06,
+      "loss": 0.8447,
+      "step": 98
+    },
+    {
+      "epoch": 0.8448,
+      "grad_norm": 22.08691438633688,
+      "learning_rate": 1.9956351945975495e-06,
+      "loss": 0.7767,
+      "step": 99
+    },
+    {
+      "epoch": 0.8533333333333334,
+      "grad_norm": 12.061092630617507,
+      "learning_rate": 1.9999999999999995e-06,
+      "loss": 0.7029,
+      "step": 100
+    },
+    {
+      "epoch": 0.8618666666666667,
+      "grad_norm": 13.563356715273038,
+      "learning_rate": 2e-06,
+      "loss": 0.6925,
+      "step": 101
+    },
+    {
+      "epoch": 0.8704,
+      "grad_norm": 15.841269394844565,
+      "learning_rate": 2e-06,
+      "loss": 0.5944,
+      "step": 102
+    },
+    {
+      "epoch": 0.8789333333333333,
+      "grad_norm": 14.854530419188935,
+      "learning_rate": 2e-06,
+      "loss": 0.6077,
+      "step": 103
+    },
+    {
+      "epoch": 0.8874666666666666,
+      "grad_norm": 12.791409733249388,
+      "learning_rate": 2e-06,
+      "loss": 0.543,
+      "step": 104
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 14.362179553886866,
+      "learning_rate": 2e-06,
+      "loss": 0.5541,
+      "step": 105
+    },
+    {
+      "epoch": 0.9045333333333333,
+      "grad_norm": 12.49157003340309,
+      "learning_rate": 2e-06,
+      "loss": 0.5295,
+      "step": 106
+    },
+    {
+      "epoch": 0.9130666666666667,
+      "grad_norm": 13.208695431789076,
+      "learning_rate": 2e-06,
+      "loss": 0.4524,
+      "step": 107
+    },
+    {
+      "epoch": 0.9216,
+      "grad_norm": 15.463267948258817,
+      "learning_rate": 2e-06,
+      "loss": 0.4159,
+      "step": 108
+    },
+    {
+      "epoch": 0.9301333333333334,
+      "grad_norm": 12.140164988871907,
+      "learning_rate": 2e-06,
+      "loss": 0.4014,
+      "step": 109
+    },
+    {
+      "epoch": 0.9386666666666666,
+      "grad_norm": 11.905397079235023,
+      "learning_rate": 2e-06,
+      "loss": 0.3618,
+      "step": 110
+    },
+    {
+      "epoch": 0.9472,
+      "grad_norm": 9.415569770415516,
+      "learning_rate": 2e-06,
+      "loss": 0.3057,
+      "step": 111
+    },
+    {
+      "epoch": 0.9557333333333333,
+      "grad_norm": 8.352190984303535,
+      "learning_rate": 2e-06,
+      "loss": 0.3047,
+      "step": 112
+    },
+    {
+      "epoch": 0.9642666666666667,
+      "grad_norm": 7.302691540894704,
+      "learning_rate": 2e-06,
+      "loss": 0.2524,
+      "step": 113
+    },
+    {
+      "epoch": 0.9728,
+      "grad_norm": 5.9832616024269045,
+      "learning_rate": 2e-06,
+      "loss": 0.2518,
+      "step": 114
+    },
+    {
+      "epoch": 0.9813333333333333,
+      "grad_norm": 4.358646236198701,
+      "learning_rate": 2e-06,
+      "loss": 0.2175,
+      "step": 115
+    },
+    {
+      "epoch": 0.9898666666666667,
+      "grad_norm": 4.760656349386409,
+      "learning_rate": 2e-06,
+      "loss": 0.2447,
+      "step": 116
+    },
+    {
+      "epoch": 0.9984,
+      "grad_norm": 3.7039196594291646,
+      "learning_rate": 2e-06,
+      "loss": 0.2033,
+      "step": 117
+    },
+    {
+      "epoch": 1.0069333333333332,
+      "grad_norm": 3.8277716025893387,
+      "learning_rate": 2e-06,
+      "loss": 0.1974,
+      "step": 118
+    },
+    {
+      "epoch": 1.0154666666666667,
+      "grad_norm": 5.741701523334058,
+      "learning_rate": 2e-06,
+      "loss": 0.2018,
+      "step": 119
+    },
+    {
+      "epoch": 1.024,
+      "grad_norm": 4.6412716809913706,
+      "learning_rate": 2e-06,
+      "loss": 0.1875,
+      "step": 120
+    },
+    {
+      "epoch": 1.0325333333333333,
+      "grad_norm": 6.425208050517247,
+      "learning_rate": 2e-06,
+      "loss": 0.2435,
+      "step": 121
+    },
+    {
+      "epoch": 1.0410666666666666,
+      "grad_norm": 6.635511300825933,
+      "learning_rate": 2e-06,
+      "loss": 0.2273,
+      "step": 122
+    },
+    {
+      "epoch": 1.0496,
+      "grad_norm": 23.988717288110806,
+      "learning_rate": 2e-06,
+      "loss": 0.1956,
+      "step": 123
+    },
+    {
+      "epoch": 1.0581333333333334,
+      "grad_norm": 21.03839887669501,
+      "learning_rate": 2e-06,
+      "loss": 0.2077,
+      "step": 124
+    },
+    {
+      "epoch": 1.0666666666666667,
+      "grad_norm": 15.128721173506475,
+      "learning_rate": 2e-06,
+      "loss": 0.1945,
+      "step": 125
+    },
+    {
+      "epoch": 1.0752,
+      "grad_norm": 8.391874252322712,
+      "learning_rate": 2e-06,
+      "loss": 0.1946,
+      "step": 126
+    },
+    {
+      "epoch": 1.0837333333333334,
+      "grad_norm": 10.540931846931757,
+      "learning_rate": 2e-06,
+      "loss": 0.1618,
+      "step": 127
+    },
+    {
+      "epoch": 1.0922666666666667,
+      "grad_norm": 8.206233599981232,
+      "learning_rate": 2e-06,
+      "loss": 0.1827,
+      "step": 128
+    },
+    {
+      "epoch": 1.1008,
+      "grad_norm": 6.272810243100556,
+      "learning_rate": 2e-06,
+      "loss": 0.1814,
+      "step": 129
+    },
+    {
+      "epoch": 1.1093333333333333,
+      "grad_norm": 9.002629637670674,
+      "learning_rate": 2e-06,
+      "loss": 0.1972,
+      "step": 130
+    },
+    {
+      "epoch": 1.1178666666666666,
+      "grad_norm": 8.389486509120731,
+      "learning_rate": 2e-06,
+      "loss": 0.1893,
+      "step": 131
+    },
+    {
+      "epoch": 1.1264,
+      "grad_norm": 2.376071688138911,
+      "learning_rate": 2e-06,
+      "loss": 0.1823,
+      "step": 132
+    },
+    {
+      "epoch": 1.1349333333333333,
+      "grad_norm": 6.541491787591643,
+      "learning_rate": 2e-06,
+      "loss": 0.1509,
+      "step": 133
+    },
+    {
+      "epoch": 1.1434666666666666,
+      "grad_norm": 10.002331652608282,
+      "learning_rate": 2e-06,
+      "loss": 0.1558,
+      "step": 134
+    },
+    {
+      "epoch": 1.152,
+      "grad_norm": 12.229245412509105,
+      "learning_rate": 2e-06,
+      "loss": 0.1466,
+      "step": 135
+    },
+    {
+      "epoch": 1.1605333333333334,
+      "grad_norm": 3.5907402372003747,
+      "learning_rate": 2e-06,
+      "loss": 0.169,
+      "step": 136
+    },
+    {
+      "epoch": 1.1690666666666667,
+      "grad_norm": 3.9213952002791634,
+      "learning_rate": 2e-06,
+      "loss": 0.1652,
+      "step": 137
+    },
+    {
+      "epoch": 1.1776,
+      "grad_norm": 2.624221504666517,
+      "learning_rate": 2e-06,
+      "loss": 0.1739,
+      "step": 138
+    },
+    {
+      "epoch": 1.1861333333333333,
+      "grad_norm": 4.260287456216391,
+      "learning_rate": 2e-06,
+      "loss": 0.1383,
+      "step": 139
+    },
+    {
+      "epoch": 1.1946666666666665,
+      "grad_norm": 4.1658524035506765,
+      "learning_rate": 2e-06,
+      "loss": 0.1483,
+      "step": 140
+    },
+    {
+      "epoch": 1.2032,
+      "grad_norm": 2.6622973548531275,
+      "learning_rate": 2e-06,
+      "loss": 0.1137,
+      "step": 141
+    },
+    {
+      "epoch": 1.2117333333333333,
+      "grad_norm": 1.8268136431856756,
+      "learning_rate": 2e-06,
+      "loss": 0.1522,
+      "step": 142
+    },
+    {
+      "epoch": 1.2202666666666666,
+      "grad_norm": 2.159227057549406,
+      "learning_rate": 2e-06,
+      "loss": 0.1566,
+      "step": 143
+    },
+    {
+      "epoch": 1.2288000000000001,
+      "grad_norm": 5.621018929477287,
+      "learning_rate": 2e-06,
+      "loss": 0.1571,
+      "step": 144
+    },
+    {
+      "epoch": 1.2373333333333334,
+      "grad_norm": 3.2320886169314345,
+      "learning_rate": 2e-06,
+      "loss": 0.1608,
+      "step": 145
+    },
+    {
+      "epoch": 1.2458666666666667,
+      "grad_norm": 3.480936860223943,
+      "learning_rate": 2e-06,
+      "loss": 0.1551,
+      "step": 146
+    },
+    {
+      "epoch": 1.2544,
+      "grad_norm": 2.2690959782213107,
+      "learning_rate": 2e-06,
+      "loss": 0.1355,
+      "step": 147
+    },
+    {
+      "epoch": 1.2629333333333332,
+      "grad_norm": 2.3703700662762834,
+      "learning_rate": 2e-06,
+      "loss": 0.1327,
+      "step": 148
+    },
+    {
+      "epoch": 1.2714666666666667,
+      "grad_norm": 1.2745731181425082,
+      "learning_rate": 2e-06,
+      "loss": 0.1063,
+      "step": 149
+    },
+    {
+      "epoch": 1.28,
+      "grad_norm": 1.310936814354591,
+      "learning_rate": 2e-06,
+      "loss": 0.1179,
+      "step": 150
+    },
+    {
+      "epoch": 1.2885333333333333,
+      "grad_norm": 1.4086038175488638,
+      "learning_rate": 2e-06,
+      "loss": 0.1074,
+      "step": 151
+    },
+    {
+      "epoch": 1.2970666666666666,
+      "grad_norm": 2.152004651394323,
+      "learning_rate": 2e-06,
+      "loss": 0.134,
+      "step": 152
+    },
+    {
+      "epoch": 1.3056,
+      "grad_norm": 2.588001729631509,
+      "learning_rate": 2e-06,
+      "loss": 0.1122,
+      "step": 153
+    },
+    {
+      "epoch": 1.3141333333333334,
+      "grad_norm": 1.389336792990672,
+      "learning_rate": 2e-06,
+      "loss": 0.1635,
+      "step": 154
+    },
+    {
+      "epoch": 1.3226666666666667,
+      "grad_norm": 1.8320820034780816,
+      "learning_rate": 2e-06,
+      "loss": 0.1361,
+      "step": 155
+    },
+    {
+      "epoch": 1.3312,
+      "grad_norm": 3.370202929219345,
+      "learning_rate": 2e-06,
+      "loss": 0.1004,
+      "step": 156
+    },
+    {
+      "epoch": 1.3397333333333332,
+      "grad_norm": 3.0170886979206637,
+      "learning_rate": 2e-06,
+      "loss": 0.0933,
+      "step": 157
+    },
+    {
+      "epoch": 1.3482666666666667,
+      "grad_norm": 3.8244363257261442,
+      "learning_rate": 2e-06,
+      "loss": 0.0987,
+      "step": 158
+    },
+    {
+      "epoch": 1.3568,
+      "grad_norm": 1.4117181217464534,
+      "learning_rate": 2e-06,
+      "loss": 0.1031,
+      "step": 159
+    },
+    {
+      "epoch": 1.3653333333333333,
+      "grad_norm": 2.486160979904412,
+      "learning_rate": 2e-06,
+      "loss": 0.1053,
+      "step": 160
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 201,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 20,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.512320058349322e+18,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-160/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e37c6d1ee3e42c776d697d1638e2504df8ccee33d99629ecefbef2fbb06988e3
+size 6840

checkpoint-180/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ../ckpts/Mistral-7B-v0.3
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

checkpoint-180/adapter_config.json

@@ -0,0 +1,35 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "../ckpts/Mistral-7B-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "up_proj",
+    "v_proj",
+    "lm_head",
+    "gate_proj",
+    "k_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-180/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e8c47ed0cdf5a11afe030e38193bd2bc8c67106c5d2489bccfa1e24cea6def4
+size 353562640

checkpoint-180/trainer_state.json

@@ -0,0 +1,1293 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.536,
+  "eval_steps": 500,
+  "global_step": 180,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.008533333333333334,
+      "grad_norm": 250.03163081148352,
+      "learning_rate": 0.0,
+      "loss": 9.7972,
+      "step": 1
+    },
+    {
+      "epoch": 0.017066666666666667,
+      "grad_norm": 248.51781720582932,
+      "learning_rate": 3.010299956639811e-07,
+      "loss": 9.6851,
+      "step": 2
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 242.7004728698525,
+      "learning_rate": 4.771212547196623e-07,
+      "loss": 9.7332,
+      "step": 3
+    },
+    {
+      "epoch": 0.034133333333333335,
+      "grad_norm": 244.87426620614914,
+      "learning_rate": 6.020599913279622e-07,
+      "loss": 9.6243,
+      "step": 4
+    },
+    {
+      "epoch": 0.042666666666666665,
+      "grad_norm": 243.51473421797294,
+      "learning_rate": 6.989700043360186e-07,
+      "loss": 9.6145,
+      "step": 5
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 237.0939456470789,
+      "learning_rate": 7.781512503836435e-07,
+      "loss": 9.5558,
+      "step": 6
+    },
+    {
+      "epoch": 0.05973333333333333,
+      "grad_norm": 246.42426416396174,
+      "learning_rate": 8.450980400142567e-07,
+      "loss": 9.5691,
+      "step": 7
+    },
+    {
+      "epoch": 0.06826666666666667,
+      "grad_norm": 234.25935480253506,
+      "learning_rate": 9.030899869919433e-07,
+      "loss": 9.4209,
+      "step": 8
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 239.2983885236171,
+      "learning_rate": 9.542425094393247e-07,
+      "loss": 9.4733,
+      "step": 9
+    },
+    {
+      "epoch": 0.08533333333333333,
+      "grad_norm": 236.6105466399374,
+      "learning_rate": 9.999999999999997e-07,
+      "loss": 9.5095,
+      "step": 10
+    },
+    {
+      "epoch": 0.09386666666666667,
+      "grad_norm": 231.46864663491684,
+      "learning_rate": 1.0413926851582248e-06,
+      "loss": 9.1118,
+      "step": 11
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 231.50741585044312,
+      "learning_rate": 1.0791812460476246e-06,
+      "loss": 9.0775,
+      "step": 12
+    },
+    {
+      "epoch": 0.11093333333333333,
+      "grad_norm": 232.88580221859812,
+      "learning_rate": 1.1139433523068364e-06,
+      "loss": 8.7966,
+      "step": 13
+    },
+    {
+      "epoch": 0.11946666666666667,
+      "grad_norm": 227.17208510290166,
+      "learning_rate": 1.1461280356782378e-06,
+      "loss": 8.6432,
+      "step": 14
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 222.61277865106936,
+      "learning_rate": 1.176091259055681e-06,
+      "loss": 8.6176,
+      "step": 15
+    },
+    {
+      "epoch": 0.13653333333333334,
+      "grad_norm": 216.5108794899231,
+      "learning_rate": 1.2041199826559244e-06,
+      "loss": 8.1218,
+      "step": 16
+    },
+    {
+      "epoch": 0.14506666666666668,
+      "grad_norm": 219.20955299667028,
+      "learning_rate": 1.230448921378274e-06,
+      "loss": 8.0343,
+      "step": 17
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 199.72085988949678,
+      "learning_rate": 1.2552725051033058e-06,
+      "loss": 7.964,
+      "step": 18
+    },
+    {
+      "epoch": 0.16213333333333332,
+      "grad_norm": 200.24268479147997,
+      "learning_rate": 1.2787536009528286e-06,
+      "loss": 7.647,
+      "step": 19
+    },
+    {
+      "epoch": 0.17066666666666666,
+      "grad_norm": 186.85104361496084,
+      "learning_rate": 1.301029995663981e-06,
+      "loss": 7.3774,
+      "step": 20
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 182.71120056663193,
+      "learning_rate": 1.322219294733919e-06,
+      "loss": 7.1637,
+      "step": 21
+    },
+    {
+      "epoch": 0.18773333333333334,
+      "grad_norm": 182.76003520668846,
+      "learning_rate": 1.3424226808222062e-06,
+      "loss": 7.012,
+      "step": 22
+    },
+    {
+      "epoch": 0.19626666666666667,
+      "grad_norm": 168.48191088919606,
+      "learning_rate": 1.3617278360175927e-06,
+      "loss": 6.6768,
+      "step": 23
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 156.229917459413,
+      "learning_rate": 1.3802112417116059e-06,
+      "loss": 6.7099,
+      "step": 24
+    },
+    {
+      "epoch": 0.21333333333333335,
+      "grad_norm": 154.37179905588326,
+      "learning_rate": 1.3979400086720373e-06,
+      "loss": 6.4922,
+      "step": 25
+    },
+    {
+      "epoch": 0.22186666666666666,
+      "grad_norm": 143.71528792851163,
+      "learning_rate": 1.4149733479708177e-06,
+      "loss": 6.1601,
+      "step": 26
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 146.34965888668822,
+      "learning_rate": 1.431363764158987e-06,
+      "loss": 5.8541,
+      "step": 27
+    },
+    {
+      "epoch": 0.23893333333333333,
+      "grad_norm": 121.92596290244133,
+      "learning_rate": 1.4471580313422189e-06,
+      "loss": 5.4416,
+      "step": 28
+    },
+    {
+      "epoch": 0.24746666666666667,
+      "grad_norm": 120.75132633847133,
+      "learning_rate": 1.4623979978989559e-06,
+      "loss": 5.1664,
+      "step": 29
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 127.72268390687302,
+      "learning_rate": 1.477121254719662e-06,
+      "loss": 4.9839,
+      "step": 30
+    },
+    {
+      "epoch": 0.26453333333333334,
+      "grad_norm": 101.32005196251875,
+      "learning_rate": 1.4913616938342723e-06,
+      "loss": 5.1121,
+      "step": 31
+    },
+    {
+      "epoch": 0.2730666666666667,
+      "grad_norm": 90.5009306970445,
+      "learning_rate": 1.5051499783199057e-06,
+      "loss": 4.3866,
+      "step": 32
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 84.35234360166734,
+      "learning_rate": 1.5185139398778872e-06,
+      "loss": 4.4437,
+      "step": 33
+    },
+    {
+      "epoch": 0.29013333333333335,
+      "grad_norm": 75.2060158778455,
+      "learning_rate": 1.5314789170422548e-06,
+      "loss": 4.3798,
+      "step": 34
+    },
+    {
+      "epoch": 0.2986666666666667,
+      "grad_norm": 73.14821012527409,
+      "learning_rate": 1.544068044350275e-06,
+      "loss": 4.1696,
+      "step": 35
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 65.68340407068212,
+      "learning_rate": 1.556302500767287e-06,
+      "loss": 3.8687,
+      "step": 36
+    },
+    {
+      "epoch": 0.3157333333333333,
+      "grad_norm": 63.72172703320077,
+      "learning_rate": 1.5682017240669948e-06,
+      "loss": 3.9212,
+      "step": 37
+    },
+    {
+      "epoch": 0.32426666666666665,
+      "grad_norm": 55.96549796875415,
+      "learning_rate": 1.57978359661681e-06,
+      "loss": 3.7807,
+      "step": 38
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 50.34103179735071,
+      "learning_rate": 1.5910646070264987e-06,
+      "loss": 3.6901,
+      "step": 39
+    },
+    {
+      "epoch": 0.3413333333333333,
+      "grad_norm": 53.230164751324274,
+      "learning_rate": 1.602059991327962e-06,
+      "loss": 3.6057,
+      "step": 40
+    },
+    {
+      "epoch": 0.34986666666666666,
+      "grad_norm": 42.48618083004681,
+      "learning_rate": 1.6127838567197353e-06,
+      "loss": 3.5198,
+      "step": 41
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 39.512047584209995,
+      "learning_rate": 1.6232492903979003e-06,
+      "loss": 3.4435,
+      "step": 42
+    },
+    {
+      "epoch": 0.36693333333333333,
+      "grad_norm": 37.88712129419359,
+      "learning_rate": 1.633468455579586e-06,
+      "loss": 3.2061,
+      "step": 43
+    },
+    {
+      "epoch": 0.37546666666666667,
+      "grad_norm": 35.24892126286012,
+      "learning_rate": 1.643452676486187e-06,
+      "loss": 3.2579,
+      "step": 44
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 33.04491165858123,
+      "learning_rate": 1.6532125137753431e-06,
+      "loss": 2.8525,
+      "step": 45
+    },
+    {
+      "epoch": 0.39253333333333335,
+      "grad_norm": 31.165406009520897,
+      "learning_rate": 1.6627578316815738e-06,
+      "loss": 3.1049,
+      "step": 46
+    },
+    {
+      "epoch": 0.4010666666666667,
+      "grad_norm": 28.391582735290466,
+      "learning_rate": 1.672097857935717e-06,
+      "loss": 2.6253,
+      "step": 47
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 25.829470766134442,
+      "learning_rate": 1.6812412373755868e-06,
+      "loss": 2.9859,
+      "step": 48
+    },
+    {
+      "epoch": 0.41813333333333336,
+      "grad_norm": 24.083581978153447,
+      "learning_rate": 1.6901960800285134e-06,
+      "loss": 2.6692,
+      "step": 49
+    },
+    {
+      "epoch": 0.4266666666666667,
+      "grad_norm": 22.867521136957954,
+      "learning_rate": 1.6989700043360184e-06,
+      "loss": 2.8821,
+      "step": 50
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 19.900767003905905,
+      "learning_rate": 1.707570176097936e-06,
+      "loss": 2.4392,
+      "step": 51
+    },
+    {
+      "epoch": 0.4437333333333333,
+      "grad_norm": 18.380866742540494,
+      "learning_rate": 1.716003343634799e-06,
+      "loss": 2.6001,
+      "step": 52
+    },
+    {
+      "epoch": 0.45226666666666665,
+      "grad_norm": 18.893434932402364,
+      "learning_rate": 1.7242758696007888e-06,
+      "loss": 2.4671,
+      "step": 53
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 17.676419618997368,
+      "learning_rate": 1.7323937598229684e-06,
+      "loss": 2.6264,
+      "step": 54
+    },
+    {
+      "epoch": 0.4693333333333333,
+      "grad_norm": 16.505413670063586,
+      "learning_rate": 1.7403626894942437e-06,
+      "loss": 2.5316,
+      "step": 55
+    },
+    {
+      "epoch": 0.47786666666666666,
+      "grad_norm": 16.56193892770173,
+      "learning_rate": 1.7481880270062002e-06,
+      "loss": 2.4163,
+      "step": 56
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 14.824844560242497,
+      "learning_rate": 1.7558748556724912e-06,
+      "loss": 2.2749,
+      "step": 57
+    },
+    {
+      "epoch": 0.49493333333333334,
+      "grad_norm": 16.122860315584806,
+      "learning_rate": 1.7634279935629368e-06,
+      "loss": 2.3755,
+      "step": 58
+    },
+    {
+      "epoch": 0.5034666666666666,
+      "grad_norm": 14.99798519590293,
+      "learning_rate": 1.7708520116421439e-06,
+      "loss": 2.2518,
+      "step": 59
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 14.833498207347544,
+      "learning_rate": 1.7781512503836432e-06,
+      "loss": 2.2296,
+      "step": 60
+    },
+    {
+      "epoch": 0.5205333333333333,
+      "grad_norm": 14.084920820041408,
+      "learning_rate": 1.7853298350107666e-06,
+      "loss": 1.9893,
+      "step": 61
+    },
+    {
+      "epoch": 0.5290666666666667,
+      "grad_norm": 14.495022149491797,
+      "learning_rate": 1.7923916894982536e-06,
+      "loss": 1.9802,
+      "step": 62
+    },
+    {
+      "epoch": 0.5376,
+      "grad_norm": 15.49634090425538,
+      "learning_rate": 1.7993405494535814e-06,
+      "loss": 1.8676,
+      "step": 63
+    },
+    {
+      "epoch": 0.5461333333333334,
+      "grad_norm": 16.434037108501176,
+      "learning_rate": 1.8061799739838866e-06,
+      "loss": 1.9443,
+      "step": 64
+    },
+    {
+      "epoch": 0.5546666666666666,
+      "grad_norm": 17.17498656692979,
+      "learning_rate": 1.8129133566428552e-06,
+      "loss": 1.9935,
+      "step": 65
+    },
+    {
+      "epoch": 0.5632,
+      "grad_norm": 18.914832117241684,
+      "learning_rate": 1.8195439355418683e-06,
+      "loss": 1.8664,
+      "step": 66
+    },
+    {
+      "epoch": 0.5717333333333333,
+      "grad_norm": 18.78089008208585,
+      "learning_rate": 1.826074802700826e-06,
+      "loss": 1.7597,
+      "step": 67
+    },
+    {
+      "epoch": 0.5802666666666667,
+      "grad_norm": 18.908370877942197,
+      "learning_rate": 1.8325089127062361e-06,
+      "loss": 1.5813,
+      "step": 68
+    },
+    {
+      "epoch": 0.5888,
+      "grad_norm": 22.558658346768986,
+      "learning_rate": 1.8388490907372552e-06,
+      "loss": 1.598,
+      "step": 69
+    },
+    {
+      "epoch": 0.5973333333333334,
+      "grad_norm": 25.269744707032434,
+      "learning_rate": 1.8450980400142566e-06,
+      "loss": 1.642,
+      "step": 70
+    },
+    {
+      "epoch": 0.6058666666666667,
+      "grad_norm": 29.404854136304497,
+      "learning_rate": 1.851258348719075e-06,
+      "loss": 1.5158,
+      "step": 71
+    },
+    {
+      "epoch": 0.6144,
+      "grad_norm": 28.123079361503738,
+      "learning_rate": 1.857332496431268e-06,
+      "loss": 1.316,
+      "step": 72
+    },
+    {
+      "epoch": 0.6229333333333333,
+      "grad_norm": 33.051693280540675,
+      "learning_rate": 1.8633228601204554e-06,
+      "loss": 1.329,
+      "step": 73
+    },
+    {
+      "epoch": 0.6314666666666666,
+      "grad_norm": 34.63716787552787,
+      "learning_rate": 1.8692317197309759e-06,
+      "loss": 1.2845,
+      "step": 74
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 39.96989376051133,
+      "learning_rate": 1.8750612633916996e-06,
+      "loss": 1.3346,
+      "step": 75
+    },
+    {
+      "epoch": 0.6485333333333333,
+      "grad_norm": 33.72574540792705,
+      "learning_rate": 1.880813592280791e-06,
+      "loss": 1.1742,
+      "step": 76
+    },
+    {
+      "epoch": 0.6570666666666667,
+      "grad_norm": 26.56407853765471,
+      "learning_rate": 1.8864907251724815e-06,
+      "loss": 1.0382,
+      "step": 77
+    },
+    {
+      "epoch": 0.6656,
+      "grad_norm": 17.783020253212666,
+      "learning_rate": 1.89209460269048e-06,
+      "loss": 1.0619,
+      "step": 78
+    },
+    {
+      "epoch": 0.6741333333333334,
+      "grad_norm": 12.089890448213527,
+      "learning_rate": 1.897627091290441e-06,
+      "loss": 1.0258,
+      "step": 79
+    },
+    {
+      "epoch": 0.6826666666666666,
+      "grad_norm": 20.871344149138956,
+      "learning_rate": 1.903089986991943e-06,
+      "loss": 1.1339,
+      "step": 80
+    },
+    {
+      "epoch": 0.6912,
+      "grad_norm": 33.37131665653342,
+      "learning_rate": 1.9084850188786494e-06,
+      "loss": 1.1282,
+      "step": 81
+    },
+    {
+      "epoch": 0.6997333333333333,
+      "grad_norm": 10.57984322098236,
+      "learning_rate": 1.9138138523837166e-06,
+      "loss": 0.9893,
+      "step": 82
+    },
+    {
+      "epoch": 0.7082666666666667,
+      "grad_norm": 85.446420757762,
+      "learning_rate": 1.919078092376074e-06,
+      "loss": 0.9865,
+      "step": 83
+    },
+    {
+      "epoch": 0.7168,
+      "grad_norm": 107.5638346082223,
+      "learning_rate": 1.9242792860618812e-06,
+      "loss": 1.027,
+      "step": 84
+    },
+    {
+      "epoch": 0.7253333333333334,
+      "grad_norm": 101.08530683443811,
+      "learning_rate": 1.9294189257142923e-06,
+      "loss": 1.0515,
+      "step": 85
+    },
+    {
+      "epoch": 0.7338666666666667,
+      "grad_norm": 49.462738500814936,
+      "learning_rate": 1.934498451243567e-06,
+      "loss": 1.0222,
+      "step": 86
+    },
+    {
+      "epoch": 0.7424,
+      "grad_norm": 18.272590764371305,
+      "learning_rate": 1.939519252618618e-06,
+      "loss": 0.9933,
+      "step": 87
+    },
+    {
+      "epoch": 0.7509333333333333,
+      "grad_norm": 25.55179138472976,
+      "learning_rate": 1.9444826721501684e-06,
+      "loss": 0.8641,
+      "step": 88
+    },
+    {
+      "epoch": 0.7594666666666666,
+      "grad_norm": 25.348564922936834,
+      "learning_rate": 1.949390006644912e-06,
+      "loss": 0.9519,
+      "step": 89
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 11.011938725520993,
+      "learning_rate": 1.9542425094393244e-06,
+      "loss": 0.8895,
+      "step": 90
+    },
+    {
+      "epoch": 0.7765333333333333,
+      "grad_norm": 19.925323690467913,
+      "learning_rate": 1.9590413923210933e-06,
+      "loss": 0.8825,
+      "step": 91
+    },
+    {
+      "epoch": 0.7850666666666667,
+      "grad_norm": 24.37047609260986,
+      "learning_rate": 1.963787827345555e-06,
+      "loss": 0.8658,
+      "step": 92
+    },
+    {
+      "epoch": 0.7936,
+      "grad_norm": 22.06573030419392,
+      "learning_rate": 1.968482948553935e-06,
+      "loss": 0.9117,
+      "step": 93
+    },
+    {
+      "epoch": 0.8021333333333334,
+      "grad_norm": 12.409310297339818,
+      "learning_rate": 1.9731278535996984e-06,
+      "loss": 0.8574,
+      "step": 94
+    },
+    {
+      "epoch": 0.8106666666666666,
+      "grad_norm": 16.67272340712659,
+      "learning_rate": 1.9777236052888472e-06,
+      "loss": 0.7485,
+      "step": 95
+    },
+    {
+      "epoch": 0.8192,
+      "grad_norm": 18.424007742054844,
+      "learning_rate": 1.982271233039568e-06,
+      "loss": 0.8916,
+      "step": 96
+    },
+    {
+      "epoch": 0.8277333333333333,
+      "grad_norm": 13.372669052507455,
+      "learning_rate": 1.9867717342662444e-06,
+      "loss": 0.823,
+      "step": 97
+    },
+    {
+      "epoch": 0.8362666666666667,
+      "grad_norm": 20.67631223635667,
+      "learning_rate": 1.9912260756924947e-06,
+      "loss": 0.8447,
+      "step": 98
+    },
+    {
+      "epoch": 0.8448,
+      "grad_norm": 22.08691438633688,
+      "learning_rate": 1.9956351945975495e-06,
+      "loss": 0.7767,
+      "step": 99
+    },
+    {
+      "epoch": 0.8533333333333334,
+      "grad_norm": 12.061092630617507,
+      "learning_rate": 1.9999999999999995e-06,
+      "loss": 0.7029,
+      "step": 100
+    },
+    {
+      "epoch": 0.8618666666666667,
+      "grad_norm": 13.563356715273038,
+      "learning_rate": 2e-06,
+      "loss": 0.6925,
+      "step": 101
+    },
+    {
+      "epoch": 0.8704,
+      "grad_norm": 15.841269394844565,
+      "learning_rate": 2e-06,
+      "loss": 0.5944,
+      "step": 102
+    },
+    {
+      "epoch": 0.8789333333333333,
+      "grad_norm": 14.854530419188935,
+      "learning_rate": 2e-06,
+      "loss": 0.6077,
+      "step": 103
+    },
+    {
+      "epoch": 0.8874666666666666,
+      "grad_norm": 12.791409733249388,
+      "learning_rate": 2e-06,
+      "loss": 0.543,
+      "step": 104
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 14.362179553886866,
+      "learning_rate": 2e-06,
+      "loss": 0.5541,
+      "step": 105
+    },
+    {
+      "epoch": 0.9045333333333333,
+      "grad_norm": 12.49157003340309,
+      "learning_rate": 2e-06,
+      "loss": 0.5295,
+      "step": 106
+    },
+    {
+      "epoch": 0.9130666666666667,
+      "grad_norm": 13.208695431789076,
+      "learning_rate": 2e-06,
+      "loss": 0.4524,
+      "step": 107
+    },
+    {
+      "epoch": 0.9216,
+      "grad_norm": 15.463267948258817,
+      "learning_rate": 2e-06,
+      "loss": 0.4159,
+      "step": 108
+    },
+    {
+      "epoch": 0.9301333333333334,
+      "grad_norm": 12.140164988871907,
+      "learning_rate": 2e-06,
+      "loss": 0.4014,
+      "step": 109
+    },
+    {
+      "epoch": 0.9386666666666666,
+      "grad_norm": 11.905397079235023,
+      "learning_rate": 2e-06,
+      "loss": 0.3618,
+      "step": 110
+    },
+    {
+      "epoch": 0.9472,
+      "grad_norm": 9.415569770415516,
+      "learning_rate": 2e-06,
+      "loss": 0.3057,
+      "step": 111
+    },
+    {
+      "epoch": 0.9557333333333333,
+      "grad_norm": 8.352190984303535,
+      "learning_rate": 2e-06,
+      "loss": 0.3047,
+      "step": 112
+    },
+    {
+      "epoch": 0.9642666666666667,
+      "grad_norm": 7.302691540894704,
+      "learning_rate": 2e-06,
+      "loss": 0.2524,
+      "step": 113
+    },
+    {
+      "epoch": 0.9728,
+      "grad_norm": 5.9832616024269045,
+      "learning_rate": 2e-06,
+      "loss": 0.2518,
+      "step": 114
+    },
+    {
+      "epoch": 0.9813333333333333,
+      "grad_norm": 4.358646236198701,
+      "learning_rate": 2e-06,
+      "loss": 0.2175,
+      "step": 115
+    },
+    {
+      "epoch": 0.9898666666666667,
+      "grad_norm": 4.760656349386409,
+      "learning_rate": 2e-06,
+      "loss": 0.2447,
+      "step": 116
+    },
+    {
+      "epoch": 0.9984,
+      "grad_norm": 3.7039196594291646,
+      "learning_rate": 2e-06,
+      "loss": 0.2033,
+      "step": 117
+    },
+    {
+      "epoch": 1.0069333333333332,
+      "grad_norm": 3.8277716025893387,
+      "learning_rate": 2e-06,
+      "loss": 0.1974,
+      "step": 118
+    },
+    {
+      "epoch": 1.0154666666666667,
+      "grad_norm": 5.741701523334058,
+      "learning_rate": 2e-06,
+      "loss": 0.2018,
+      "step": 119
+    },
+    {
+      "epoch": 1.024,
+      "grad_norm": 4.6412716809913706,
+      "learning_rate": 2e-06,
+      "loss": 0.1875,
+      "step": 120
+    },
+    {
+      "epoch": 1.0325333333333333,
+      "grad_norm": 6.425208050517247,
+      "learning_rate": 2e-06,
+      "loss": 0.2435,
+      "step": 121
+    },
+    {
+      "epoch": 1.0410666666666666,
+      "grad_norm": 6.635511300825933,
+      "learning_rate": 2e-06,
+      "loss": 0.2273,
+      "step": 122
+    },
+    {
+      "epoch": 1.0496,
+      "grad_norm": 23.988717288110806,
+      "learning_rate": 2e-06,
+      "loss": 0.1956,
+      "step": 123
+    },
+    {
+      "epoch": 1.0581333333333334,
+      "grad_norm": 21.03839887669501,
+      "learning_rate": 2e-06,
+      "loss": 0.2077,
+      "step": 124
+    },
+    {
+      "epoch": 1.0666666666666667,
+      "grad_norm": 15.128721173506475,
+      "learning_rate": 2e-06,
+      "loss": 0.1945,
+      "step": 125
+    },
+    {
+      "epoch": 1.0752,
+      "grad_norm": 8.391874252322712,
+      "learning_rate": 2e-06,
+      "loss": 0.1946,
+      "step": 126
+    },
+    {
+      "epoch": 1.0837333333333334,
+      "grad_norm": 10.540931846931757,
+      "learning_rate": 2e-06,
+      "loss": 0.1618,
+      "step": 127
+    },
+    {
+      "epoch": 1.0922666666666667,
+      "grad_norm": 8.206233599981232,
+      "learning_rate": 2e-06,
+      "loss": 0.1827,
+      "step": 128
+    },
+    {
+      "epoch": 1.1008,
+      "grad_norm": 6.272810243100556,
+      "learning_rate": 2e-06,
+      "loss": 0.1814,
+      "step": 129
+    },
+    {
+      "epoch": 1.1093333333333333,
+      "grad_norm": 9.002629637670674,
+      "learning_rate": 2e-06,
+      "loss": 0.1972,
+      "step": 130
+    },
+    {
+      "epoch": 1.1178666666666666,
+      "grad_norm": 8.389486509120731,
+      "learning_rate": 2e-06,
+      "loss": 0.1893,
+      "step": 131
+    },
+    {
+      "epoch": 1.1264,
+      "grad_norm": 2.376071688138911,
+      "learning_rate": 2e-06,
+      "loss": 0.1823,
+      "step": 132
+    },
+    {
+      "epoch": 1.1349333333333333,
+      "grad_norm": 6.541491787591643,
+      "learning_rate": 2e-06,
+      "loss": 0.1509,
+      "step": 133
+    },
+    {
+      "epoch": 1.1434666666666666,
+      "grad_norm": 10.002331652608282,
+      "learning_rate": 2e-06,
+      "loss": 0.1558,
+      "step": 134
+    },
+    {
+      "epoch": 1.152,
+      "grad_norm": 12.229245412509105,
+      "learning_rate": 2e-06,
+      "loss": 0.1466,
+      "step": 135
+    },
+    {
+      "epoch": 1.1605333333333334,
+      "grad_norm": 3.5907402372003747,
+      "learning_rate": 2e-06,
+      "loss": 0.169,
+      "step": 136
+    },
+    {
+      "epoch": 1.1690666666666667,
+      "grad_norm": 3.9213952002791634,
+      "learning_rate": 2e-06,
+      "loss": 0.1652,
+      "step": 137
+    },
+    {
+      "epoch": 1.1776,
+      "grad_norm": 2.624221504666517,
+      "learning_rate": 2e-06,
+      "loss": 0.1739,
+      "step": 138
+    },
+    {
+      "epoch": 1.1861333333333333,
+      "grad_norm": 4.260287456216391,
+      "learning_rate": 2e-06,
+      "loss": 0.1383,
+      "step": 139
+    },
+    {
+      "epoch": 1.1946666666666665,
+      "grad_norm": 4.1658524035506765,
+      "learning_rate": 2e-06,
+      "loss": 0.1483,
+      "step": 140
+    },
+    {
+      "epoch": 1.2032,
+      "grad_norm": 2.6622973548531275,
+      "learning_rate": 2e-06,
+      "loss": 0.1137,
+      "step": 141
+    },
+    {
+      "epoch": 1.2117333333333333,
+      "grad_norm": 1.8268136431856756,
+      "learning_rate": 2e-06,
+      "loss": 0.1522,
+      "step": 142
+    },
+    {
+      "epoch": 1.2202666666666666,
+      "grad_norm": 2.159227057549406,
+      "learning_rate": 2e-06,
+      "loss": 0.1566,
+      "step": 143
+    },
+    {
+      "epoch": 1.2288000000000001,
+      "grad_norm": 5.621018929477287,
+      "learning_rate": 2e-06,
+      "loss": 0.1571,
+      "step": 144
+    },
+    {
+      "epoch": 1.2373333333333334,
+      "grad_norm": 3.2320886169314345,
+      "learning_rate": 2e-06,
+      "loss": 0.1608,
+      "step": 145
+    },
+    {
+      "epoch": 1.2458666666666667,
+      "grad_norm": 3.480936860223943,
+      "learning_rate": 2e-06,
+      "loss": 0.1551,
+      "step": 146
+    },
+    {
+      "epoch": 1.2544,
+      "grad_norm": 2.2690959782213107,
+      "learning_rate": 2e-06,
+      "loss": 0.1355,
+      "step": 147
+    },
+    {
+      "epoch": 1.2629333333333332,
+      "grad_norm": 2.3703700662762834,
+      "learning_rate": 2e-06,
+      "loss": 0.1327,
+      "step": 148
+    },
+    {
+      "epoch": 1.2714666666666667,
+      "grad_norm": 1.2745731181425082,
+      "learning_rate": 2e-06,
+      "loss": 0.1063,
+      "step": 149
+    },
+    {
+      "epoch": 1.28,
+      "grad_norm": 1.310936814354591,
+      "learning_rate": 2e-06,
+      "loss": 0.1179,
+      "step": 150
+    },
+    {
+      "epoch": 1.2885333333333333,
+      "grad_norm": 1.4086038175488638,
+      "learning_rate": 2e-06,
+      "loss": 0.1074,
+      "step": 151
+    },
+    {
+      "epoch": 1.2970666666666666,
+      "grad_norm": 2.152004651394323,
+      "learning_rate": 2e-06,
+      "loss": 0.134,
+      "step": 152
+    },
+    {
+      "epoch": 1.3056,
+      "grad_norm": 2.588001729631509,
+      "learning_rate": 2e-06,
+      "loss": 0.1122,
+      "step": 153
+    },
+    {
+      "epoch": 1.3141333333333334,
+      "grad_norm": 1.389336792990672,
+      "learning_rate": 2e-06,
+      "loss": 0.1635,
+      "step": 154
+    },
+    {
+      "epoch": 1.3226666666666667,
+      "grad_norm": 1.8320820034780816,
+      "learning_rate": 2e-06,
+      "loss": 0.1361,
+      "step": 155
+    },
+    {
+      "epoch": 1.3312,
+      "grad_norm": 3.370202929219345,
+      "learning_rate": 2e-06,
+      "loss": 0.1004,
+      "step": 156
+    },
+    {
+      "epoch": 1.3397333333333332,
+      "grad_norm": 3.0170886979206637,
+      "learning_rate": 2e-06,
+      "loss": 0.0933,
+      "step": 157
+    },
+    {
+      "epoch": 1.3482666666666667,
+      "grad_norm": 3.8244363257261442,
+      "learning_rate": 2e-06,
+      "loss": 0.0987,
+      "step": 158
+    },
+    {
+      "epoch": 1.3568,
+      "grad_norm": 1.4117181217464534,
+      "learning_rate": 2e-06,
+      "loss": 0.1031,
+      "step": 159
+    },
+    {
+      "epoch": 1.3653333333333333,
+      "grad_norm": 2.486160979904412,
+      "learning_rate": 2e-06,
+      "loss": 0.1053,
+      "step": 160
+    },
+    {
+      "epoch": 1.3738666666666668,
+      "grad_norm": 1.8895272464973183,
+      "learning_rate": 2e-06,
+      "loss": 0.0957,
+      "step": 161
+    },
+    {
+      "epoch": 1.3824,
+      "grad_norm": 2.39025307548428,
+      "learning_rate": 2e-06,
+      "loss": 0.1261,
+      "step": 162
+    },
+    {
+      "epoch": 1.3909333333333334,
+      "grad_norm": 1.1201850850579902,
+      "learning_rate": 2e-06,
+      "loss": 0.1155,
+      "step": 163
+    },
+    {
+      "epoch": 1.3994666666666666,
+      "grad_norm": 2.9250890017279447,
+      "learning_rate": 2e-06,
+      "loss": 0.1118,
+      "step": 164
+    },
+    {
+      "epoch": 1.408,
+      "grad_norm": 1.0675125252661144,
+      "learning_rate": 2e-06,
+      "loss": 0.1114,
+      "step": 165
+    },
+    {
+      "epoch": 1.4165333333333332,
+      "grad_norm": 1.5729321695873222,
+      "learning_rate": 2e-06,
+      "loss": 0.0863,
+      "step": 166
+    },
+    {
+      "epoch": 1.4250666666666667,
+      "grad_norm": 0.4475031725808317,
+      "learning_rate": 2e-06,
+      "loss": 0.1012,
+      "step": 167
+    },
+    {
+      "epoch": 1.4336,
+      "grad_norm": 0.8158056431812921,
+      "learning_rate": 2e-06,
+      "loss": 0.1051,
+      "step": 168
+    },
+    {
+      "epoch": 1.4421333333333333,
+      "grad_norm": 0.7485981396518739,
+      "learning_rate": 2e-06,
+      "loss": 0.1008,
+      "step": 169
+    },
+    {
+      "epoch": 1.4506666666666668,
+      "grad_norm": 0.7816796075268913,
+      "learning_rate": 2e-06,
+      "loss": 0.1111,
+      "step": 170
+    },
+    {
+      "epoch": 1.4592,
+      "grad_norm": 1.1242243424012062,
+      "learning_rate": 2e-06,
+      "loss": 0.099,
+      "step": 171
+    },
+    {
+      "epoch": 1.4677333333333333,
+      "grad_norm": 0.47101002102515854,
+      "learning_rate": 2e-06,
+      "loss": 0.1022,
+      "step": 172
+    },
+    {
+      "epoch": 1.4762666666666666,
+      "grad_norm": 1.2793247365465568,
+      "learning_rate": 2e-06,
+      "loss": 0.0574,
+      "step": 173
+    },
+    {
+      "epoch": 1.4848,
+      "grad_norm": 0.8702420574287141,
+      "learning_rate": 2e-06,
+      "loss": 0.0894,
+      "step": 174
+    },
+    {
+      "epoch": 1.4933333333333334,
+      "grad_norm": 0.28382422642192057,
+      "learning_rate": 2e-06,
+      "loss": 0.092,
+      "step": 175
+    },
+    {
+      "epoch": 1.5018666666666667,
+      "grad_norm": 0.39455360878012,
+      "learning_rate": 2e-06,
+      "loss": 0.1031,
+      "step": 176
+    },
+    {
+      "epoch": 1.5104,
+      "grad_norm": 0.5331804319590009,
+      "learning_rate": 2e-06,
+      "loss": 0.0851,
+      "step": 177
+    },
+    {
+      "epoch": 1.5189333333333335,
+      "grad_norm": 0.4073631368632165,
+      "learning_rate": 2e-06,
+      "loss": 0.0813,
+      "step": 178
+    },
+    {
+      "epoch": 1.5274666666666668,
+      "grad_norm": 0.8514734974117434,
+      "learning_rate": 2e-06,
+      "loss": 0.1076,
+      "step": 179
+    },
+    {
+      "epoch": 1.536,
+      "grad_norm": 0.28586907323674177,
+      "learning_rate": 2e-06,
+      "loss": 0.1013,
+      "step": 180
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 201,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 20,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.834094922578002e+18,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-180/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e37c6d1ee3e42c776d697d1638e2504df8ccee33d99629ecefbef2fbb06988e3
+size 6840

checkpoint-20/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ../ckpts/Mistral-7B-v0.3
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

checkpoint-20/adapter_config.json

@@ -0,0 +1,35 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "../ckpts/Mistral-7B-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "up_proj",
+    "v_proj",
+    "lm_head",
+    "gate_proj",
+    "k_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-20/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7545c313618ae1ac3a8e20bd631625e89b28abbae41f55fa5a9d60351f9b66a
+size 353562640

checkpoint-20/trainer_state.json

@@ -0,0 +1,173 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.17066666666666666,
+  "eval_steps": 500,
+  "global_step": 20,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.008533333333333334,
+      "grad_norm": 250.03163081148352,
+      "learning_rate": 0.0,
+      "loss": 9.7972,
+      "step": 1
+    },
+    {
+      "epoch": 0.017066666666666667,
+      "grad_norm": 248.51781720582932,
+      "learning_rate": 3.010299956639811e-07,
+      "loss": 9.6851,
+      "step": 2
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 242.7004728698525,
+      "learning_rate": 4.771212547196623e-07,
+      "loss": 9.7332,
+      "step": 3
+    },
+    {
+      "epoch": 0.034133333333333335,
+      "grad_norm": 244.87426620614914,
+      "learning_rate": 6.020599913279622e-07,
+      "loss": 9.6243,
+      "step": 4
+    },
+    {
+      "epoch": 0.042666666666666665,
+      "grad_norm": 243.51473421797294,
+      "learning_rate": 6.989700043360186e-07,
+      "loss": 9.6145,
+      "step": 5
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 237.0939456470789,
+      "learning_rate": 7.781512503836435e-07,
+      "loss": 9.5558,
+      "step": 6
+    },
+    {
+      "epoch": 0.05973333333333333,
+      "grad_norm": 246.42426416396174,
+      "learning_rate": 8.450980400142567e-07,
+      "loss": 9.5691,
+      "step": 7
+    },
+    {
+      "epoch": 0.06826666666666667,
+      "grad_norm": 234.25935480253506,
+      "learning_rate": 9.030899869919433e-07,
+      "loss": 9.4209,
+      "step": 8
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 239.2983885236171,
+      "learning_rate": 9.542425094393247e-07,
+      "loss": 9.4733,
+      "step": 9
+    },
+    {
+      "epoch": 0.08533333333333333,
+      "grad_norm": 236.6105466399374,
+      "learning_rate": 9.999999999999997e-07,
+      "loss": 9.5095,
+      "step": 10
+    },
+    {
+      "epoch": 0.09386666666666667,
+      "grad_norm": 231.46864663491684,
+      "learning_rate": 1.0413926851582248e-06,
+      "loss": 9.1118,
+      "step": 11
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 231.50741585044312,
+      "learning_rate": 1.0791812460476246e-06,
+      "loss": 9.0775,
+      "step": 12
+    },
+    {
+      "epoch": 0.11093333333333333,
+      "grad_norm": 232.88580221859812,
+      "learning_rate": 1.1139433523068364e-06,
+      "loss": 8.7966,
+      "step": 13
+    },
+    {
+      "epoch": 0.11946666666666667,
+      "grad_norm": 227.17208510290166,
+      "learning_rate": 1.1461280356782378e-06,
+      "loss": 8.6432,
+      "step": 14
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 222.61277865106936,
+      "learning_rate": 1.176091259055681e-06,
+      "loss": 8.6176,
+      "step": 15
+    },
+    {
+      "epoch": 0.13653333333333334,
+      "grad_norm": 216.5108794899231,
+      "learning_rate": 1.2041199826559244e-06,
+      "loss": 8.1218,
+      "step": 16
+    },
+    {
+      "epoch": 0.14506666666666668,
+      "grad_norm": 219.20955299667028,
+      "learning_rate": 1.230448921378274e-06,
+      "loss": 8.0343,
+      "step": 17
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 199.72085988949678,
+      "learning_rate": 1.2552725051033058e-06,
+      "loss": 7.964,
+      "step": 18
+    },
+    {
+      "epoch": 0.16213333333333332,
+      "grad_norm": 200.24268479147997,
+      "learning_rate": 1.2787536009528286e-06,
+      "loss": 7.647,
+      "step": 19
+    },
+    {
+      "epoch": 0.17066666666666666,
+      "grad_norm": 186.85104361496084,
+      "learning_rate": 1.301029995663981e-06,
+      "loss": 7.3774,
+      "step": 20
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 201,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 20,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 3.227827051482317e+17,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-20/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e37c6d1ee3e42c776d697d1638e2504df8ccee33d99629ecefbef2fbb06988e3
+size 6840

checkpoint-200/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ../ckpts/Mistral-7B-v0.3
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

checkpoint-200/adapter_config.json

@@ -0,0 +1,35 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "../ckpts/Mistral-7B-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "up_proj",
+    "v_proj",
+    "lm_head",
+    "gate_proj",
+    "k_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-200/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:98fd618514a8c4d6ab5d51244da318835f6c4a51a260f3fba29ddb4c34dc3482
+size 353562640

checkpoint-200/trainer_state.json

@@ -0,0 +1,1433 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.7066666666666666,
+  "eval_steps": 500,
+  "global_step": 200,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.008533333333333334,
+      "grad_norm": 250.03163081148352,
+      "learning_rate": 0.0,
+      "loss": 9.7972,
+      "step": 1
+    },
+    {
+      "epoch": 0.017066666666666667,
+      "grad_norm": 248.51781720582932,
+      "learning_rate": 3.010299956639811e-07,
+      "loss": 9.6851,
+      "step": 2
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 242.7004728698525,
+      "learning_rate": 4.771212547196623e-07,
+      "loss": 9.7332,
+      "step": 3
+    },
+    {
+      "epoch": 0.034133333333333335,
+      "grad_norm": 244.87426620614914,
+      "learning_rate": 6.020599913279622e-07,
+      "loss": 9.6243,
+      "step": 4
+    },
+    {
+      "epoch": 0.042666666666666665,
+      "grad_norm": 243.51473421797294,
+      "learning_rate": 6.989700043360186e-07,
+      "loss": 9.6145,
+      "step": 5
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 237.0939456470789,
+      "learning_rate": 7.781512503836435e-07,
+      "loss": 9.5558,
+      "step": 6
+    },
+    {
+      "epoch": 0.05973333333333333,
+      "grad_norm": 246.42426416396174,
+      "learning_rate": 8.450980400142567e-07,
+      "loss": 9.5691,
+      "step": 7
+    },
+    {
+      "epoch": 0.06826666666666667,
+      "grad_norm": 234.25935480253506,
+      "learning_rate": 9.030899869919433e-07,
+      "loss": 9.4209,
+      "step": 8
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 239.2983885236171,
+      "learning_rate": 9.542425094393247e-07,
+      "loss": 9.4733,
+      "step": 9
+    },
+    {
+      "epoch": 0.08533333333333333,
+      "grad_norm": 236.6105466399374,
+      "learning_rate": 9.999999999999997e-07,
+      "loss": 9.5095,
+      "step": 10
+    },
+    {
+      "epoch": 0.09386666666666667,
+      "grad_norm": 231.46864663491684,
+      "learning_rate": 1.0413926851582248e-06,
+      "loss": 9.1118,
+      "step": 11
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 231.50741585044312,
+      "learning_rate": 1.0791812460476246e-06,
+      "loss": 9.0775,
+      "step": 12
+    },
+    {
+      "epoch": 0.11093333333333333,
+      "grad_norm": 232.88580221859812,
+      "learning_rate": 1.1139433523068364e-06,
+      "loss": 8.7966,
+      "step": 13
+    },
+    {
+      "epoch": 0.11946666666666667,
+      "grad_norm": 227.17208510290166,
+      "learning_rate": 1.1461280356782378e-06,
+      "loss": 8.6432,
+      "step": 14
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 222.61277865106936,
+      "learning_rate": 1.176091259055681e-06,
+      "loss": 8.6176,
+      "step": 15
+    },
+    {
+      "epoch": 0.13653333333333334,
+      "grad_norm": 216.5108794899231,
+      "learning_rate": 1.2041199826559244e-06,
+      "loss": 8.1218,
+      "step": 16
+    },
+    {
+      "epoch": 0.14506666666666668,
+      "grad_norm": 219.20955299667028,
+      "learning_rate": 1.230448921378274e-06,
+      "loss": 8.0343,
+      "step": 17
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 199.72085988949678,
+      "learning_rate": 1.2552725051033058e-06,
+      "loss": 7.964,
+      "step": 18
+    },
+    {
+      "epoch": 0.16213333333333332,
+      "grad_norm": 200.24268479147997,
+      "learning_rate": 1.2787536009528286e-06,
+      "loss": 7.647,
+      "step": 19
+    },
+    {
+      "epoch": 0.17066666666666666,
+      "grad_norm": 186.85104361496084,
+      "learning_rate": 1.301029995663981e-06,
+      "loss": 7.3774,
+      "step": 20
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 182.71120056663193,
+      "learning_rate": 1.322219294733919e-06,
+      "loss": 7.1637,
+      "step": 21
+    },
+    {
+      "epoch": 0.18773333333333334,
+      "grad_norm": 182.76003520668846,
+      "learning_rate": 1.3424226808222062e-06,
+      "loss": 7.012,
+      "step": 22
+    },
+    {
+      "epoch": 0.19626666666666667,
+      "grad_norm": 168.48191088919606,
+      "learning_rate": 1.3617278360175927e-06,
+      "loss": 6.6768,
+      "step": 23
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 156.229917459413,
+      "learning_rate": 1.3802112417116059e-06,
+      "loss": 6.7099,
+      "step": 24
+    },
+    {
+      "epoch": 0.21333333333333335,
+      "grad_norm": 154.37179905588326,
+      "learning_rate": 1.3979400086720373e-06,
+      "loss": 6.4922,
+      "step": 25
+    },
+    {
+      "epoch": 0.22186666666666666,
+      "grad_norm": 143.71528792851163,
+      "learning_rate": 1.4149733479708177e-06,
+      "loss": 6.1601,
+      "step": 26
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 146.34965888668822,
+      "learning_rate": 1.431363764158987e-06,
+      "loss": 5.8541,
+      "step": 27
+    },
+    {
+      "epoch": 0.23893333333333333,
+      "grad_norm": 121.92596290244133,
+      "learning_rate": 1.4471580313422189e-06,
+      "loss": 5.4416,
+      "step": 28
+    },
+    {
+      "epoch": 0.24746666666666667,
+      "grad_norm": 120.75132633847133,
+      "learning_rate": 1.4623979978989559e-06,
+      "loss": 5.1664,
+      "step": 29
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 127.72268390687302,
+      "learning_rate": 1.477121254719662e-06,
+      "loss": 4.9839,
+      "step": 30
+    },
+    {
+      "epoch": 0.26453333333333334,
+      "grad_norm": 101.32005196251875,
+      "learning_rate": 1.4913616938342723e-06,
+      "loss": 5.1121,
+      "step": 31
+    },
+    {
+      "epoch": 0.2730666666666667,
+      "grad_norm": 90.5009306970445,
+      "learning_rate": 1.5051499783199057e-06,
+      "loss": 4.3866,
+      "step": 32
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 84.35234360166734,
+      "learning_rate": 1.5185139398778872e-06,
+      "loss": 4.4437,
+      "step": 33
+    },
+    {
+      "epoch": 0.29013333333333335,
+      "grad_norm": 75.2060158778455,
+      "learning_rate": 1.5314789170422548e-06,
+      "loss": 4.3798,
+      "step": 34
+    },
+    {
+      "epoch": 0.2986666666666667,
+      "grad_norm": 73.14821012527409,
+      "learning_rate": 1.544068044350275e-06,
+      "loss": 4.1696,
+      "step": 35
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 65.68340407068212,
+      "learning_rate": 1.556302500767287e-06,
+      "loss": 3.8687,
+      "step": 36
+    },
+    {
+      "epoch": 0.3157333333333333,
+      "grad_norm": 63.72172703320077,
+      "learning_rate": 1.5682017240669948e-06,
+      "loss": 3.9212,
+      "step": 37
+    },
+    {
+      "epoch": 0.32426666666666665,
+      "grad_norm": 55.96549796875415,
+      "learning_rate": 1.57978359661681e-06,
+      "loss": 3.7807,
+      "step": 38
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 50.34103179735071,
+      "learning_rate": 1.5910646070264987e-06,
+      "loss": 3.6901,
+      "step": 39
+    },
+    {
+      "epoch": 0.3413333333333333,
+      "grad_norm": 53.230164751324274,
+      "learning_rate": 1.602059991327962e-06,
+      "loss": 3.6057,
+      "step": 40
+    },
+    {
+      "epoch": 0.34986666666666666,
+      "grad_norm": 42.48618083004681,
+      "learning_rate": 1.6127838567197353e-06,
+      "loss": 3.5198,
+      "step": 41
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 39.512047584209995,
+      "learning_rate": 1.6232492903979003e-06,
+      "loss": 3.4435,
+      "step": 42
+    },
+    {
+      "epoch": 0.36693333333333333,
+      "grad_norm": 37.88712129419359,
+      "learning_rate": 1.633468455579586e-06,
+      "loss": 3.2061,
+      "step": 43
+    },
+    {
+      "epoch": 0.37546666666666667,
+      "grad_norm": 35.24892126286012,
+      "learning_rate": 1.643452676486187e-06,
+      "loss": 3.2579,
+      "step": 44
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 33.04491165858123,
+      "learning_rate": 1.6532125137753431e-06,
+      "loss": 2.8525,
+      "step": 45
+    },
+    {
+      "epoch": 0.39253333333333335,
+      "grad_norm": 31.165406009520897,
+      "learning_rate": 1.6627578316815738e-06,
+      "loss": 3.1049,
+      "step": 46
+    },
+    {
+      "epoch": 0.4010666666666667,
+      "grad_norm": 28.391582735290466,
+      "learning_rate": 1.672097857935717e-06,
+      "loss": 2.6253,
+      "step": 47
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 25.829470766134442,
+      "learning_rate": 1.6812412373755868e-06,
+      "loss": 2.9859,
+      "step": 48
+    },
+    {
+      "epoch": 0.41813333333333336,
+      "grad_norm": 24.083581978153447,
+      "learning_rate": 1.6901960800285134e-06,
+      "loss": 2.6692,
+      "step": 49
+    },
+    {
+      "epoch": 0.4266666666666667,
+      "grad_norm": 22.867521136957954,
+      "learning_rate": 1.6989700043360184e-06,
+      "loss": 2.8821,
+      "step": 50
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 19.900767003905905,
+      "learning_rate": 1.707570176097936e-06,
+      "loss": 2.4392,
+      "step": 51
+    },
+    {
+      "epoch": 0.4437333333333333,
+      "grad_norm": 18.380866742540494,
+      "learning_rate": 1.716003343634799e-06,
+      "loss": 2.6001,
+      "step": 52
+    },
+    {
+      "epoch": 0.45226666666666665,
+      "grad_norm": 18.893434932402364,
+      "learning_rate": 1.7242758696007888e-06,
+      "loss": 2.4671,
+      "step": 53
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 17.676419618997368,
+      "learning_rate": 1.7323937598229684e-06,
+      "loss": 2.6264,
+      "step": 54
+    },
+    {
+      "epoch": 0.4693333333333333,
+      "grad_norm": 16.505413670063586,
+      "learning_rate": 1.7403626894942437e-06,
+      "loss": 2.5316,
+      "step": 55
+    },
+    {
+      "epoch": 0.47786666666666666,
+      "grad_norm": 16.56193892770173,
+      "learning_rate": 1.7481880270062002e-06,
+      "loss": 2.4163,
+      "step": 56
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 14.824844560242497,
+      "learning_rate": 1.7558748556724912e-06,
+      "loss": 2.2749,
+      "step": 57
+    },
+    {
+      "epoch": 0.49493333333333334,
+      "grad_norm": 16.122860315584806,
+      "learning_rate": 1.7634279935629368e-06,
+      "loss": 2.3755,
+      "step": 58
+    },
+    {
+      "epoch": 0.5034666666666666,
+      "grad_norm": 14.99798519590293,
+      "learning_rate": 1.7708520116421439e-06,
+      "loss": 2.2518,
+      "step": 59
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 14.833498207347544,
+      "learning_rate": 1.7781512503836432e-06,
+      "loss": 2.2296,
+      "step": 60
+    },
+    {
+      "epoch": 0.5205333333333333,
+      "grad_norm": 14.084920820041408,
+      "learning_rate": 1.7853298350107666e-06,
+      "loss": 1.9893,
+      "step": 61
+    },
+    {
+      "epoch": 0.5290666666666667,
+      "grad_norm": 14.495022149491797,
+      "learning_rate": 1.7923916894982536e-06,
+      "loss": 1.9802,
+      "step": 62
+    },
+    {
+      "epoch": 0.5376,
+      "grad_norm": 15.49634090425538,
+      "learning_rate": 1.7993405494535814e-06,
+      "loss": 1.8676,
+      "step": 63
+    },
+    {
+      "epoch": 0.5461333333333334,
+      "grad_norm": 16.434037108501176,
+      "learning_rate": 1.8061799739838866e-06,
+      "loss": 1.9443,
+      "step": 64
+    },
+    {
+      "epoch": 0.5546666666666666,
+      "grad_norm": 17.17498656692979,
+      "learning_rate": 1.8129133566428552e-06,
+      "loss": 1.9935,
+      "step": 65
+    },
+    {
+      "epoch": 0.5632,
+      "grad_norm": 18.914832117241684,
+      "learning_rate": 1.8195439355418683e-06,
+      "loss": 1.8664,
+      "step": 66
+    },
+    {
+      "epoch": 0.5717333333333333,
+      "grad_norm": 18.78089008208585,
+      "learning_rate": 1.826074802700826e-06,
+      "loss": 1.7597,
+      "step": 67
+    },
+    {
+      "epoch": 0.5802666666666667,
+      "grad_norm": 18.908370877942197,
+      "learning_rate": 1.8325089127062361e-06,
+      "loss": 1.5813,
+      "step": 68
+    },
+    {
+      "epoch": 0.5888,
+      "grad_norm": 22.558658346768986,
+      "learning_rate": 1.8388490907372552e-06,
+      "loss": 1.598,
+      "step": 69
+    },
+    {
+      "epoch": 0.5973333333333334,
+      "grad_norm": 25.269744707032434,
+      "learning_rate": 1.8450980400142566e-06,
+      "loss": 1.642,
+      "step": 70
+    },
+    {
+      "epoch": 0.6058666666666667,
+      "grad_norm": 29.404854136304497,
+      "learning_rate": 1.851258348719075e-06,
+      "loss": 1.5158,
+      "step": 71
+    },
+    {
+      "epoch": 0.6144,
+      "grad_norm": 28.123079361503738,
+      "learning_rate": 1.857332496431268e-06,
+      "loss": 1.316,
+      "step": 72
+    },
+    {
+      "epoch": 0.6229333333333333,
+      "grad_norm": 33.051693280540675,
+      "learning_rate": 1.8633228601204554e-06,
+      "loss": 1.329,
+      "step": 73
+    },
+    {
+      "epoch": 0.6314666666666666,
+      "grad_norm": 34.63716787552787,
+      "learning_rate": 1.8692317197309759e-06,
+      "loss": 1.2845,
+      "step": 74
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 39.96989376051133,
+      "learning_rate": 1.8750612633916996e-06,
+      "loss": 1.3346,
+      "step": 75
+    },
+    {
+      "epoch": 0.6485333333333333,
+      "grad_norm": 33.72574540792705,
+      "learning_rate": 1.880813592280791e-06,
+      "loss": 1.1742,
+      "step": 76
+    },
+    {
+      "epoch": 0.6570666666666667,
+      "grad_norm": 26.56407853765471,
+      "learning_rate": 1.8864907251724815e-06,
+      "loss": 1.0382,
+      "step": 77
+    },
+    {
+      "epoch": 0.6656,
+      "grad_norm": 17.783020253212666,
+      "learning_rate": 1.89209460269048e-06,
+      "loss": 1.0619,
+      "step": 78
+    },
+    {
+      "epoch": 0.6741333333333334,
+      "grad_norm": 12.089890448213527,
+      "learning_rate": 1.897627091290441e-06,
+      "loss": 1.0258,
+      "step": 79
+    },
+    {
+      "epoch": 0.6826666666666666,
+      "grad_norm": 20.871344149138956,
+      "learning_rate": 1.903089986991943e-06,
+      "loss": 1.1339,
+      "step": 80
+    },
+    {
+      "epoch": 0.6912,
+      "grad_norm": 33.37131665653342,
+      "learning_rate": 1.9084850188786494e-06,
+      "loss": 1.1282,
+      "step": 81
+    },
+    {
+      "epoch": 0.6997333333333333,
+      "grad_norm": 10.57984322098236,
+      "learning_rate": 1.9138138523837166e-06,
+      "loss": 0.9893,
+      "step": 82
+    },
+    {
+      "epoch": 0.7082666666666667,
+      "grad_norm": 85.446420757762,
+      "learning_rate": 1.919078092376074e-06,
+      "loss": 0.9865,
+      "step": 83
+    },
+    {
+      "epoch": 0.7168,
+      "grad_norm": 107.5638346082223,
+      "learning_rate": 1.9242792860618812e-06,
+      "loss": 1.027,
+      "step": 84
+    },
+    {
+      "epoch": 0.7253333333333334,
+      "grad_norm": 101.08530683443811,
+      "learning_rate": 1.9294189257142923e-06,
+      "loss": 1.0515,
+      "step": 85
+    },
+    {
+      "epoch": 0.7338666666666667,
+      "grad_norm": 49.462738500814936,
+      "learning_rate": 1.934498451243567e-06,
+      "loss": 1.0222,
+      "step": 86
+    },
+    {
+      "epoch": 0.7424,
+      "grad_norm": 18.272590764371305,
+      "learning_rate": 1.939519252618618e-06,
+      "loss": 0.9933,
+      "step": 87
+    },
+    {
+      "epoch": 0.7509333333333333,
+      "grad_norm": 25.55179138472976,
+      "learning_rate": 1.9444826721501684e-06,
+      "loss": 0.8641,
+      "step": 88
+    },
+    {
+      "epoch": 0.7594666666666666,
+      "grad_norm": 25.348564922936834,
+      "learning_rate": 1.949390006644912e-06,
+      "loss": 0.9519,
+      "step": 89
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 11.011938725520993,
+      "learning_rate": 1.9542425094393244e-06,
+      "loss": 0.8895,
+      "step": 90
+    },
+    {
+      "epoch": 0.7765333333333333,
+      "grad_norm": 19.925323690467913,
+      "learning_rate": 1.9590413923210933e-06,
+      "loss": 0.8825,
+      "step": 91
+    },
+    {
+      "epoch": 0.7850666666666667,
+      "grad_norm": 24.37047609260986,
+      "learning_rate": 1.963787827345555e-06,
+      "loss": 0.8658,
+      "step": 92
+    },
+    {
+      "epoch": 0.7936,
+      "grad_norm": 22.06573030419392,
+      "learning_rate": 1.968482948553935e-06,
+      "loss": 0.9117,
+      "step": 93
+    },
+    {
+      "epoch": 0.8021333333333334,
+      "grad_norm": 12.409310297339818,
+      "learning_rate": 1.9731278535996984e-06,
+      "loss": 0.8574,
+      "step": 94
+    },
+    {
+      "epoch": 0.8106666666666666,
+      "grad_norm": 16.67272340712659,
+      "learning_rate": 1.9777236052888472e-06,
+      "loss": 0.7485,
+      "step": 95
+    },
+    {
+      "epoch": 0.8192,
+      "grad_norm": 18.424007742054844,
+      "learning_rate": 1.982271233039568e-06,
+      "loss": 0.8916,
+      "step": 96
+    },
+    {
+      "epoch": 0.8277333333333333,
+      "grad_norm": 13.372669052507455,
+      "learning_rate": 1.9867717342662444e-06,
+      "loss": 0.823,
+      "step": 97
+    },
+    {
+      "epoch": 0.8362666666666667,
+      "grad_norm": 20.67631223635667,
+      "learning_rate": 1.9912260756924947e-06,
+      "loss": 0.8447,
+      "step": 98
+    },
+    {
+      "epoch": 0.8448,
+      "grad_norm": 22.08691438633688,
+      "learning_rate": 1.9956351945975495e-06,
+      "loss": 0.7767,
+      "step": 99
+    },
+    {
+      "epoch": 0.8533333333333334,
+      "grad_norm": 12.061092630617507,
+      "learning_rate": 1.9999999999999995e-06,
+      "loss": 0.7029,
+      "step": 100
+    },
+    {
+      "epoch": 0.8618666666666667,
+      "grad_norm": 13.563356715273038,
+      "learning_rate": 2e-06,
+      "loss": 0.6925,
+      "step": 101
+    },
+    {
+      "epoch": 0.8704,
+      "grad_norm": 15.841269394844565,
+      "learning_rate": 2e-06,
+      "loss": 0.5944,
+      "step": 102
+    },
+    {
+      "epoch": 0.8789333333333333,
+      "grad_norm": 14.854530419188935,
+      "learning_rate": 2e-06,
+      "loss": 0.6077,
+      "step": 103
+    },
+    {
+      "epoch": 0.8874666666666666,
+      "grad_norm": 12.791409733249388,
+      "learning_rate": 2e-06,
+      "loss": 0.543,
+      "step": 104
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 14.362179553886866,
+      "learning_rate": 2e-06,
+      "loss": 0.5541,
+      "step": 105
+    },
+    {
+      "epoch": 0.9045333333333333,
+      "grad_norm": 12.49157003340309,
+      "learning_rate": 2e-06,
+      "loss": 0.5295,
+      "step": 106
+    },
+    {
+      "epoch": 0.9130666666666667,
+      "grad_norm": 13.208695431789076,
+      "learning_rate": 2e-06,
+      "loss": 0.4524,
+      "step": 107
+    },
+    {
+      "epoch": 0.9216,
+      "grad_norm": 15.463267948258817,
+      "learning_rate": 2e-06,
+      "loss": 0.4159,
+      "step": 108
+    },
+    {
+      "epoch": 0.9301333333333334,
+      "grad_norm": 12.140164988871907,
+      "learning_rate": 2e-06,
+      "loss": 0.4014,
+      "step": 109
+    },
+    {
+      "epoch": 0.9386666666666666,
+      "grad_norm": 11.905397079235023,
+      "learning_rate": 2e-06,
+      "loss": 0.3618,
+      "step": 110
+    },
+    {
+      "epoch": 0.9472,
+      "grad_norm": 9.415569770415516,
+      "learning_rate": 2e-06,
+      "loss": 0.3057,
+      "step": 111
+    },
+    {
+      "epoch": 0.9557333333333333,
+      "grad_norm": 8.352190984303535,
+      "learning_rate": 2e-06,
+      "loss": 0.3047,
+      "step": 112
+    },
+    {
+      "epoch": 0.9642666666666667,
+      "grad_norm": 7.302691540894704,
+      "learning_rate": 2e-06,
+      "loss": 0.2524,
+      "step": 113
+    },
+    {
+      "epoch": 0.9728,
+      "grad_norm": 5.9832616024269045,
+      "learning_rate": 2e-06,
+      "loss": 0.2518,
+      "step": 114
+    },
+    {
+      "epoch": 0.9813333333333333,
+      "grad_norm": 4.358646236198701,
+      "learning_rate": 2e-06,
+      "loss": 0.2175,
+      "step": 115
+    },
+    {
+      "epoch": 0.9898666666666667,
+      "grad_norm": 4.760656349386409,
+      "learning_rate": 2e-06,
+      "loss": 0.2447,
+      "step": 116
+    },
+    {
+      "epoch": 0.9984,
+      "grad_norm": 3.7039196594291646,
+      "learning_rate": 2e-06,
+      "loss": 0.2033,
+      "step": 117
+    },
+    {
+      "epoch": 1.0069333333333332,
+      "grad_norm": 3.8277716025893387,
+      "learning_rate": 2e-06,
+      "loss": 0.1974,
+      "step": 118
+    },
+    {
+      "epoch": 1.0154666666666667,
+      "grad_norm": 5.741701523334058,
+      "learning_rate": 2e-06,
+      "loss": 0.2018,
+      "step": 119
+    },
+    {
+      "epoch": 1.024,
+      "grad_norm": 4.6412716809913706,
+      "learning_rate": 2e-06,
+      "loss": 0.1875,
+      "step": 120
+    },
+    {
+      "epoch": 1.0325333333333333,
+      "grad_norm": 6.425208050517247,
+      "learning_rate": 2e-06,
+      "loss": 0.2435,
+      "step": 121
+    },
+    {
+      "epoch": 1.0410666666666666,
+      "grad_norm": 6.635511300825933,
+      "learning_rate": 2e-06,
+      "loss": 0.2273,
+      "step": 122
+    },
+    {
+      "epoch": 1.0496,
+      "grad_norm": 23.988717288110806,
+      "learning_rate": 2e-06,
+      "loss": 0.1956,
+      "step": 123
+    },
+    {
+      "epoch": 1.0581333333333334,
+      "grad_norm": 21.03839887669501,
+      "learning_rate": 2e-06,
+      "loss": 0.2077,
+      "step": 124
+    },
+    {
+      "epoch": 1.0666666666666667,
+      "grad_norm": 15.128721173506475,
+      "learning_rate": 2e-06,
+      "loss": 0.1945,
+      "step": 125
+    },
+    {
+      "epoch": 1.0752,
+      "grad_norm": 8.391874252322712,
+      "learning_rate": 2e-06,
+      "loss": 0.1946,
+      "step": 126
+    },
+    {
+      "epoch": 1.0837333333333334,
+      "grad_norm": 10.540931846931757,
+      "learning_rate": 2e-06,
+      "loss": 0.1618,
+      "step": 127
+    },
+    {
+      "epoch": 1.0922666666666667,
+      "grad_norm": 8.206233599981232,
+      "learning_rate": 2e-06,
+      "loss": 0.1827,
+      "step": 128
+    },
+    {
+      "epoch": 1.1008,
+      "grad_norm": 6.272810243100556,
+      "learning_rate": 2e-06,
+      "loss": 0.1814,
+      "step": 129
+    },
+    {
+      "epoch": 1.1093333333333333,
+      "grad_norm": 9.002629637670674,
+      "learning_rate": 2e-06,
+      "loss": 0.1972,
+      "step": 130
+    },
+    {
+      "epoch": 1.1178666666666666,
+      "grad_norm": 8.389486509120731,
+      "learning_rate": 2e-06,
+      "loss": 0.1893,
+      "step": 131
+    },
+    {
+      "epoch": 1.1264,
+      "grad_norm": 2.376071688138911,
+      "learning_rate": 2e-06,
+      "loss": 0.1823,
+      "step": 132
+    },
+    {
+      "epoch": 1.1349333333333333,
+      "grad_norm": 6.541491787591643,
+      "learning_rate": 2e-06,
+      "loss": 0.1509,
+      "step": 133
+    },
+    {
+      "epoch": 1.1434666666666666,
+      "grad_norm": 10.002331652608282,
+      "learning_rate": 2e-06,
+      "loss": 0.1558,
+      "step": 134
+    },
+    {
+      "epoch": 1.152,
+      "grad_norm": 12.229245412509105,
+      "learning_rate": 2e-06,
+      "loss": 0.1466,
+      "step": 135
+    },
+    {
+      "epoch": 1.1605333333333334,
+      "grad_norm": 3.5907402372003747,
+      "learning_rate": 2e-06,
+      "loss": 0.169,
+      "step": 136
+    },
+    {
+      "epoch": 1.1690666666666667,
+      "grad_norm": 3.9213952002791634,
+      "learning_rate": 2e-06,
+      "loss": 0.1652,
+      "step": 137
+    },
+    {
+      "epoch": 1.1776,
+      "grad_norm": 2.624221504666517,
+      "learning_rate": 2e-06,
+      "loss": 0.1739,
+      "step": 138
+    },
+    {
+      "epoch": 1.1861333333333333,
+      "grad_norm": 4.260287456216391,
+      "learning_rate": 2e-06,
+      "loss": 0.1383,
+      "step": 139
+    },
+    {
+      "epoch": 1.1946666666666665,
+      "grad_norm": 4.1658524035506765,
+      "learning_rate": 2e-06,
+      "loss": 0.1483,
+      "step": 140
+    },
+    {
+      "epoch": 1.2032,
+      "grad_norm": 2.6622973548531275,
+      "learning_rate": 2e-06,
+      "loss": 0.1137,
+      "step": 141
+    },
+    {
+      "epoch": 1.2117333333333333,
+      "grad_norm": 1.8268136431856756,
+      "learning_rate": 2e-06,
+      "loss": 0.1522,
+      "step": 142
+    },
+    {
+      "epoch": 1.2202666666666666,
+      "grad_norm": 2.159227057549406,
+      "learning_rate": 2e-06,
+      "loss": 0.1566,
+      "step": 143
+    },
+    {
+      "epoch": 1.2288000000000001,
+      "grad_norm": 5.621018929477287,
+      "learning_rate": 2e-06,
+      "loss": 0.1571,
+      "step": 144
+    },
+    {
+      "epoch": 1.2373333333333334,
+      "grad_norm": 3.2320886169314345,
+      "learning_rate": 2e-06,
+      "loss": 0.1608,
+      "step": 145
+    },
+    {
+      "epoch": 1.2458666666666667,
+      "grad_norm": 3.480936860223943,
+      "learning_rate": 2e-06,
+      "loss": 0.1551,
+      "step": 146
+    },
+    {
+      "epoch": 1.2544,
+      "grad_norm": 2.2690959782213107,
+      "learning_rate": 2e-06,
+      "loss": 0.1355,
+      "step": 147
+    },
+    {
+      "epoch": 1.2629333333333332,
+      "grad_norm": 2.3703700662762834,
+      "learning_rate": 2e-06,
+      "loss": 0.1327,
+      "step": 148
+    },
+    {
+      "epoch": 1.2714666666666667,
+      "grad_norm": 1.2745731181425082,
+      "learning_rate": 2e-06,
+      "loss": 0.1063,
+      "step": 149
+    },
+    {
+      "epoch": 1.28,
+      "grad_norm": 1.310936814354591,
+      "learning_rate": 2e-06,
+      "loss": 0.1179,
+      "step": 150
+    },
+    {
+      "epoch": 1.2885333333333333,
+      "grad_norm": 1.4086038175488638,
+      "learning_rate": 2e-06,
+      "loss": 0.1074,
+      "step": 151
+    },
+    {
+      "epoch": 1.2970666666666666,
+      "grad_norm": 2.152004651394323,
+      "learning_rate": 2e-06,
+      "loss": 0.134,
+      "step": 152
+    },
+    {
+      "epoch": 1.3056,
+      "grad_norm": 2.588001729631509,
+      "learning_rate": 2e-06,
+      "loss": 0.1122,
+      "step": 153
+    },
+    {
+      "epoch": 1.3141333333333334,
+      "grad_norm": 1.389336792990672,
+      "learning_rate": 2e-06,
+      "loss": 0.1635,
+      "step": 154
+    },
+    {
+      "epoch": 1.3226666666666667,
+      "grad_norm": 1.8320820034780816,
+      "learning_rate": 2e-06,
+      "loss": 0.1361,
+      "step": 155
+    },
+    {
+      "epoch": 1.3312,
+      "grad_norm": 3.370202929219345,
+      "learning_rate": 2e-06,
+      "loss": 0.1004,
+      "step": 156
+    },
+    {
+      "epoch": 1.3397333333333332,
+      "grad_norm": 3.0170886979206637,
+      "learning_rate": 2e-06,
+      "loss": 0.0933,
+      "step": 157
+    },
+    {
+      "epoch": 1.3482666666666667,
+      "grad_norm": 3.8244363257261442,
+      "learning_rate": 2e-06,
+      "loss": 0.0987,
+      "step": 158
+    },
+    {
+      "epoch": 1.3568,
+      "grad_norm": 1.4117181217464534,
+      "learning_rate": 2e-06,
+      "loss": 0.1031,
+      "step": 159
+    },
+    {
+      "epoch": 1.3653333333333333,
+      "grad_norm": 2.486160979904412,
+      "learning_rate": 2e-06,
+      "loss": 0.1053,
+      "step": 160
+    },
+    {
+      "epoch": 1.3738666666666668,
+      "grad_norm": 1.8895272464973183,
+      "learning_rate": 2e-06,
+      "loss": 0.0957,
+      "step": 161
+    },
+    {
+      "epoch": 1.3824,
+      "grad_norm": 2.39025307548428,
+      "learning_rate": 2e-06,
+      "loss": 0.1261,
+      "step": 162
+    },
+    {
+      "epoch": 1.3909333333333334,
+      "grad_norm": 1.1201850850579902,
+      "learning_rate": 2e-06,
+      "loss": 0.1155,
+      "step": 163
+    },
+    {
+      "epoch": 1.3994666666666666,
+      "grad_norm": 2.9250890017279447,
+      "learning_rate": 2e-06,
+      "loss": 0.1118,
+      "step": 164
+    },
+    {
+      "epoch": 1.408,
+      "grad_norm": 1.0675125252661144,
+      "learning_rate": 2e-06,
+      "loss": 0.1114,
+      "step": 165
+    },
+    {
+      "epoch": 1.4165333333333332,
+      "grad_norm": 1.5729321695873222,
+      "learning_rate": 2e-06,
+      "loss": 0.0863,
+      "step": 166
+    },
+    {
+      "epoch": 1.4250666666666667,
+      "grad_norm": 0.4475031725808317,
+      "learning_rate": 2e-06,
+      "loss": 0.1012,
+      "step": 167
+    },
+    {
+      "epoch": 1.4336,
+      "grad_norm": 0.8158056431812921,
+      "learning_rate": 2e-06,
+      "loss": 0.1051,
+      "step": 168
+    },
+    {
+      "epoch": 1.4421333333333333,
+      "grad_norm": 0.7485981396518739,
+      "learning_rate": 2e-06,
+      "loss": 0.1008,
+      "step": 169
+    },
+    {
+      "epoch": 1.4506666666666668,
+      "grad_norm": 0.7816796075268913,
+      "learning_rate": 2e-06,
+      "loss": 0.1111,
+      "step": 170
+    },
+    {
+      "epoch": 1.4592,
+      "grad_norm": 1.1242243424012062,
+      "learning_rate": 2e-06,
+      "loss": 0.099,
+      "step": 171
+    },
+    {
+      "epoch": 1.4677333333333333,
+      "grad_norm": 0.47101002102515854,
+      "learning_rate": 2e-06,
+      "loss": 0.1022,
+      "step": 172
+    },
+    {
+      "epoch": 1.4762666666666666,
+      "grad_norm": 1.2793247365465568,
+      "learning_rate": 2e-06,
+      "loss": 0.0574,
+      "step": 173
+    },
+    {
+      "epoch": 1.4848,
+      "grad_norm": 0.8702420574287141,
+      "learning_rate": 2e-06,
+      "loss": 0.0894,
+      "step": 174
+    },
+    {
+      "epoch": 1.4933333333333334,
+      "grad_norm": 0.28382422642192057,
+      "learning_rate": 2e-06,
+      "loss": 0.092,
+      "step": 175
+    },
+    {
+      "epoch": 1.5018666666666667,
+      "grad_norm": 0.39455360878012,
+      "learning_rate": 2e-06,
+      "loss": 0.1031,
+      "step": 176
+    },
+    {
+      "epoch": 1.5104,
+      "grad_norm": 0.5331804319590009,
+      "learning_rate": 2e-06,
+      "loss": 0.0851,
+      "step": 177
+    },
+    {
+      "epoch": 1.5189333333333335,
+      "grad_norm": 0.4073631368632165,
+      "learning_rate": 2e-06,
+      "loss": 0.0813,
+      "step": 178
+    },
+    {
+      "epoch": 1.5274666666666668,
+      "grad_norm": 0.8514734974117434,
+      "learning_rate": 2e-06,
+      "loss": 0.1076,
+      "step": 179
+    },
+    {
+      "epoch": 1.536,
+      "grad_norm": 0.28586907323674177,
+      "learning_rate": 2e-06,
+      "loss": 0.1013,
+      "step": 180
+    },
+    {
+      "epoch": 1.5445333333333333,
+      "grad_norm": 0.3787621686258494,
+      "learning_rate": 2e-06,
+      "loss": 0.0981,
+      "step": 181
+    },
+    {
+      "epoch": 1.5530666666666666,
+      "grad_norm": 0.3489634658189311,
+      "learning_rate": 2e-06,
+      "loss": 0.1008,
+      "step": 182
+    },
+    {
+      "epoch": 1.5615999999999999,
+      "grad_norm": 0.6467831892014061,
+      "learning_rate": 2e-06,
+      "loss": 0.081,
+      "step": 183
+    },
+    {
+      "epoch": 1.5701333333333334,
+      "grad_norm": 0.22665718993777426,
+      "learning_rate": 2e-06,
+      "loss": 0.0711,
+      "step": 184
+    },
+    {
+      "epoch": 1.5786666666666667,
+      "grad_norm": 0.25143762830104677,
+      "learning_rate": 2e-06,
+      "loss": 0.08,
+      "step": 185
+    },
+    {
+      "epoch": 1.5872000000000002,
+      "grad_norm": 0.23617567709822399,
+      "learning_rate": 2e-06,
+      "loss": 0.095,
+      "step": 186
+    },
+    {
+      "epoch": 1.5957333333333334,
+      "grad_norm": 0.24719932263840536,
+      "learning_rate": 2e-06,
+      "loss": 0.0878,
+      "step": 187
+    },
+    {
+      "epoch": 1.6042666666666667,
+      "grad_norm": 0.2871553942071573,
+      "learning_rate": 2e-06,
+      "loss": 0.0801,
+      "step": 188
+    },
+    {
+      "epoch": 1.6128,
+      "grad_norm": 0.4558493501137857,
+      "learning_rate": 2e-06,
+      "loss": 0.0778,
+      "step": 189
+    },
+    {
+      "epoch": 1.6213333333333333,
+      "grad_norm": 0.2922446903826556,
+      "learning_rate": 2e-06,
+      "loss": 0.0791,
+      "step": 190
+    },
+    {
+      "epoch": 1.6298666666666666,
+      "grad_norm": 0.1778189594087765,
+      "learning_rate": 2e-06,
+      "loss": 0.0798,
+      "step": 191
+    },
+    {
+      "epoch": 1.6383999999999999,
+      "grad_norm": 0.3027405276750027,
+      "learning_rate": 2e-06,
+      "loss": 0.0865,
+      "step": 192
+    },
+    {
+      "epoch": 1.6469333333333334,
+      "grad_norm": 0.3158116699810283,
+      "learning_rate": 2e-06,
+      "loss": 0.085,
+      "step": 193
+    },
+    {
+      "epoch": 1.6554666666666666,
+      "grad_norm": 0.2055051898190747,
+      "learning_rate": 2e-06,
+      "loss": 0.1027,
+      "step": 194
+    },
+    {
+      "epoch": 1.6640000000000001,
+      "grad_norm": 0.3078971678574147,
+      "learning_rate": 2e-06,
+      "loss": 0.0908,
+      "step": 195
+    },
+    {
+      "epoch": 1.6725333333333334,
+      "grad_norm": 0.37910835539103654,
+      "learning_rate": 2e-06,
+      "loss": 0.1194,
+      "step": 196
+    },
+    {
+      "epoch": 1.6810666666666667,
+      "grad_norm": 0.3369019547650329,
+      "learning_rate": 2e-06,
+      "loss": 0.1236,
+      "step": 197
+    },
+    {
+      "epoch": 1.6896,
+      "grad_norm": 0.18860600034279623,
+      "learning_rate": 2e-06,
+      "loss": 0.1094,
+      "step": 198
+    },
+    {
+      "epoch": 1.6981333333333333,
+      "grad_norm": 0.16216346688530353,
+      "learning_rate": 2e-06,
+      "loss": 0.0906,
+      "step": 199
+    },
+    {
+      "epoch": 1.7066666666666666,
+      "grad_norm": 0.19192382355970816,
+      "learning_rate": 2e-06,
+      "loss": 0.0743,
+      "step": 200
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 201,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 20,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 3.157253422332248e+18,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-200/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e37c6d1ee3e42c776d697d1638e2504df8ccee33d99629ecefbef2fbb06988e3
+size 6840

checkpoint-40/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ../ckpts/Mistral-7B-v0.3
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

checkpoint-40/adapter_config.json

@@ -0,0 +1,35 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "../ckpts/Mistral-7B-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "up_proj",
+    "v_proj",
+    "lm_head",
+    "gate_proj",
+    "k_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-40/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2630177caf4e3e1fdfaba74efbc5bda93efb8ad362a3eb2c8289c13e5b1d1c68
+size 353562640

checkpoint-40/trainer_state.json

@@ -0,0 +1,313 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.3413333333333333,
+  "eval_steps": 500,
+  "global_step": 40,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.008533333333333334,
+      "grad_norm": 250.03163081148352,
+      "learning_rate": 0.0,
+      "loss": 9.7972,
+      "step": 1
+    },
+    {
+      "epoch": 0.017066666666666667,
+      "grad_norm": 248.51781720582932,
+      "learning_rate": 3.010299956639811e-07,
+      "loss": 9.6851,
+      "step": 2
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 242.7004728698525,
+      "learning_rate": 4.771212547196623e-07,
+      "loss": 9.7332,
+      "step": 3
+    },
+    {
+      "epoch": 0.034133333333333335,
+      "grad_norm": 244.87426620614914,
+      "learning_rate": 6.020599913279622e-07,
+      "loss": 9.6243,
+      "step": 4
+    },
+    {
+      "epoch": 0.042666666666666665,
+      "grad_norm": 243.51473421797294,
+      "learning_rate": 6.989700043360186e-07,
+      "loss": 9.6145,
+      "step": 5
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 237.0939456470789,
+      "learning_rate": 7.781512503836435e-07,
+      "loss": 9.5558,
+      "step": 6
+    },
+    {
+      "epoch": 0.05973333333333333,
+      "grad_norm": 246.42426416396174,
+      "learning_rate": 8.450980400142567e-07,
+      "loss": 9.5691,
+      "step": 7
+    },
+    {
+      "epoch": 0.06826666666666667,
+      "grad_norm": 234.25935480253506,
+      "learning_rate": 9.030899869919433e-07,
+      "loss": 9.4209,
+      "step": 8
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 239.2983885236171,
+      "learning_rate": 9.542425094393247e-07,
+      "loss": 9.4733,
+      "step": 9
+    },
+    {
+      "epoch": 0.08533333333333333,
+      "grad_norm": 236.6105466399374,
+      "learning_rate": 9.999999999999997e-07,
+      "loss": 9.5095,
+      "step": 10
+    },
+    {
+      "epoch": 0.09386666666666667,
+      "grad_norm": 231.46864663491684,
+      "learning_rate": 1.0413926851582248e-06,
+      "loss": 9.1118,
+      "step": 11
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 231.50741585044312,
+      "learning_rate": 1.0791812460476246e-06,
+      "loss": 9.0775,
+      "step": 12
+    },
+    {
+      "epoch": 0.11093333333333333,
+      "grad_norm": 232.88580221859812,
+      "learning_rate": 1.1139433523068364e-06,
+      "loss": 8.7966,
+      "step": 13
+    },
+    {
+      "epoch": 0.11946666666666667,
+      "grad_norm": 227.17208510290166,
+      "learning_rate": 1.1461280356782378e-06,
+      "loss": 8.6432,
+      "step": 14
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 222.61277865106936,
+      "learning_rate": 1.176091259055681e-06,
+      "loss": 8.6176,
+      "step": 15
+    },
+    {
+      "epoch": 0.13653333333333334,
+      "grad_norm": 216.5108794899231,
+      "learning_rate": 1.2041199826559244e-06,
+      "loss": 8.1218,
+      "step": 16
+    },
+    {
+      "epoch": 0.14506666666666668,
+      "grad_norm": 219.20955299667028,
+      "learning_rate": 1.230448921378274e-06,
+      "loss": 8.0343,
+      "step": 17
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 199.72085988949678,
+      "learning_rate": 1.2552725051033058e-06,
+      "loss": 7.964,
+      "step": 18
+    },
+    {
+      "epoch": 0.16213333333333332,
+      "grad_norm": 200.24268479147997,
+      "learning_rate": 1.2787536009528286e-06,
+      "loss": 7.647,
+      "step": 19
+    },
+    {
+      "epoch": 0.17066666666666666,
+      "grad_norm": 186.85104361496084,
+      "learning_rate": 1.301029995663981e-06,
+      "loss": 7.3774,
+      "step": 20
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 182.71120056663193,
+      "learning_rate": 1.322219294733919e-06,
+      "loss": 7.1637,
+      "step": 21
+    },
+    {
+      "epoch": 0.18773333333333334,
+      "grad_norm": 182.76003520668846,
+      "learning_rate": 1.3424226808222062e-06,
+      "loss": 7.012,
+      "step": 22
+    },
+    {
+      "epoch": 0.19626666666666667,
+      "grad_norm": 168.48191088919606,
+      "learning_rate": 1.3617278360175927e-06,
+      "loss": 6.6768,
+      "step": 23
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 156.229917459413,
+      "learning_rate": 1.3802112417116059e-06,
+      "loss": 6.7099,
+      "step": 24
+    },
+    {
+      "epoch": 0.21333333333333335,
+      "grad_norm": 154.37179905588326,
+      "learning_rate": 1.3979400086720373e-06,
+      "loss": 6.4922,
+      "step": 25
+    },
+    {
+      "epoch": 0.22186666666666666,
+      "grad_norm": 143.71528792851163,
+      "learning_rate": 1.4149733479708177e-06,
+      "loss": 6.1601,
+      "step": 26
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 146.34965888668822,
+      "learning_rate": 1.431363764158987e-06,
+      "loss": 5.8541,
+      "step": 27
+    },
+    {
+      "epoch": 0.23893333333333333,
+      "grad_norm": 121.92596290244133,
+      "learning_rate": 1.4471580313422189e-06,
+      "loss": 5.4416,
+      "step": 28
+    },
+    {
+      "epoch": 0.24746666666666667,
+      "grad_norm": 120.75132633847133,
+      "learning_rate": 1.4623979978989559e-06,
+      "loss": 5.1664,
+      "step": 29
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 127.72268390687302,
+      "learning_rate": 1.477121254719662e-06,
+      "loss": 4.9839,
+      "step": 30
+    },
+    {
+      "epoch": 0.26453333333333334,
+      "grad_norm": 101.32005196251875,
+      "learning_rate": 1.4913616938342723e-06,
+      "loss": 5.1121,
+      "step": 31
+    },
+    {
+      "epoch": 0.2730666666666667,
+      "grad_norm": 90.5009306970445,
+      "learning_rate": 1.5051499783199057e-06,
+      "loss": 4.3866,
+      "step": 32
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 84.35234360166734,
+      "learning_rate": 1.5185139398778872e-06,
+      "loss": 4.4437,
+      "step": 33
+    },
+    {
+      "epoch": 0.29013333333333335,
+      "grad_norm": 75.2060158778455,
+      "learning_rate": 1.5314789170422548e-06,
+      "loss": 4.3798,
+      "step": 34
+    },
+    {
+      "epoch": 0.2986666666666667,
+      "grad_norm": 73.14821012527409,
+      "learning_rate": 1.544068044350275e-06,
+      "loss": 4.1696,
+      "step": 35
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 65.68340407068212,
+      "learning_rate": 1.556302500767287e-06,
+      "loss": 3.8687,
+      "step": 36
+    },
+    {
+      "epoch": 0.3157333333333333,
+      "grad_norm": 63.72172703320077,
+      "learning_rate": 1.5682017240669948e-06,
+      "loss": 3.9212,
+      "step": 37
+    },
+    {
+      "epoch": 0.32426666666666665,
+      "grad_norm": 55.96549796875415,
+      "learning_rate": 1.57978359661681e-06,
+      "loss": 3.7807,
+      "step": 38
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 50.34103179735071,
+      "learning_rate": 1.5910646070264987e-06,
+      "loss": 3.6901,
+      "step": 39
+    },
+    {
+      "epoch": 0.3413333333333333,
+      "grad_norm": 53.230164751324274,
+      "learning_rate": 1.602059991327962e-06,
+      "loss": 3.6057,
+      "step": 40
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 201,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 20,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6.308393837039452e+17,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-40/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e37c6d1ee3e42c776d697d1638e2504df8ccee33d99629ecefbef2fbb06988e3
+size 6840

checkpoint-60/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ../ckpts/Mistral-7B-v0.3
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

checkpoint-60/adapter_config.json

@@ -0,0 +1,35 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "../ckpts/Mistral-7B-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "up_proj",
+    "v_proj",
+    "lm_head",
+    "gate_proj",
+    "k_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-60/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e342aefe4362bf670079b4ac87a64d8876cd43627e058bdfbea4b0afe598ad5e
+size 353562640

checkpoint-60/trainer_state.json

@@ -0,0 +1,453 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.512,
+  "eval_steps": 500,
+  "global_step": 60,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.008533333333333334,
+      "grad_norm": 250.03163081148352,
+      "learning_rate": 0.0,
+      "loss": 9.7972,
+      "step": 1
+    },
+    {
+      "epoch": 0.017066666666666667,
+      "grad_norm": 248.51781720582932,
+      "learning_rate": 3.010299956639811e-07,
+      "loss": 9.6851,
+      "step": 2
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 242.7004728698525,
+      "learning_rate": 4.771212547196623e-07,
+      "loss": 9.7332,
+      "step": 3
+    },
+    {
+      "epoch": 0.034133333333333335,
+      "grad_norm": 244.87426620614914,
+      "learning_rate": 6.020599913279622e-07,
+      "loss": 9.6243,
+      "step": 4
+    },
+    {
+      "epoch": 0.042666666666666665,
+      "grad_norm": 243.51473421797294,
+      "learning_rate": 6.989700043360186e-07,
+      "loss": 9.6145,
+      "step": 5
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 237.0939456470789,
+      "learning_rate": 7.781512503836435e-07,
+      "loss": 9.5558,
+      "step": 6
+    },
+    {
+      "epoch": 0.05973333333333333,
+      "grad_norm": 246.42426416396174,
+      "learning_rate": 8.450980400142567e-07,
+      "loss": 9.5691,
+      "step": 7
+    },
+    {
+      "epoch": 0.06826666666666667,
+      "grad_norm": 234.25935480253506,
+      "learning_rate": 9.030899869919433e-07,
+      "loss": 9.4209,
+      "step": 8
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 239.2983885236171,
+      "learning_rate": 9.542425094393247e-07,
+      "loss": 9.4733,
+      "step": 9
+    },
+    {
+      "epoch": 0.08533333333333333,
+      "grad_norm": 236.6105466399374,
+      "learning_rate": 9.999999999999997e-07,
+      "loss": 9.5095,
+      "step": 10
+    },
+    {
+      "epoch": 0.09386666666666667,
+      "grad_norm": 231.46864663491684,
+      "learning_rate": 1.0413926851582248e-06,
+      "loss": 9.1118,
+      "step": 11
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 231.50741585044312,
+      "learning_rate": 1.0791812460476246e-06,
+      "loss": 9.0775,
+      "step": 12
+    },
+    {
+      "epoch": 0.11093333333333333,
+      "grad_norm": 232.88580221859812,
+      "learning_rate": 1.1139433523068364e-06,
+      "loss": 8.7966,
+      "step": 13
+    },
+    {
+      "epoch": 0.11946666666666667,
+      "grad_norm": 227.17208510290166,
+      "learning_rate": 1.1461280356782378e-06,
+      "loss": 8.6432,
+      "step": 14
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 222.61277865106936,
+      "learning_rate": 1.176091259055681e-06,
+      "loss": 8.6176,
+      "step": 15
+    },
+    {
+      "epoch": 0.13653333333333334,
+      "grad_norm": 216.5108794899231,
+      "learning_rate": 1.2041199826559244e-06,
+      "loss": 8.1218,
+      "step": 16
+    },
+    {
+      "epoch": 0.14506666666666668,
+      "grad_norm": 219.20955299667028,
+      "learning_rate": 1.230448921378274e-06,
+      "loss": 8.0343,
+      "step": 17
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 199.72085988949678,
+      "learning_rate": 1.2552725051033058e-06,
+      "loss": 7.964,
+      "step": 18
+    },
+    {
+      "epoch": 0.16213333333333332,
+      "grad_norm": 200.24268479147997,
+      "learning_rate": 1.2787536009528286e-06,
+      "loss": 7.647,
+      "step": 19
+    },
+    {
+      "epoch": 0.17066666666666666,
+      "grad_norm": 186.85104361496084,
+      "learning_rate": 1.301029995663981e-06,
+      "loss": 7.3774,
+      "step": 20
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 182.71120056663193,
+      "learning_rate": 1.322219294733919e-06,
+      "loss": 7.1637,
+      "step": 21
+    },
+    {
+      "epoch": 0.18773333333333334,
+      "grad_norm": 182.76003520668846,
+      "learning_rate": 1.3424226808222062e-06,
+      "loss": 7.012,
+      "step": 22
+    },
+    {
+      "epoch": 0.19626666666666667,
+      "grad_norm": 168.48191088919606,
+      "learning_rate": 1.3617278360175927e-06,
+      "loss": 6.6768,
+      "step": 23
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 156.229917459413,
+      "learning_rate": 1.3802112417116059e-06,
+      "loss": 6.7099,
+      "step": 24
+    },
+    {
+      "epoch": 0.21333333333333335,
+      "grad_norm": 154.37179905588326,
+      "learning_rate": 1.3979400086720373e-06,
+      "loss": 6.4922,
+      "step": 25
+    },
+    {
+      "epoch": 0.22186666666666666,
+      "grad_norm": 143.71528792851163,
+      "learning_rate": 1.4149733479708177e-06,
+      "loss": 6.1601,
+      "step": 26
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 146.34965888668822,
+      "learning_rate": 1.431363764158987e-06,
+      "loss": 5.8541,
+      "step": 27
+    },
+    {
+      "epoch": 0.23893333333333333,
+      "grad_norm": 121.92596290244133,
+      "learning_rate": 1.4471580313422189e-06,
+      "loss": 5.4416,
+      "step": 28
+    },
+    {
+      "epoch": 0.24746666666666667,
+      "grad_norm": 120.75132633847133,
+      "learning_rate": 1.4623979978989559e-06,
+      "loss": 5.1664,
+      "step": 29
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 127.72268390687302,
+      "learning_rate": 1.477121254719662e-06,
+      "loss": 4.9839,
+      "step": 30
+    },
+    {
+      "epoch": 0.26453333333333334,
+      "grad_norm": 101.32005196251875,
+      "learning_rate": 1.4913616938342723e-06,
+      "loss": 5.1121,
+      "step": 31
+    },
+    {
+      "epoch": 0.2730666666666667,
+      "grad_norm": 90.5009306970445,
+      "learning_rate": 1.5051499783199057e-06,
+      "loss": 4.3866,
+      "step": 32
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 84.35234360166734,
+      "learning_rate": 1.5185139398778872e-06,
+      "loss": 4.4437,
+      "step": 33
+    },
+    {
+      "epoch": 0.29013333333333335,
+      "grad_norm": 75.2060158778455,
+      "learning_rate": 1.5314789170422548e-06,
+      "loss": 4.3798,
+      "step": 34
+    },
+    {
+      "epoch": 0.2986666666666667,
+      "grad_norm": 73.14821012527409,
+      "learning_rate": 1.544068044350275e-06,
+      "loss": 4.1696,
+      "step": 35
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 65.68340407068212,
+      "learning_rate": 1.556302500767287e-06,
+      "loss": 3.8687,
+      "step": 36
+    },
+    {
+      "epoch": 0.3157333333333333,
+      "grad_norm": 63.72172703320077,
+      "learning_rate": 1.5682017240669948e-06,
+      "loss": 3.9212,
+      "step": 37
+    },
+    {
+      "epoch": 0.32426666666666665,
+      "grad_norm": 55.96549796875415,
+      "learning_rate": 1.57978359661681e-06,
+      "loss": 3.7807,
+      "step": 38
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 50.34103179735071,
+      "learning_rate": 1.5910646070264987e-06,
+      "loss": 3.6901,
+      "step": 39
+    },
+    {
+      "epoch": 0.3413333333333333,
+      "grad_norm": 53.230164751324274,
+      "learning_rate": 1.602059991327962e-06,
+      "loss": 3.6057,
+      "step": 40
+    },
+    {
+      "epoch": 0.34986666666666666,
+      "grad_norm": 42.48618083004681,
+      "learning_rate": 1.6127838567197353e-06,
+      "loss": 3.5198,
+      "step": 41
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 39.512047584209995,
+      "learning_rate": 1.6232492903979003e-06,
+      "loss": 3.4435,
+      "step": 42
+    },
+    {
+      "epoch": 0.36693333333333333,
+      "grad_norm": 37.88712129419359,
+      "learning_rate": 1.633468455579586e-06,
+      "loss": 3.2061,
+      "step": 43
+    },
+    {
+      "epoch": 0.37546666666666667,
+      "grad_norm": 35.24892126286012,
+      "learning_rate": 1.643452676486187e-06,
+      "loss": 3.2579,
+      "step": 44
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 33.04491165858123,
+      "learning_rate": 1.6532125137753431e-06,
+      "loss": 2.8525,
+      "step": 45
+    },
+    {
+      "epoch": 0.39253333333333335,
+      "grad_norm": 31.165406009520897,
+      "learning_rate": 1.6627578316815738e-06,
+      "loss": 3.1049,
+      "step": 46
+    },
+    {
+      "epoch": 0.4010666666666667,
+      "grad_norm": 28.391582735290466,
+      "learning_rate": 1.672097857935717e-06,
+      "loss": 2.6253,
+      "step": 47
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 25.829470766134442,
+      "learning_rate": 1.6812412373755868e-06,
+      "loss": 2.9859,
+      "step": 48
+    },
+    {
+      "epoch": 0.41813333333333336,
+      "grad_norm": 24.083581978153447,
+      "learning_rate": 1.6901960800285134e-06,
+      "loss": 2.6692,
+      "step": 49
+    },
+    {
+      "epoch": 0.4266666666666667,
+      "grad_norm": 22.867521136957954,
+      "learning_rate": 1.6989700043360184e-06,
+      "loss": 2.8821,
+      "step": 50
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 19.900767003905905,
+      "learning_rate": 1.707570176097936e-06,
+      "loss": 2.4392,
+      "step": 51
+    },
+    {
+      "epoch": 0.4437333333333333,
+      "grad_norm": 18.380866742540494,
+      "learning_rate": 1.716003343634799e-06,
+      "loss": 2.6001,
+      "step": 52
+    },
+    {
+      "epoch": 0.45226666666666665,
+      "grad_norm": 18.893434932402364,
+      "learning_rate": 1.7242758696007888e-06,
+      "loss": 2.4671,
+      "step": 53
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 17.676419618997368,
+      "learning_rate": 1.7323937598229684e-06,
+      "loss": 2.6264,
+      "step": 54
+    },
+    {
+      "epoch": 0.4693333333333333,
+      "grad_norm": 16.505413670063586,
+      "learning_rate": 1.7403626894942437e-06,
+      "loss": 2.5316,
+      "step": 55
+    },
+    {
+      "epoch": 0.47786666666666666,
+      "grad_norm": 16.56193892770173,
+      "learning_rate": 1.7481880270062002e-06,
+      "loss": 2.4163,
+      "step": 56
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 14.824844560242497,
+      "learning_rate": 1.7558748556724912e-06,
+      "loss": 2.2749,
+      "step": 57
+    },
+    {
+      "epoch": 0.49493333333333334,
+      "grad_norm": 16.122860315584806,
+      "learning_rate": 1.7634279935629368e-06,
+      "loss": 2.3755,
+      "step": 58
+    },
+    {
+      "epoch": 0.5034666666666666,
+      "grad_norm": 14.99798519590293,
+      "learning_rate": 1.7708520116421439e-06,
+      "loss": 2.2518,
+      "step": 59
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 14.833498207347544,
+      "learning_rate": 1.7781512503836432e-06,
+      "loss": 2.2296,
+      "step": 60
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 201,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 20,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 9.368370968138875e+17,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-60/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e37c6d1ee3e42c776d697d1638e2504df8ccee33d99629ecefbef2fbb06988e3
+size 6840

checkpoint-80/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: ../ckpts/Mistral-7B-v0.3
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

checkpoint-80/adapter_config.json

@@ -0,0 +1,35 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "../ckpts/Mistral-7B-v0.3",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "up_proj",
+    "v_proj",
+    "lm_head",
+    "gate_proj",
+    "k_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}