Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,202 @@
----
-license: unknown
----
+---
+base_model: Qwen/Qwen2.5-VL-3B-Instruct
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.15.2

adapter_config.json

@@ -0,0 +1,145 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen2.5-VL-3B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "layers.31.mlp.up_proj",
+    "layers.2.mlp.gate_proj",
+    "layers.29.mlp.down_proj",
+    "layers.31.mlp.gate_proj",
+    "k_proj",
+    "layers.8.mlp.gate_proj",
+    "o_proj",
+    "layers.28.mlp.up_proj",
+    "32.mlp.up_proj",
+    "layers.0.mlp.down_proj",
+    "layers.19.mlp.gate_proj",
+    "layers.16.mlp.gate_proj",
+    "layers.24.mlp.up_proj",
+    "layers.14.mlp.gate_proj",
+    "layers.12.mlp.down_proj",
+    "layers.3.mlp.gate_proj",
+    "q_proj",
+    "layers.4.mlp.down_proj",
+    "layers.21.mlp.down_proj",
+    "layers.0.mlp.gate_proj",
+    "layers.7.mlp.up_proj",
+    "layers.27.mlp.up_proj",
+    "layers.29.mlp.gate_proj",
+    "layers.20.mlp.up_proj",
+    "layers.4.mlp.up_proj",
+    "layers.26.mlp.gate_proj",
+    "layers.9.mlp.down_proj",
+    "layers.1.mlp.down_proj",
+    "layers.5.mlp.up_proj",
+    "layers.16.mlp.up_proj",
+    "layers.19.mlp.up_proj",
+    "layers.23.mlp.up_proj",
+    "layers.29.mlp.up_proj",
+    "layers.16.mlp.down_proj",
+    "lm_head",
+    "layers.6.mlp.down_proj",
+    "33.mlp.up_proj",
+    "layers.12.mlp.gate_proj",
+    "layers.17.mlp.gate_proj",
+    "34.mlp.up_proj",
+    "layers.18.mlp.gate_proj",
+    "layers.20.mlp.down_proj",
+    "layers.28.mlp.gate_proj",
+    "layers.14.mlp.down_proj",
+    "layers.26.mlp.down_proj",
+    "layers.21.mlp.up_proj",
+    "layers.21.mlp.gate_proj",
+    "layers.1.mlp.up_proj",
+    "layers.30.mlp.up_proj",
+    "layers.27.mlp.gate_proj",
+    "layers.5.mlp.down_proj",
+    "layers.8.mlp.up_proj",
+    "layers.7.mlp.down_proj",
+    "layers.19.mlp.down_proj",
+    "33.mlp.gate_proj",
+    "layers.22.mlp.down_proj",
+    "layers.0.mlp.up_proj",
+    "35.mlp.gate_proj",
+    "layers.25.mlp.up_proj",
+    "layers.4.mlp.gate_proj",
+    "33.mlp.down_proj",
+    "layers.14.mlp.up_proj",
+    "layers.15.mlp.gate_proj",
+    "layers.8.mlp.down_proj",
+    "layers.31.mlp.down_proj",
+    "layers.22.mlp.gate_proj",
+    "layers.11.mlp.gate_proj",
+    "layers.24.mlp.down_proj",
+    "layers.11.mlp.up_proj",
+    "layers.2.mlp.up_proj",
+    "layers.24.mlp.gate_proj",
+    "layers.6.mlp.gate_proj",
+    "layers.7.mlp.gate_proj",
+    "layers.9.mlp.gate_proj",
+    "layers.10.mlp.down_proj",
+    "layers.20.mlp.gate_proj",
+    "layers.30.mlp.gate_proj",
+    "layers.13.mlp.gate_proj",
+    "layers.17.mlp.down_proj",
+    "32.mlp.down_proj",
+    "layers.18.mlp.up_proj",
+    "32.mlp.gate_proj",
+    "layers.3.mlp.down_proj",
+    "layers.11.mlp.down_proj",
+    "layers.17.mlp.up_proj",
+    "layers.10.mlp.up_proj",
+    "v_proj",
+    "35.mlp.down_proj",
+    "layers.1.mlp.gate_proj",
+    "layers.12.mlp.up_proj",
+    "layers.15.mlp.up_proj",
+    "34.mlp.gate_proj",
+    "layers.15.mlp.down_proj",
+    "layers.18.mlp.down_proj",
+    "34.mlp.down_proj",
+    "layers.23.mlp.down_proj",
+    "layers.27.mlp.down_proj",
+    "layers.10.mlp.gate_proj",
+    "layers.6.mlp.up_proj",
+    "layers.23.mlp.gate_proj",
+    "layers.13.mlp.down_proj",
+    "layers.26.mlp.up_proj",
+    "layers.9.mlp.up_proj",
+    "layers.25.mlp.gate_proj",
+    "layers.25.mlp.down_proj",
+    "layers.13.mlp.up_proj",
+    "layers.3.mlp.up_proj",
+    "layers.2.mlp.down_proj",
+    "35.mlp.up_proj",
+    "layers.28.mlp.down_proj",
+    "layers.30.mlp.down_proj",
+    "layers.5.mlp.gate_proj",
+    "layers.22.mlp.up_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b13517ad060a2729d26c58754c186c5dc5596fd3e2d78f649e3a7a4783353099
+size 881577392

added_tokens.json

@@ -0,0 +1,24 @@
+{
+  "</tool_call>": 151658,
+  "<tool_call>": 151657,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.json

@@ -0,0 +1,3 @@
+{
+  "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}"
+}

global_step200/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:481bb61df5e3b32db700046bc9492f8055e2dd43ef261aa13f013d82ea6ea463
+size 777569296

global_step200/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:512c0b94301d6a472826d3044c56e2f476dcb14b4abfe4d7e5679aae3ef0ec8a
+size 777564368

global_step200/mp_rank_00_model_states.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1360cdffd684fc9a4d58e724f92c38ef52a87b28b35400ad7128b5ed983ff31f
+size 881924536

latest

@@ -0,0 +1 @@
+global_step200

preprocessor_config.json

@@ -0,0 +1,29 @@
+{
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "Qwen2VLImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "max_pixels": 12845056,
+  "merge_size": 2,
+  "min_pixels": 3136,
+  "patch_size": 14,
+  "processor_class": "Qwen2_5_VLProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "longest_edge": 12845056,
+    "shortest_edge": 3136
+  },
+  "temporal_patch_size": 2
+}

rng_state_0.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b70e8ccdc10215494e0c425708b960e8fa5b44650404f00877b72b073d94b178
+size 14512

rng_state_1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c0133f22fdd1fdcbce9e5106bc5b407a34b36866e3047076b1f5aee06230b5a9
+size 14512

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5eee858c5123a4279c3e1f7b81247343f356ac767940b2692a928ad929543214
+size 11422063

tokenizer_config.json

@@ -0,0 +1,209 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "processor_class": "Qwen2_5_VLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

trainer_state.json

@@ -0,0 +1,2833 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.0024896678783050343,
+  "eval_steps": 500,
+  "global_step": 200,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 98.46875,
+      "epoch": 1.244833939152517e-05,
+      "grad_norm": 0.7918580174446106,
+      "kl": 0.0,
+      "learning_rate": 9.999937758303043e-06,
+      "loss": 0.0,
+      "reward": 0.06212758191395551,
+      "reward_std": 0.17572333943098783,
+      "rewards/format_reward_rec": 0.03125,
+      "rewards/iou_reward": 0.03087758377660066,
+      "step": 1
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 96.3125,
+      "epoch": 2.489667878305034e-05,
+      "grad_norm": 0.4728325307369232,
+      "kl": 0.001476287841796875,
+      "learning_rate": 9.999875516606086e-06,
+      "loss": 0.0001,
+      "reward": 0.05815719813108444,
+      "reward_std": 0.1644933968782425,
+      "rewards/format_reward_rec": 0.03125,
+      "rewards/iou_reward": 0.02690719999372959,
+      "step": 2
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 104.34375,
+      "epoch": 3.7345018174575515e-05,
+      "grad_norm": 0.6685078740119934,
+      "kl": 0.0016937255859375,
+      "learning_rate": 9.999813274909128e-06,
+      "loss": 0.0001,
+      "reward": 0.2839890792965889,
+      "reward_std": 0.4464326649904251,
+      "rewards/format_reward_rec": 0.15625,
+      "rewards/iou_reward": 0.1277390792965889,
+      "step": 3
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 92.71875,
+      "epoch": 4.979335756610068e-05,
+      "grad_norm": 0.7356534004211426,
+      "kl": 0.00652313232421875,
+      "learning_rate": 9.99975103321217e-06,
+      "loss": 0.0003,
+      "reward": 0.43574684858322144,
+      "reward_std": 0.5346276015043259,
+      "rewards/format_reward_rec": 0.28125,
+      "rewards/iou_reward": 0.15449684113264084,
+      "step": 4
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 116.8125,
+      "epoch": 6.224169695762585e-05,
+      "grad_norm": 1.5218911170959473,
+      "kl": 0.01190185546875,
+      "learning_rate": 9.999688791515214e-06,
+      "loss": 0.0005,
+      "reward": 0.5201369971036911,
+      "reward_std": 0.7230685949325562,
+      "rewards/format_reward_rec": 0.34375,
+      "rewards/iou_reward": 0.1763869971036911,
+      "step": 5
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 100.28125,
+      "epoch": 7.469003634915103e-05,
+      "grad_norm": 1.0370837450027466,
+      "kl": 0.03338623046875,
+      "learning_rate": 9.999626549818255e-06,
+      "loss": 0.0013,
+      "reward": 0.45021720230579376,
+      "reward_std": 0.6490348875522614,
+      "rewards/format_reward_rec": 0.21875,
+      "rewards/iou_reward": 0.23146723210811615,
+      "step": 6
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 109.84375,
+      "epoch": 8.71383757406762e-05,
+      "grad_norm": 0.8345764875411987,
+      "kl": 0.03363037109375,
+      "learning_rate": 9.999564308121297e-06,
+      "loss": 0.0013,
+      "reward": 0.4857834577560425,
+      "reward_std": 0.7779734134674072,
+      "rewards/format_reward_rec": 0.3125,
+      "rewards/iou_reward": 0.17328345030546188,
+      "step": 7
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 101.71875,
+      "epoch": 9.958671513220136e-05,
+      "grad_norm": 0.857598066329956,
+      "kl": 0.03662109375,
+      "learning_rate": 9.999502066424341e-06,
+      "loss": 0.0015,
+      "reward": 0.9932036697864532,
+      "reward_std": 0.8116889894008636,
+      "rewards/format_reward_rec": 0.5625,
+      "rewards/iou_reward": 0.43070371448993683,
+      "step": 8
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 118.28125,
+      "epoch": 0.00011203505452372654,
+      "grad_norm": 0.6808584928512573,
+      "kl": 0.0491943359375,
+      "learning_rate": 9.999439824727383e-06,
+      "loss": 0.002,
+      "reward": 1.22513547539711,
+      "reward_std": 0.6850164234638214,
+      "rewards/format_reward_rec": 0.71875,
+      "rewards/iou_reward": 0.5063855350017548,
+      "step": 9
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 130.65625,
+      "epoch": 0.0001244833939152517,
+      "grad_norm": 0.7755190134048462,
+      "kl": 0.047607421875,
+      "learning_rate": 9.999377583030425e-06,
+      "loss": 0.0019,
+      "reward": 1.080191969871521,
+      "reward_std": 0.8103933334350586,
+      "rewards/format_reward_rec": 0.6875,
+      "rewards/iou_reward": 0.3926919847726822,
+      "step": 10
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 119.65625,
+      "epoch": 0.00013693173330677687,
+      "grad_norm": 0.8545856475830078,
+      "kl": 0.055908203125,
+      "learning_rate": 9.999315341333466e-06,
+      "loss": 0.0022,
+      "reward": 1.0447435975074768,
+      "reward_std": 0.7395115196704865,
+      "rewards/format_reward_rec": 0.71875,
+      "rewards/iou_reward": 0.3259935975074768,
+      "step": 11
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 138.25,
+      "epoch": 0.00014938007269830206,
+      "grad_norm": 0.5796331167221069,
+      "kl": 0.059326171875,
+      "learning_rate": 9.999253099636508e-06,
+      "loss": 0.0024,
+      "reward": 1.3631539940834045,
+      "reward_std": 0.5613991022109985,
+      "rewards/format_reward_rec": 0.875,
+      "rewards/iou_reward": 0.48815396428108215,
+      "step": 12
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 139.9375,
+      "epoch": 0.00016182841208982722,
+      "grad_norm": 0.7014759182929993,
+      "kl": 0.1033935546875,
+      "learning_rate": 9.999190857939552e-06,
+      "loss": 0.0041,
+      "reward": 1.2715444564819336,
+      "reward_std": 0.6332900673151016,
+      "rewards/format_reward_rec": 0.75,
+      "rewards/iou_reward": 0.5215444564819336,
+      "step": 13
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 109.25,
+      "epoch": 0.0001742767514813524,
+      "grad_norm": 0.6986764073371887,
+      "kl": 0.08349609375,
+      "learning_rate": 9.999128616242594e-06,
+      "loss": 0.0033,
+      "reward": 1.4144143462181091,
+      "reward_std": 0.5491724014282227,
+      "rewards/format_reward_rec": 0.84375,
+      "rewards/iou_reward": 0.5706643462181091,
+      "step": 14
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 138.71875,
+      "epoch": 0.00018672509087287755,
+      "grad_norm": 718.58984375,
+      "kl": 53.529296875,
+      "learning_rate": 9.999066374545636e-06,
+      "loss": 2.145,
+      "reward": 1.1624789834022522,
+      "reward_std": 0.586757242679596,
+      "rewards/format_reward_rec": 0.78125,
+      "rewards/iou_reward": 0.381228968501091,
+      "step": 15
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 106.46875,
+      "epoch": 0.00019917343026440272,
+      "grad_norm": 0.6615357995033264,
+      "kl": 0.064208984375,
+      "learning_rate": 9.99900413284868e-06,
+      "loss": 0.0026,
+      "reward": 1.4353669881820679,
+      "reward_std": 0.5828511416912079,
+      "rewards/format_reward_rec": 0.84375,
+      "rewards/iou_reward": 0.5916168987751007,
+      "step": 16
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 122.78125,
+      "epoch": 0.0002116217696559279,
+      "grad_norm": 0.719054102897644,
+      "kl": 0.081787109375,
+      "learning_rate": 9.998941891151721e-06,
+      "loss": 0.0033,
+      "reward": 1.6010385155677795,
+      "reward_std": 0.29098983108997345,
+      "rewards/format_reward_rec": 0.9375,
+      "rewards/iou_reward": 0.6635385453701019,
+      "step": 17
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 109.125,
+      "epoch": 0.00022407010904745308,
+      "grad_norm": 0.6034934520721436,
+      "kl": 0.075927734375,
+      "learning_rate": 9.998879649454763e-06,
+      "loss": 0.003,
+      "reward": 1.747658133506775,
+      "reward_std": 0.19360525161027908,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7476581633090973,
+      "step": 18
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 94.21875,
+      "epoch": 0.00023651844843897824,
+      "grad_norm": 0.7481162548065186,
+      "kl": 0.1044921875,
+      "learning_rate": 9.998817407757807e-06,
+      "loss": 0.0042,
+      "reward": 1.5863367319107056,
+      "reward_std": 0.4242909550666809,
+      "rewards/format_reward_rec": 0.9375,
+      "rewards/iou_reward": 0.6488366723060608,
+      "step": 19
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 104.75,
+      "epoch": 0.0002489667878305034,
+      "grad_norm": 0.643527626991272,
+      "kl": 0.07177734375,
+      "learning_rate": 9.998755166060848e-06,
+      "loss": 0.0029,
+      "reward": 1.3996891975402832,
+      "reward_std": 0.37390778958797455,
+      "rewards/format_reward_rec": 0.9375,
+      "rewards/iou_reward": 0.4621891975402832,
+      "step": 20
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 107.625,
+      "epoch": 0.00026141512722202857,
+      "grad_norm": 0.7482960224151611,
+      "kl": 0.098388671875,
+      "learning_rate": 9.99869292436389e-06,
+      "loss": 0.0039,
+      "reward": 1.5653824210166931,
+      "reward_std": 0.34851640462875366,
+      "rewards/format_reward_rec": 0.9375,
+      "rewards/iou_reward": 0.6278825104236603,
+      "step": 21
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 102.1875,
+      "epoch": 0.00027386346661355373,
+      "grad_norm": 0.6192396283149719,
+      "kl": 0.074462890625,
+      "learning_rate": 9.998630682666934e-06,
+      "loss": 0.003,
+      "reward": 1.8187173008918762,
+      "reward_std": 0.20407778769731522,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8187173306941986,
+      "step": 22
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 119.6875,
+      "epoch": 0.0002863118060050789,
+      "grad_norm": 0.6803566217422485,
+      "kl": 0.0623779296875,
+      "learning_rate": 9.998568440969976e-06,
+      "loss": 0.0025,
+      "reward": 1.60598224401474,
+      "reward_std": 0.34990501403808594,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.63723224401474,
+      "step": 23
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 95.78125,
+      "epoch": 0.0002987601453966041,
+      "grad_norm": 0.6281342506408691,
+      "kl": 0.092529296875,
+      "learning_rate": 9.998506199273018e-06,
+      "loss": 0.0037,
+      "reward": 1.672945261001587,
+      "reward_std": 0.4158570021390915,
+      "rewards/format_reward_rec": 0.9375,
+      "rewards/iou_reward": 0.7354452311992645,
+      "step": 24
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 80.875,
+      "epoch": 0.0003112084847881293,
+      "grad_norm": 0.7759396433830261,
+      "kl": 0.1015625,
+      "learning_rate": 9.998443957576061e-06,
+      "loss": 0.0041,
+      "reward": 1.627230942249298,
+      "reward_std": 0.34825482964515686,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.6584809124469757,
+      "step": 25
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 97.6875,
+      "epoch": 0.00032365682417965445,
+      "grad_norm": 0.6152196526527405,
+      "kl": 0.103515625,
+      "learning_rate": 9.998381715879103e-06,
+      "loss": 0.0041,
+      "reward": 1.6838378310203552,
+      "reward_std": 0.11833954602479935,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6838378608226776,
+      "step": 26
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 80.78125,
+      "epoch": 0.0003361051635711796,
+      "grad_norm": 0.5609109997749329,
+      "kl": 0.1142578125,
+      "learning_rate": 9.998319474182145e-06,
+      "loss": 0.0046,
+      "reward": 1.8090254664421082,
+      "reward_std": 0.062307149171829224,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8090254366397858,
+      "step": 27
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 76.71875,
+      "epoch": 0.0003485535029627048,
+      "grad_norm": 0.6533063054084778,
+      "kl": 0.13818359375,
+      "learning_rate": 9.998257232485187e-06,
+      "loss": 0.0055,
+      "reward": 1.6686185598373413,
+      "reward_std": 0.1674296036362648,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6686184704303741,
+      "step": 28
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 85.15625,
+      "epoch": 0.00036100184235422994,
+      "grad_norm": 0.5266671180725098,
+      "kl": 0.09375,
+      "learning_rate": 9.998194990788229e-06,
+      "loss": 0.0037,
+      "reward": 1.7991711497306824,
+      "reward_std": 0.2697625942528248,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.8304212093353271,
+      "step": 29
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 74.15625,
+      "epoch": 0.0003734501817457551,
+      "grad_norm": 0.680327296257019,
+      "kl": 0.12109375,
+      "learning_rate": 9.998132749091272e-06,
+      "loss": 0.0048,
+      "reward": 1.719760000705719,
+      "reward_std": 0.18976478278636932,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.7510099709033966,
+      "step": 30
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 89.96875,
+      "epoch": 0.00038589852113728027,
+      "grad_norm": 0.5414130687713623,
+      "kl": 0.090087890625,
+      "learning_rate": 9.998070507394314e-06,
+      "loss": 0.0036,
+      "reward": 1.6798821687698364,
+      "reward_std": 0.21795213222503662,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.7111322581768036,
+      "step": 31
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 84.9375,
+      "epoch": 0.00039834686052880544,
+      "grad_norm": 0.46933692693710327,
+      "kl": 0.1025390625,
+      "learning_rate": 9.998008265697356e-06,
+      "loss": 0.0041,
+      "reward": 1.7964898943901062,
+      "reward_std": 0.22818857431411743,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.827739804983139,
+      "step": 32
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 79.21875,
+      "epoch": 0.0004107951999203306,
+      "grad_norm": 0.6017041802406311,
+      "kl": 0.13720703125,
+      "learning_rate": 9.9979460240004e-06,
+      "loss": 0.0055,
+      "reward": 1.6301453113555908,
+      "reward_std": 0.27136652171611786,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6301453560590744,
+      "step": 33
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 82.5,
+      "epoch": 0.0004232435393118558,
+      "grad_norm": 0.5675345659255981,
+      "kl": 0.125732421875,
+      "learning_rate": 9.997883782303441e-06,
+      "loss": 0.005,
+      "reward": 1.5771546363830566,
+      "reward_std": 0.1715424843132496,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.5771546363830566,
+      "step": 34
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 95.15625,
+      "epoch": 0.000435691878703381,
+      "grad_norm": 0.6039296388626099,
+      "kl": 0.11669921875,
+      "learning_rate": 9.997821540606483e-06,
+      "loss": 0.0047,
+      "reward": 1.747186541557312,
+      "reward_std": 0.30833982676267624,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.7784366011619568,
+      "step": 35
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 79.28125,
+      "epoch": 0.00044814021809490615,
+      "grad_norm": 1.9511338472366333,
+      "kl": 0.32470703125,
+      "learning_rate": 9.997759298909527e-06,
+      "loss": 0.0129,
+      "reward": 1.6023600101470947,
+      "reward_std": 0.3212697505950928,
+      "rewards/format_reward_rec": 0.9375,
+      "rewards/iou_reward": 0.6648600399494171,
+      "step": 36
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 91.96875,
+      "epoch": 0.0004605885574864313,
+      "grad_norm": 0.6094495058059692,
+      "kl": 0.1591796875,
+      "learning_rate": 9.997697057212569e-06,
+      "loss": 0.0064,
+      "reward": 1.5261580348014832,
+      "reward_std": 0.3047035411000252,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.557407945394516,
+      "step": 37
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 84.28125,
+      "epoch": 0.0004730368968779565,
+      "grad_norm": 0.5133370161056519,
+      "kl": 0.10693359375,
+      "learning_rate": 9.99763481551561e-06,
+      "loss": 0.0043,
+      "reward": 1.5371721386909485,
+      "reward_std": 0.2661096602678299,
+      "rewards/format_reward_rec": 0.9375,
+      "rewards/iou_reward": 0.5996721386909485,
+      "step": 38
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 95.9375,
+      "epoch": 0.00048548523626948165,
+      "grad_norm": 0.47525426745414734,
+      "kl": 0.12255859375,
+      "learning_rate": 9.997572573818654e-06,
+      "loss": 0.0049,
+      "reward": 1.6892600059509277,
+      "reward_std": 0.1822982355952263,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6892601251602173,
+      "step": 39
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 90.0,
+      "epoch": 0.0004979335756610068,
+      "grad_norm": 0.5481269955635071,
+      "kl": 0.0859375,
+      "learning_rate": 9.997510332121696e-06,
+      "loss": 0.0034,
+      "reward": 1.669859528541565,
+      "reward_std": 0.16043629869818687,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6698595285415649,
+      "step": 40
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 87.34375,
+      "epoch": 0.000510381915052532,
+      "grad_norm": 0.574486494064331,
+      "kl": 0.110107421875,
+      "learning_rate": 9.997448090424738e-06,
+      "loss": 0.0044,
+      "reward": 1.6145691275596619,
+      "reward_std": 0.16043317317962646,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.6458190977573395,
+      "step": 41
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 105.21875,
+      "epoch": 0.0005228302544440571,
+      "grad_norm": 0.5472383499145508,
+      "kl": 0.091552734375,
+      "learning_rate": 9.997385848727781e-06,
+      "loss": 0.0037,
+      "reward": 1.6457700729370117,
+      "reward_std": 0.3335290104150772,
+      "rewards/format_reward_rec": 0.9375,
+      "rewards/iou_reward": 0.7082701325416565,
+      "step": 42
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 83.125,
+      "epoch": 0.0005352785938355823,
+      "grad_norm": 0.5541640520095825,
+      "kl": 0.14306640625,
+      "learning_rate": 9.997323607030823e-06,
+      "loss": 0.0057,
+      "reward": 1.6855499744415283,
+      "reward_std": 0.2959246411919594,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.7167999744415283,
+      "step": 43
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 81.5625,
+      "epoch": 0.0005477269332271075,
+      "grad_norm": 0.5674740672111511,
+      "kl": 0.13134765625,
+      "learning_rate": 9.997261365333865e-06,
+      "loss": 0.0053,
+      "reward": 1.514305830001831,
+      "reward_std": 0.22089600563049316,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.5143058151006699,
+      "step": 44
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 91.53125,
+      "epoch": 0.0005601752726186326,
+      "grad_norm": 0.5152494311332703,
+      "kl": 0.089599609375,
+      "learning_rate": 9.997199123636909e-06,
+      "loss": 0.0036,
+      "reward": 1.794326663017273,
+      "reward_std": 0.043453872203826904,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7943267226219177,
+      "step": 45
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 87.96875,
+      "epoch": 0.0005726236120101578,
+      "grad_norm": 0.5079981088638306,
+      "kl": 0.125,
+      "learning_rate": 9.99713688193995e-06,
+      "loss": 0.005,
+      "reward": 1.7662574648857117,
+      "reward_std": 0.17677413672208786,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7662573754787445,
+      "step": 46
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 84.0,
+      "epoch": 0.000585071951401683,
+      "grad_norm": 0.5033786296844482,
+      "kl": 0.1220703125,
+      "learning_rate": 9.997074640242992e-06,
+      "loss": 0.0049,
+      "reward": 1.4638007879257202,
+      "reward_std": 0.2003663182258606,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.4638008028268814,
+      "step": 47
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 80.875,
+      "epoch": 0.0005975202907932082,
+      "grad_norm": 0.5780296325683594,
+      "kl": 0.12060546875,
+      "learning_rate": 9.997012398546034e-06,
+      "loss": 0.0048,
+      "reward": 1.7631506323814392,
+      "reward_std": 0.2995036095380783,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.7944006323814392,
+      "step": 48
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 89.125,
+      "epoch": 0.0006099686301847334,
+      "grad_norm": 0.523960530757904,
+      "kl": 0.13037109375,
+      "learning_rate": 9.996950156849076e-06,
+      "loss": 0.0052,
+      "reward": 1.5915122628211975,
+      "reward_std": 0.292996384203434,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.5915123075246811,
+      "step": 49
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 84.875,
+      "epoch": 0.0006224169695762586,
+      "grad_norm": 0.5157907605171204,
+      "kl": 0.114013671875,
+      "learning_rate": 9.99688791515212e-06,
+      "loss": 0.0046,
+      "reward": 1.8732710480690002,
+      "reward_std": 0.039826540276408195,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8732710778713226,
+      "step": 50
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 88.9375,
+      "epoch": 0.0006348653089677837,
+      "grad_norm": 0.49286431074142456,
+      "kl": 0.13818359375,
+      "learning_rate": 9.996825673455162e-06,
+      "loss": 0.0055,
+      "reward": 1.7973515391349792,
+      "reward_std": 0.11208843067288399,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7973516285419464,
+      "step": 51
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 79.53125,
+      "epoch": 0.0006473136483593089,
+      "grad_norm": 0.5944603085517883,
+      "kl": 0.13671875,
+      "learning_rate": 9.996763431758203e-06,
+      "loss": 0.0054,
+      "reward": 1.6538516283035278,
+      "reward_std": 0.3369443193078041,
+      "rewards/format_reward_rec": 0.9375,
+      "rewards/iou_reward": 0.7163516879081726,
+      "step": 52
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 86.0,
+      "epoch": 0.0006597619877508341,
+      "grad_norm": 0.5229175090789795,
+      "kl": 0.113525390625,
+      "learning_rate": 9.996701190061247e-06,
+      "loss": 0.0045,
+      "reward": 1.688271403312683,
+      "reward_std": 0.16166172549128532,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6882712692022324,
+      "step": 53
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 76.84375,
+      "epoch": 0.0006722103271423592,
+      "grad_norm": 0.5642473697662354,
+      "kl": 0.13916015625,
+      "learning_rate": 9.996638948364289e-06,
+      "loss": 0.0056,
+      "reward": 1.757770597934723,
+      "reward_std": 0.1410532221198082,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7577706277370453,
+      "step": 54
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 79.96875,
+      "epoch": 0.0006846586665338844,
+      "grad_norm": 0.6692479848861694,
+      "kl": 0.1494140625,
+      "learning_rate": 9.99657670666733e-06,
+      "loss": 0.006,
+      "reward": 1.5787354111671448,
+      "reward_std": 0.18042169511318207,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.5787354409694672,
+      "step": 55
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 88.4375,
+      "epoch": 0.0006971070059254096,
+      "grad_norm": 0.6885099411010742,
+      "kl": 0.11669921875,
+      "learning_rate": 9.996514464970374e-06,
+      "loss": 0.0047,
+      "reward": 1.361689031124115,
+      "reward_std": 0.36215692572295666,
+      "rewards/format_reward_rec": 0.9375,
+      "rewards/iou_reward": 0.4241890572011471,
+      "step": 56
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 89.75,
+      "epoch": 0.0007095553453169347,
+      "grad_norm": 0.474343478679657,
+      "kl": 0.1025390625,
+      "learning_rate": 9.996452223273416e-06,
+      "loss": 0.0041,
+      "reward": 1.723258912563324,
+      "reward_std": 0.2072291597723961,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.723258912563324,
+      "step": 57
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 81.59375,
+      "epoch": 0.0007220036847084599,
+      "grad_norm": 0.5668300986289978,
+      "kl": 0.12841796875,
+      "learning_rate": 9.996389981576458e-06,
+      "loss": 0.0052,
+      "reward": 1.6465070843696594,
+      "reward_std": 0.09229656681418419,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6465071439743042,
+      "step": 58
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 86.59375,
+      "epoch": 0.000734452024099985,
+      "grad_norm": 0.8369623422622681,
+      "kl": 0.140625,
+      "learning_rate": 9.996327739879502e-06,
+      "loss": 0.0056,
+      "reward": 1.6136068105697632,
+      "reward_std": 0.3137252777814865,
+      "rewards/format_reward_rec": 0.9375,
+      "rewards/iou_reward": 0.6761067807674408,
+      "step": 59
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 78.0625,
+      "epoch": 0.0007469003634915102,
+      "grad_norm": 0.5556425452232361,
+      "kl": 0.15625,
+      "learning_rate": 9.996265498182544e-06,
+      "loss": 0.0063,
+      "reward": 1.7592704892158508,
+      "reward_std": 0.14777341671288013,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.759270429611206,
+      "step": 60
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 85.71875,
+      "epoch": 0.0007593487028830354,
+      "grad_norm": 0.5661209225654602,
+      "kl": 0.12939453125,
+      "learning_rate": 9.996203256485585e-06,
+      "loss": 0.0052,
+      "reward": 1.732073187828064,
+      "reward_std": 0.2148379534482956,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7320732176303864,
+      "step": 61
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 78.71875,
+      "epoch": 0.0007717970422745605,
+      "grad_norm": 0.6321675777435303,
+      "kl": 0.1748046875,
+      "learning_rate": 9.996141014788629e-06,
+      "loss": 0.007,
+      "reward": 1.6298596262931824,
+      "reward_std": 0.21304509788751602,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.6611096560955048,
+      "step": 62
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 93.71875,
+      "epoch": 0.0007842453816660857,
+      "grad_norm": 0.5161328911781311,
+      "kl": 0.1357421875,
+      "learning_rate": 9.996078773091671e-06,
+      "loss": 0.0054,
+      "reward": 1.7697656154632568,
+      "reward_std": 0.15015805140137672,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.769765704870224,
+      "step": 63
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 78.28125,
+      "epoch": 0.0007966937210576109,
+      "grad_norm": 0.5633348822593689,
+      "kl": 0.1611328125,
+      "learning_rate": 9.996016531394713e-06,
+      "loss": 0.0064,
+      "reward": 1.4602358937263489,
+      "reward_std": 0.115456972271204,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.4602358639240265,
+      "step": 64
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 87.46875,
+      "epoch": 0.000809142060449136,
+      "grad_norm": 0.6145336031913757,
+      "kl": 0.12646484375,
+      "learning_rate": 9.995954289697755e-06,
+      "loss": 0.0051,
+      "reward": 1.716072678565979,
+      "reward_std": 0.27911266684532166,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.747322678565979,
+      "step": 65
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 89.6875,
+      "epoch": 0.0008215903998406612,
+      "grad_norm": 0.5064393281936646,
+      "kl": 0.115966796875,
+      "learning_rate": 9.995892048000796e-06,
+      "loss": 0.0046,
+      "reward": 1.5941734313964844,
+      "reward_std": 0.1153794713318348,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.5941733419895172,
+      "step": 66
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 80.34375,
+      "epoch": 0.0008340387392321865,
+      "grad_norm": 0.502927303314209,
+      "kl": 0.140625,
+      "learning_rate": 9.99582980630384e-06,
+      "loss": 0.0056,
+      "reward": 1.737270712852478,
+      "reward_std": 0.09697960317134857,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7372707426548004,
+      "step": 67
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 79.75,
+      "epoch": 0.0008464870786237116,
+      "grad_norm": 0.5995090007781982,
+      "kl": 0.1298828125,
+      "learning_rate": 9.995767564606882e-06,
+      "loss": 0.0052,
+      "reward": 1.6859315037727356,
+      "reward_std": 0.10032923426479101,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6859315186738968,
+      "step": 68
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 91.15625,
+      "epoch": 0.0008589354180152368,
+      "grad_norm": 0.4770706295967102,
+      "kl": 0.10986328125,
+      "learning_rate": 9.995705322909924e-06,
+      "loss": 0.0044,
+      "reward": 1.5923967361450195,
+      "reward_std": 0.17281262017786503,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.5923967510461807,
+      "step": 69
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 84.3125,
+      "epoch": 0.000871383757406762,
+      "grad_norm": 0.5044773817062378,
+      "kl": 0.1328125,
+      "learning_rate": 9.995643081212967e-06,
+      "loss": 0.0053,
+      "reward": 1.5754359364509583,
+      "reward_std": 0.17663145437836647,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.5754359662532806,
+      "step": 70
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 69.8125,
+      "epoch": 0.0008838320967982871,
+      "grad_norm": 0.5421180725097656,
+      "kl": 0.14794921875,
+      "learning_rate": 9.99558083951601e-06,
+      "loss": 0.0059,
+      "reward": 1.800459086894989,
+      "reward_std": 0.07165651768445969,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8004591166973114,
+      "step": 71
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 84.65625,
+      "epoch": 0.0008962804361898123,
+      "grad_norm": 0.5785159468650818,
+      "kl": 0.1337890625,
+      "learning_rate": 9.995518597819051e-06,
+      "loss": 0.0054,
+      "reward": 1.6627878546714783,
+      "reward_std": 0.26127464324235916,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6627878248691559,
+      "step": 72
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 73.625,
+      "epoch": 0.0009087287755813375,
+      "grad_norm": 0.5469045042991638,
+      "kl": 0.11474609375,
+      "learning_rate": 9.995456356122095e-06,
+      "loss": 0.0046,
+      "reward": 1.7520496249198914,
+      "reward_std": 0.15470531303435564,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7520496249198914,
+      "step": 73
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 72.15625,
+      "epoch": 0.0009211771149728626,
+      "grad_norm": 0.5782791972160339,
+      "kl": 0.15185546875,
+      "learning_rate": 9.995394114425137e-06,
+      "loss": 0.0061,
+      "reward": 1.6950392723083496,
+      "reward_std": 0.20423278957605362,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.726289302110672,
+      "step": 74
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 70.3125,
+      "epoch": 0.0009336254543643878,
+      "grad_norm": 0.5352646112442017,
+      "kl": 0.1396484375,
+      "learning_rate": 9.995331872728178e-06,
+      "loss": 0.0056,
+      "reward": 1.8100340962409973,
+      "reward_std": 0.07194224745035172,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8100341260433197,
+      "step": 75
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 73.625,
+      "epoch": 0.000946073793755913,
+      "grad_norm": 0.5901323556900024,
+      "kl": 0.1396484375,
+      "learning_rate": 9.995269631031222e-06,
+      "loss": 0.0056,
+      "reward": 1.9025427103042603,
+      "reward_std": 0.09781728684902191,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.902542769908905,
+      "step": 76
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 76.0625,
+      "epoch": 0.0009585221331474381,
+      "grad_norm": 0.5916361212730408,
+      "kl": 0.14111328125,
+      "learning_rate": 9.995207389334264e-06,
+      "loss": 0.0056,
+      "reward": 1.637428343296051,
+      "reward_std": 0.2861202023923397,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6374283134937286,
+      "step": 77
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 78.0,
+      "epoch": 0.0009709704725389633,
+      "grad_norm": 0.534542441368103,
+      "kl": 0.12890625,
+      "learning_rate": 9.995145147637306e-06,
+      "loss": 0.0052,
+      "reward": 1.5602301359176636,
+      "reward_std": 0.26418060809373856,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.560230165719986,
+      "step": 78
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 80.78125,
+      "epoch": 0.0009834188119304886,
+      "grad_norm": 0.5377347469329834,
+      "kl": 0.1337890625,
+      "learning_rate": 9.99508290594035e-06,
+      "loss": 0.0054,
+      "reward": 1.7412559986114502,
+      "reward_std": 0.19341769814491272,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7412559390068054,
+      "step": 79
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 83.09375,
+      "epoch": 0.0009958671513220136,
+      "grad_norm": 0.5326669812202454,
+      "kl": 0.1416015625,
+      "learning_rate": 9.995020664243391e-06,
+      "loss": 0.0057,
+      "reward": 1.6354157328605652,
+      "reward_std": 0.09996325895190239,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6354157030582428,
+      "step": 80
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 74.375,
+      "epoch": 0.001008315490713539,
+      "grad_norm": 0.5222757458686829,
+      "kl": 0.14599609375,
+      "learning_rate": 9.994958422546433e-06,
+      "loss": 0.0058,
+      "reward": 1.8184998631477356,
+      "reward_std": 0.21548935770988464,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8184998333454132,
+      "step": 81
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 76.90625,
+      "epoch": 0.001020763830105064,
+      "grad_norm": 0.5388038158416748,
+      "kl": 0.158203125,
+      "learning_rate": 9.994896180849477e-06,
+      "loss": 0.0063,
+      "reward": 1.760369062423706,
+      "reward_std": 0.20370811223983765,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7603690922260284,
+      "step": 82
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 77.96875,
+      "epoch": 0.0010332121694965892,
+      "grad_norm": 0.6683233380317688,
+      "kl": 0.14599609375,
+      "learning_rate": 9.994833939152517e-06,
+      "loss": 0.0058,
+      "reward": 1.6503952145576477,
+      "reward_std": 0.28510017693042755,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.6816451847553253,
+      "step": 83
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 76.5,
+      "epoch": 0.0010456605088881143,
+      "grad_norm": 0.8178681135177612,
+      "kl": 0.17626953125,
+      "learning_rate": 9.99477169745556e-06,
+      "loss": 0.007,
+      "reward": 1.8033937811851501,
+      "reward_std": 0.11837778985500336,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8033937513828278,
+      "step": 84
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 75.4375,
+      "epoch": 0.0010581088482796396,
+      "grad_norm": 0.6322414875030518,
+      "kl": 0.1943359375,
+      "learning_rate": 9.994709455758602e-06,
+      "loss": 0.0078,
+      "reward": 1.789516270160675,
+      "reward_std": 0.1459340425208211,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.8207662999629974,
+      "step": 85
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 73.625,
+      "epoch": 0.0010705571876711646,
+      "grad_norm": 0.7803683876991272,
+      "kl": 0.1806640625,
+      "learning_rate": 9.994647214061644e-06,
+      "loss": 0.0072,
+      "reward": 1.580109417438507,
+      "reward_std": 0.17563226073980331,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.5801093876361847,
+      "step": 86
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 68.375,
+      "epoch": 0.0010830055270626899,
+      "grad_norm": 0.497260719537735,
+      "kl": 0.14697265625,
+      "learning_rate": 9.994584972364688e-06,
+      "loss": 0.0059,
+      "reward": 1.6402252912521362,
+      "reward_std": 0.11211172491312027,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6402253359556198,
+      "step": 87
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 76.375,
+      "epoch": 0.001095453866454215,
+      "grad_norm": 0.5990146398544312,
+      "kl": 0.17041015625,
+      "learning_rate": 9.99452273066773e-06,
+      "loss": 0.0068,
+      "reward": 1.7737177610397339,
+      "reward_std": 0.11195245012640953,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7737177312374115,
+      "step": 88
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 90.3125,
+      "epoch": 0.0011079022058457402,
+      "grad_norm": 0.579544186592102,
+      "kl": 0.14794921875,
+      "learning_rate": 9.994460488970771e-06,
+      "loss": 0.0059,
+      "reward": 1.4342612028121948,
+      "reward_std": 0.22582250833511353,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.465511217713356,
+      "step": 89
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 71.5625,
+      "epoch": 0.0011203505452372653,
+      "grad_norm": 0.626300573348999,
+      "kl": 0.15185546875,
+      "learning_rate": 9.994398247273815e-06,
+      "loss": 0.0061,
+      "reward": 1.830142080783844,
+      "reward_std": 0.09023091197013855,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.830142080783844,
+      "step": 90
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 70.4375,
+      "epoch": 0.0011327988846287905,
+      "grad_norm": 0.530746340751648,
+      "kl": 0.162109375,
+      "learning_rate": 9.994336005576857e-06,
+      "loss": 0.0065,
+      "reward": 1.7541658282279968,
+      "reward_std": 0.22513454407453537,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.754165768623352,
+      "step": 91
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 80.75,
+      "epoch": 0.0011452472240203156,
+      "grad_norm": 0.729465126991272,
+      "kl": 0.1552734375,
+      "learning_rate": 9.994273763879899e-06,
+      "loss": 0.0062,
+      "reward": 1.4567736983299255,
+      "reward_std": 0.3039446175098419,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.48802371323108673,
+      "step": 92
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 77.03125,
+      "epoch": 0.0011576955634118409,
+      "grad_norm": 0.747829020023346,
+      "kl": 0.19482421875,
+      "learning_rate": 9.994211522182942e-06,
+      "loss": 0.0078,
+      "reward": 1.6674769520759583,
+      "reward_std": 0.36814084649086,
+      "rewards/format_reward_rec": 0.9375,
+      "rewards/iou_reward": 0.7299769520759583,
+      "step": 93
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 73.1875,
+      "epoch": 0.001170143902803366,
+      "grad_norm": 0.7084731459617615,
+      "kl": 0.14501953125,
+      "learning_rate": 9.994149280485984e-06,
+      "loss": 0.0058,
+      "reward": 1.8480735421180725,
+      "reward_std": 0.19576817378401756,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.8793235719203949,
+      "step": 94
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 80.0,
+      "epoch": 0.0011825922421948912,
+      "grad_norm": 0.6167282462120056,
+      "kl": 0.14404296875,
+      "learning_rate": 9.994087038789026e-06,
+      "loss": 0.0058,
+      "reward": 1.7454318404197693,
+      "reward_std": 0.0950808608904481,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7454318404197693,
+      "step": 95
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 85.375,
+      "epoch": 0.0011950405815864165,
+      "grad_norm": 0.6630386114120483,
+      "kl": 0.1318359375,
+      "learning_rate": 9.99402479709207e-06,
+      "loss": 0.0053,
+      "reward": 1.800285816192627,
+      "reward_std": 0.22147530317306519,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8002857863903046,
+      "step": 96
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 76.5,
+      "epoch": 0.0012074889209779415,
+      "grad_norm": 0.6420366168022156,
+      "kl": 0.166015625,
+      "learning_rate": 9.993962555395111e-06,
+      "loss": 0.0066,
+      "reward": 1.8380752205848694,
+      "reward_std": 0.15962522476911545,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8380752801895142,
+      "step": 97
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 81.28125,
+      "epoch": 0.0012199372603694668,
+      "grad_norm": 0.5716676115989685,
+      "kl": 0.140625,
+      "learning_rate": 9.993900313698153e-06,
+      "loss": 0.0056,
+      "reward": 1.7347316145896912,
+      "reward_std": 0.14502982422709465,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7347316443920135,
+      "step": 98
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 77.40625,
+      "epoch": 0.0012323855997609919,
+      "grad_norm": 0.6021421551704407,
+      "kl": 0.13916015625,
+      "learning_rate": 9.993838072001197e-06,
+      "loss": 0.0056,
+      "reward": 1.6757659316062927,
+      "reward_std": 0.21641412377357483,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6757659018039703,
+      "step": 99
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 73.84375,
+      "epoch": 0.0012448339391525171,
+      "grad_norm": 0.5074762105941772,
+      "kl": 0.1884765625,
+      "learning_rate": 9.993775830304239e-06,
+      "loss": 0.0075,
+      "reward": 1.7514841556549072,
+      "reward_std": 0.17918928153812885,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.7827341556549072,
+      "step": 100
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 73.125,
+      "epoch": 0.0012572822785440422,
+      "grad_norm": 0.5177865624427795,
+      "kl": 0.128662109375,
+      "learning_rate": 9.99371358860728e-06,
+      "loss": 0.0051,
+      "reward": 1.786689281463623,
+      "reward_std": 0.14574366062879562,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7866893112659454,
+      "step": 101
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 75.65625,
+      "epoch": 0.0012697306179355675,
+      "grad_norm": 0.5310884714126587,
+      "kl": 0.1455078125,
+      "learning_rate": 9.993651346910322e-06,
+      "loss": 0.0058,
+      "reward": 1.7609416246414185,
+      "reward_std": 0.16141251474618912,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7609416246414185,
+      "step": 102
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 74.28125,
+      "epoch": 0.0012821789573270925,
+      "grad_norm": 0.6390679478645325,
+      "kl": 0.15283203125,
+      "learning_rate": 9.993589105213364e-06,
+      "loss": 0.0061,
+      "reward": 1.894408941268921,
+      "reward_std": 0.09809810575097799,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8944090008735657,
+      "step": 103
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 78.46875,
+      "epoch": 0.0012946272967186178,
+      "grad_norm": 0.6070464849472046,
+      "kl": 0.173828125,
+      "learning_rate": 9.993526863516408e-06,
+      "loss": 0.007,
+      "reward": 1.7859740853309631,
+      "reward_std": 0.11464018002152443,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7859741151332855,
+      "step": 104
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 87.875,
+      "epoch": 0.0013070756361101428,
+      "grad_norm": 0.5264771580696106,
+      "kl": 0.116943359375,
+      "learning_rate": 9.99346462181945e-06,
+      "loss": 0.0047,
+      "reward": 1.7116284370422363,
+      "reward_std": 0.19027460366487503,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7116283774375916,
+      "step": 105
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 75.46875,
+      "epoch": 0.0013195239755016681,
+      "grad_norm": 0.5975978970527649,
+      "kl": 0.15576171875,
+      "learning_rate": 9.993402380122492e-06,
+      "loss": 0.0062,
+      "reward": 1.693192720413208,
+      "reward_std": 0.23123470693826675,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6931926906108856,
+      "step": 106
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 83.96875,
+      "epoch": 0.0013319723148931932,
+      "grad_norm": 0.49909281730651855,
+      "kl": 0.150390625,
+      "learning_rate": 9.993340138425535e-06,
+      "loss": 0.006,
+      "reward": 1.4939433932304382,
+      "reward_std": 0.23037827014923096,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.49394336342811584,
+      "step": 107
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 77.71875,
+      "epoch": 0.0013444206542847185,
+      "grad_norm": 0.5423364639282227,
+      "kl": 0.15478515625,
+      "learning_rate": 9.993277896728577e-06,
+      "loss": 0.0062,
+      "reward": 1.6659827828407288,
+      "reward_std": 0.1452256366610527,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6659828424453735,
+      "step": 108
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 80.03125,
+      "epoch": 0.0013568689936762435,
+      "grad_norm": 0.4949978291988373,
+      "kl": 0.12939453125,
+      "learning_rate": 9.993215655031619e-06,
+      "loss": 0.0052,
+      "reward": 1.6230355501174927,
+      "reward_std": 0.18467745929956436,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6230354905128479,
+      "step": 109
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 77.25,
+      "epoch": 0.0013693173330677688,
+      "grad_norm": 0.5784613490104675,
+      "kl": 0.11962890625,
+      "learning_rate": 9.993153413334662e-06,
+      "loss": 0.0048,
+      "reward": 1.7517021894454956,
+      "reward_std": 0.15935295075178146,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7517021894454956,
+      "step": 110
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 85.4375,
+      "epoch": 0.0013817656724592938,
+      "grad_norm": 0.4976905286312103,
+      "kl": 0.154296875,
+      "learning_rate": 9.993091171637704e-06,
+      "loss": 0.0062,
+      "reward": 1.6756169199943542,
+      "reward_std": 0.16474511474370956,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.7068668603897095,
+      "step": 111
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 86.40625,
+      "epoch": 0.0013942140118508191,
+      "grad_norm": 0.4809585511684418,
+      "kl": 0.139404296875,
+      "learning_rate": 9.993028929940746e-06,
+      "loss": 0.0056,
+      "reward": 1.6836587190628052,
+      "reward_std": 0.16580590046942234,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6836587190628052,
+      "step": 112
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 84.84375,
+      "epoch": 0.0014066623512423442,
+      "grad_norm": 0.5611812472343445,
+      "kl": 0.12060546875,
+      "learning_rate": 9.99296668824379e-06,
+      "loss": 0.0048,
+      "reward": 1.707568645477295,
+      "reward_std": 0.4278259873390198,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.7388187050819397,
+      "step": 113
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 81.625,
+      "epoch": 0.0014191106906338694,
+      "grad_norm": 0.5960983633995056,
+      "kl": 0.16552734375,
+      "learning_rate": 9.992904446546832e-06,
+      "loss": 0.0066,
+      "reward": 1.8632564544677734,
+      "reward_std": 0.08219528943300247,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8632563948631287,
+      "step": 114
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 77.46875,
+      "epoch": 0.0014315590300253947,
+      "grad_norm": 0.6228426098823547,
+      "kl": 0.17138671875,
+      "learning_rate": 9.992842204849874e-06,
+      "loss": 0.0069,
+      "reward": 1.8939569592475891,
+      "reward_std": 0.05686322785913944,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8939569890499115,
+      "step": 115
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 79.4375,
+      "epoch": 0.0014440073694169198,
+      "grad_norm": 0.5188966989517212,
+      "kl": 0.15625,
+      "learning_rate": 9.992779963152917e-06,
+      "loss": 0.0063,
+      "reward": 1.8855347633361816,
+      "reward_std": 0.08866238407790661,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8855347335338593,
+      "step": 116
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 76.25,
+      "epoch": 0.001456455708808445,
+      "grad_norm": 0.5806636214256287,
+      "kl": 0.16650390625,
+      "learning_rate": 9.992717721455959e-06,
+      "loss": 0.0067,
+      "reward": 1.6434407830238342,
+      "reward_std": 0.1430330127477646,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6434408128261566,
+      "step": 117
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 80.625,
+      "epoch": 0.00146890404819997,
+      "grad_norm": 0.5318806767463684,
+      "kl": 0.15380859375,
+      "learning_rate": 9.992655479759e-06,
+      "loss": 0.0061,
+      "reward": 1.5443456172943115,
+      "reward_std": 0.2518259510397911,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.5443456172943115,
+      "step": 118
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 86.0,
+      "epoch": 0.0014813523875914954,
+      "grad_norm": 0.4761650264263153,
+      "kl": 0.1474609375,
+      "learning_rate": 9.992593238062044e-06,
+      "loss": 0.0059,
+      "reward": 1.7887638807296753,
+      "reward_std": 0.11934101954102516,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7887638509273529,
+      "step": 119
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 87.875,
+      "epoch": 0.0014938007269830204,
+      "grad_norm": 0.5443623661994934,
+      "kl": 0.15478515625,
+      "learning_rate": 9.992530996365085e-06,
+      "loss": 0.0062,
+      "reward": 1.6878383159637451,
+      "reward_std": 0.29080618917942047,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.7190883457660675,
+      "step": 120
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 75.0625,
+      "epoch": 0.0015062490663745457,
+      "grad_norm": 0.5305638313293457,
+      "kl": 0.1748046875,
+      "learning_rate": 9.992468754668128e-06,
+      "loss": 0.007,
+      "reward": 1.6585879921913147,
+      "reward_std": 0.08734836708754301,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6585879623889923,
+      "step": 121
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 81.84375,
+      "epoch": 0.0015186974057660708,
+      "grad_norm": 0.5158679485321045,
+      "kl": 0.1240234375,
+      "learning_rate": 9.99240651297117e-06,
+      "loss": 0.0049,
+      "reward": 1.8771040439605713,
+      "reward_std": 0.07788556441664696,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8771041035652161,
+      "step": 122
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 89.03125,
+      "epoch": 0.001531145745157596,
+      "grad_norm": 0.5314344167709351,
+      "kl": 0.11328125,
+      "learning_rate": 9.992344271274212e-06,
+      "loss": 0.0045,
+      "reward": 1.717368245124817,
+      "reward_std": 0.12311140447854996,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7173681855201721,
+      "step": 123
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 83.53125,
+      "epoch": 0.001543594084549121,
+      "grad_norm": 0.4630969762802124,
+      "kl": 0.1318359375,
+      "learning_rate": 9.992282029577255e-06,
+      "loss": 0.0053,
+      "reward": 1.5514342188835144,
+      "reward_std": 0.2621281296014786,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.5826842486858368,
+      "step": 124
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 82.25,
+      "epoch": 0.0015560424239406464,
+      "grad_norm": 0.5010308623313904,
+      "kl": 0.11767578125,
+      "learning_rate": 9.992219787880297e-06,
+      "loss": 0.0047,
+      "reward": 1.5915331840515137,
+      "reward_std": 0.1749160811305046,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.5915331840515137,
+      "step": 125
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 73.125,
+      "epoch": 0.0015684907633321714,
+      "grad_norm": 0.4550737738609314,
+      "kl": 0.139892578125,
+      "learning_rate": 9.99215754618334e-06,
+      "loss": 0.0056,
+      "reward": 1.7983530759811401,
+      "reward_std": 0.15214556828141212,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7983531057834625,
+      "step": 126
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 80.53125,
+      "epoch": 0.0015809391027236967,
+      "grad_norm": 0.47265562415122986,
+      "kl": 0.166015625,
+      "learning_rate": 9.992095304486383e-06,
+      "loss": 0.0066,
+      "reward": 1.623586893081665,
+      "reward_std": 0.0915629081428051,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6235868632793427,
+      "step": 127
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 71.96875,
+      "epoch": 0.0015933874421152217,
+      "grad_norm": 0.5316770076751709,
+      "kl": 0.12255859375,
+      "learning_rate": 9.992033062789425e-06,
+      "loss": 0.0049,
+      "reward": 1.9262670278549194,
+      "reward_std": 0.019816839136183262,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.926266998052597,
+      "step": 128
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 78.90625,
+      "epoch": 0.001605835781506747,
+      "grad_norm": 0.4828510582447052,
+      "kl": 0.131103515625,
+      "learning_rate": 9.991970821092466e-06,
+      "loss": 0.0052,
+      "reward": 1.5956175923347473,
+      "reward_std": 0.11756857857108116,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.5956175774335861,
+      "step": 129
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 92.4375,
+      "epoch": 0.001618284120898272,
+      "grad_norm": 0.46744272112846375,
+      "kl": 0.110107421875,
+      "learning_rate": 9.99190857939551e-06,
+      "loss": 0.0044,
+      "reward": 1.9298859238624573,
+      "reward_std": 0.036631692200899124,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.9298859238624573,
+      "step": 130
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 76.8125,
+      "epoch": 0.0016307324602897974,
+      "grad_norm": 0.5099858641624451,
+      "kl": 0.150390625,
+      "learning_rate": 9.991846337698552e-06,
+      "loss": 0.006,
+      "reward": 1.8364751935005188,
+      "reward_std": 0.0698028914630413,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8364751935005188,
+      "step": 131
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 83.0,
+      "epoch": 0.0016431807996813224,
+      "grad_norm": 0.5306047797203064,
+      "kl": 0.151123046875,
+      "learning_rate": 9.991784096001594e-06,
+      "loss": 0.006,
+      "reward": 1.7881332039833069,
+      "reward_std": 0.25462284684181213,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7881332635879517,
+      "step": 132
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 71.03125,
+      "epoch": 0.0016556291390728477,
+      "grad_norm": 0.5165503621101379,
+      "kl": 0.1376953125,
+      "learning_rate": 9.991721854304637e-06,
+      "loss": 0.0055,
+      "reward": 1.8669943809509277,
+      "reward_std": 0.10201262310147285,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8669944405555725,
+      "step": 133
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 80.46875,
+      "epoch": 0.001668077478464373,
+      "grad_norm": 0.5238383412361145,
+      "kl": 0.125,
+      "learning_rate": 9.99165961260768e-06,
+      "loss": 0.005,
+      "reward": 1.8666821718215942,
+      "reward_std": 0.16252516955137253,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.897932231426239,
+      "step": 134
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 86.96875,
+      "epoch": 0.001680525817855898,
+      "grad_norm": 3.0103187561035156,
+      "kl": 0.369140625,
+      "learning_rate": 9.991597370910721e-06,
+      "loss": 0.0148,
+      "reward": 1.6562628746032715,
+      "reward_std": 0.061989203095436096,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.6875128149986267,
+      "step": 135
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 77.96875,
+      "epoch": 0.0016929741572474233,
+      "grad_norm": 0.5688140988349915,
+      "kl": 0.11865234375,
+      "learning_rate": 9.991535129213765e-06,
+      "loss": 0.0047,
+      "reward": 1.6850191354751587,
+      "reward_std": 0.1395672708749771,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6850191354751587,
+      "step": 136
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 76.40625,
+      "epoch": 0.0017054224966389483,
+      "grad_norm": 0.5144866704940796,
+      "kl": 0.1123046875,
+      "learning_rate": 9.991472887516807e-06,
+      "loss": 0.0045,
+      "reward": 1.8065414428710938,
+      "reward_std": 0.11821011267602444,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8065415024757385,
+      "step": 137
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 78.59375,
+      "epoch": 0.0017178708360304736,
+      "grad_norm": 0.5437494516372681,
+      "kl": 0.12744140625,
+      "learning_rate": 9.991410645819848e-06,
+      "loss": 0.0051,
+      "reward": 1.751151978969574,
+      "reward_std": 0.278888332657516,
+      "rewards/format_reward_rec": 0.9375,
+      "rewards/iou_reward": 0.813651978969574,
+      "step": 138
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 77.8125,
+      "epoch": 0.0017303191754219987,
+      "grad_norm": 0.5118846297264099,
+      "kl": 0.13037109375,
+      "learning_rate": 9.99134840412289e-06,
+      "loss": 0.0052,
+      "reward": 1.6839856505393982,
+      "reward_std": 0.10992167890071869,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.715235635638237,
+      "step": 139
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 92.78125,
+      "epoch": 0.001742767514813524,
+      "grad_norm": 0.6501355171203613,
+      "kl": 0.1220703125,
+      "learning_rate": 9.991286162425932e-06,
+      "loss": 0.0049,
+      "reward": 1.5675995349884033,
+      "reward_std": 0.43148788809776306,
+      "rewards/format_reward_rec": 0.9375,
+      "rewards/iou_reward": 0.6300995051860809,
+      "step": 140
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 84.28125,
+      "epoch": 0.001755215854205049,
+      "grad_norm": 0.5102176666259766,
+      "kl": 0.12353515625,
+      "learning_rate": 9.991223920728976e-06,
+      "loss": 0.0049,
+      "reward": 1.8060396313667297,
+      "reward_std": 0.142087172716856,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8060396909713745,
+      "step": 141
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 87.34375,
+      "epoch": 0.0017676641935965743,
+      "grad_norm": 0.561132550239563,
+      "kl": 0.138916015625,
+      "learning_rate": 9.991161679032018e-06,
+      "loss": 0.0055,
+      "reward": 1.6661274433135986,
+      "reward_std": 0.3482854291796684,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.6973774135112762,
+      "step": 142
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 82.90625,
+      "epoch": 0.0017801125329880993,
+      "grad_norm": 0.555660605430603,
+      "kl": 0.12939453125,
+      "learning_rate": 9.99109943733506e-06,
+      "loss": 0.0052,
+      "reward": 1.7078312635421753,
+      "reward_std": 0.2983357608318329,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.7390812933444977,
+      "step": 143
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 86.15625,
+      "epoch": 0.0017925608723796246,
+      "grad_norm": 0.5917304754257202,
+      "kl": 0.115966796875,
+      "learning_rate": 9.991037195638103e-06,
+      "loss": 0.0046,
+      "reward": 1.6995146870613098,
+      "reward_std": 0.176687341183424,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6995146870613098,
+      "step": 144
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 76.5625,
+      "epoch": 0.0018050092117711497,
+      "grad_norm": 0.49189749360084534,
+      "kl": 0.122314453125,
+      "learning_rate": 9.990974953941145e-06,
+      "loss": 0.0049,
+      "reward": 1.8386791944503784,
+      "reward_std": 0.11668524146080017,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8386792242527008,
+      "step": 145
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 72.03125,
+      "epoch": 0.001817457551162675,
+      "grad_norm": 0.5640706419944763,
+      "kl": 0.101318359375,
+      "learning_rate": 9.990912712244187e-06,
+      "loss": 0.0041,
+      "reward": 1.6799483895301819,
+      "reward_std": 0.19784042239189148,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.7111983299255371,
+      "step": 146
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 82.75,
+      "epoch": 0.0018299058905542,
+      "grad_norm": 0.6259125471115112,
+      "kl": 0.131103515625,
+      "learning_rate": 9.99085047054723e-06,
+      "loss": 0.0052,
+      "reward": 1.6741862893104553,
+      "reward_std": 0.05047000013291836,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6741862595081329,
+      "step": 147
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 81.375,
+      "epoch": 0.0018423542299457253,
+      "grad_norm": 0.6438843011856079,
+      "kl": 0.16552734375,
+      "learning_rate": 9.990788228850272e-06,
+      "loss": 0.0066,
+      "reward": 1.6975774765014648,
+      "reward_std": 0.3462005481123924,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.7288274765014648,
+      "step": 148
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 82.5,
+      "epoch": 0.0018548025693372503,
+      "grad_norm": 0.5777101516723633,
+      "kl": 0.12451171875,
+      "learning_rate": 9.990725987153314e-06,
+      "loss": 0.005,
+      "reward": 1.7958465814590454,
+      "reward_std": 0.29284024983644485,
+      "rewards/format_reward_rec": 0.9375,
+      "rewards/iou_reward": 0.8583464920520782,
+      "step": 149
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 77.9375,
+      "epoch": 0.0018672509087287756,
+      "grad_norm": 0.5476696491241455,
+      "kl": 0.128662109375,
+      "learning_rate": 9.990663745456358e-06,
+      "loss": 0.0051,
+      "reward": 1.7851470112800598,
+      "reward_std": 0.15798081643879414,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.8163970410823822,
+      "step": 150
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 80.25,
+      "epoch": 0.0018796992481203006,
+      "grad_norm": 0.48750340938568115,
+      "kl": 0.122802734375,
+      "learning_rate": 9.9906015037594e-06,
+      "loss": 0.0049,
+      "reward": 1.724117934703827,
+      "reward_std": 0.17644132114946842,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7241179645061493,
+      "step": 151
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 87.5,
+      "epoch": 0.001892147587511826,
+      "grad_norm": 0.6843100786209106,
+      "kl": 0.1123046875,
+      "learning_rate": 9.990539262062441e-06,
+      "loss": 0.0045,
+      "reward": 1.801917552947998,
+      "reward_std": 0.1792924776673317,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.8331675827503204,
+      "step": 152
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 89.0625,
+      "epoch": 0.0019045959269033512,
+      "grad_norm": 0.5321158766746521,
+      "kl": 0.11376953125,
+      "learning_rate": 9.990477020365485e-06,
+      "loss": 0.0045,
+      "reward": 1.631728708744049,
+      "reward_std": 0.2040170580148697,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.6629787236452103,
+      "step": 153
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 90.53125,
+      "epoch": 0.0019170442662948763,
+      "grad_norm": 0.5559011697769165,
+      "kl": 0.100341796875,
+      "learning_rate": 9.990414778668527e-06,
+      "loss": 0.004,
+      "reward": 1.7384355068206787,
+      "reward_std": 0.3657010346651077,
+      "rewards/format_reward_rec": 0.9375,
+      "rewards/iou_reward": 0.8009354472160339,
+      "step": 154
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 80.8125,
+      "epoch": 0.0019294926056864015,
+      "grad_norm": 0.5985516905784607,
+      "kl": 0.109619140625,
+      "learning_rate": 9.990352536971569e-06,
+      "loss": 0.0044,
+      "reward": 1.7658506035804749,
+      "reward_std": 0.1951538361608982,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.7971006333827972,
+      "step": 155
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 91.375,
+      "epoch": 0.0019419409450779266,
+      "grad_norm": 0.5207005739212036,
+      "kl": 0.124267578125,
+      "learning_rate": 9.990290295274612e-06,
+      "loss": 0.005,
+      "reward": 1.6580237746238708,
+      "reward_std": 0.20409102737903595,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.6892738342285156,
+      "step": 156
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 86.40625,
+      "epoch": 0.001954389284469452,
+      "grad_norm": 0.614124059677124,
+      "kl": 0.11474609375,
+      "learning_rate": 9.990228053577652e-06,
+      "loss": 0.0046,
+      "reward": 1.6734708547592163,
+      "reward_std": 0.24323870986700058,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.6734707951545715,
+      "step": 157
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 99.1875,
+      "epoch": 0.001966837623860977,
+      "grad_norm": 0.5424925684928894,
+      "kl": 0.098388671875,
+      "learning_rate": 9.990165811880696e-06,
+      "loss": 0.0039,
+      "reward": 1.6296205520629883,
+      "reward_std": 0.20492691174149513,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.6608706414699554,
+      "step": 158
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 84.25,
+      "epoch": 0.001979285963252502,
+      "grad_norm": 0.5724992156028748,
+      "kl": 0.1376953125,
+      "learning_rate": 9.990103570183738e-06,
+      "loss": 0.0055,
+      "reward": 1.5222105383872986,
+      "reward_std": 0.28245383501052856,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.5534605383872986,
+      "step": 159
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 76.09375,
+      "epoch": 0.0019917343026440272,
+      "grad_norm": 0.5227129459381104,
+      "kl": 0.146484375,
+      "learning_rate": 9.99004132848678e-06,
+      "loss": 0.0059,
+      "reward": 1.8685023188591003,
+      "reward_std": 0.0524298120290041,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8685023784637451,
+      "step": 160
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 73.875,
+      "epoch": 0.0020041826420355525,
+      "grad_norm": 0.6109131574630737,
+      "kl": 0.1123046875,
+      "learning_rate": 9.989979086789823e-06,
+      "loss": 0.0045,
+      "reward": 1.871379554271698,
+      "reward_std": 0.09691573679447174,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8713796138763428,
+      "step": 161
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 75.34375,
+      "epoch": 0.002016630981427078,
+      "grad_norm": 0.5628411769866943,
+      "kl": 0.139892578125,
+      "learning_rate": 9.989916845092865e-06,
+      "loss": 0.0056,
+      "reward": 1.856031894683838,
+      "reward_std": 0.05577436415478587,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8560318946838379,
+      "step": 162
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 83.625,
+      "epoch": 0.0020290793208186026,
+      "grad_norm": 0.5170895457267761,
+      "kl": 0.1171875,
+      "learning_rate": 9.989854603395907e-06,
+      "loss": 0.0047,
+      "reward": 1.7793619632720947,
+      "reward_std": 0.1213589683175087,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7793619930744171,
+      "step": 163
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 90.5625,
+      "epoch": 0.002041527660210128,
+      "grad_norm": 0.49487820267677307,
+      "kl": 0.111328125,
+      "learning_rate": 9.98979236169895e-06,
+      "loss": 0.0045,
+      "reward": 1.5895143151283264,
+      "reward_std": 0.15016086027026176,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.5895143449306488,
+      "step": 164
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 78.875,
+      "epoch": 0.002053975999601653,
+      "grad_norm": 0.6349457502365112,
+      "kl": 0.1298828125,
+      "learning_rate": 9.989730120001992e-06,
+      "loss": 0.0052,
+      "reward": 1.5769969820976257,
+      "reward_std": 0.23748912662267685,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.5769969820976257,
+      "step": 165
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 92.125,
+      "epoch": 0.0020664243389931784,
+      "grad_norm": 0.7995075583457947,
+      "kl": 0.17919921875,
+      "learning_rate": 9.989667878305034e-06,
+      "loss": 0.0072,
+      "reward": 1.7468191981315613,
+      "reward_std": 0.37003619968891144,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.7780691385269165,
+      "step": 166
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 80.09375,
+      "epoch": 0.0020788726783847033,
+      "grad_norm": 0.5599526762962341,
+      "kl": 0.1416015625,
+      "learning_rate": 9.989605636608078e-06,
+      "loss": 0.0057,
+      "reward": 1.8237193822860718,
+      "reward_std": 0.10361789353191853,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8237193524837494,
+      "step": 167
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 74.15625,
+      "epoch": 0.0020913210177762286,
+      "grad_norm": 0.5949897170066833,
+      "kl": 0.1611328125,
+      "learning_rate": 9.98954339491112e-06,
+      "loss": 0.0065,
+      "reward": 1.7989261150360107,
+      "reward_std": 0.13571097142994404,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7989261448383331,
+      "step": 168
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 78.625,
+      "epoch": 0.002103769357167754,
+      "grad_norm": 0.5042181611061096,
+      "kl": 0.10107421875,
+      "learning_rate": 9.989481153214162e-06,
+      "loss": 0.004,
+      "reward": 1.9031283855438232,
+      "reward_std": 0.029928937554359436,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.9031283855438232,
+      "step": 169
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 80.96875,
+      "epoch": 0.002116217696559279,
+      "grad_norm": 0.49940019845962524,
+      "kl": 0.1201171875,
+      "learning_rate": 9.989418911517205e-06,
+      "loss": 0.0048,
+      "reward": 1.9246177077293396,
+      "reward_std": 0.02288174256682396,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.924617737531662,
+      "step": 170
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 78.8125,
+      "epoch": 0.0021286660359508044,
+      "grad_norm": 0.6179351806640625,
+      "kl": 0.1689453125,
+      "learning_rate": 9.989356669820247e-06,
+      "loss": 0.0068,
+      "reward": 1.7295928597450256,
+      "reward_std": 0.11133578047156334,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7295928299427032,
+      "step": 171
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 78.625,
+      "epoch": 0.0021411143753423292,
+      "grad_norm": 0.5767012238502502,
+      "kl": 0.13427734375,
+      "learning_rate": 9.989294428123289e-06,
+      "loss": 0.0054,
+      "reward": 1.733388900756836,
+      "reward_std": 0.09509897604584694,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7333889603614807,
+      "step": 172
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 84.5625,
+      "epoch": 0.0021535627147338545,
+      "grad_norm": 0.6296212077140808,
+      "kl": 0.110107421875,
+      "learning_rate": 9.989232186426333e-06,
+      "loss": 0.0044,
+      "reward": 1.734093189239502,
+      "reward_std": 0.1716211587190628,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7340930998325348,
+      "step": 173
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 78.875,
+      "epoch": 0.0021660110541253798,
+      "grad_norm": 0.6160764098167419,
+      "kl": 0.134765625,
+      "learning_rate": 9.989169944729374e-06,
+      "loss": 0.0054,
+      "reward": 1.6963000297546387,
+      "reward_std": 0.155071921646595,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.7275499999523163,
+      "step": 174
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 75.625,
+      "epoch": 0.002178459393516905,
+      "grad_norm": 0.7459288239479065,
+      "kl": 0.16015625,
+      "learning_rate": 9.989107703032416e-06,
+      "loss": 0.0064,
+      "reward": 1.748288869857788,
+      "reward_std": 0.06110543105751276,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7482888698577881,
+      "step": 175
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 85.4375,
+      "epoch": 0.00219090773290843,
+      "grad_norm": 0.5298272371292114,
+      "kl": 0.11181640625,
+      "learning_rate": 9.989045461335458e-06,
+      "loss": 0.0045,
+      "reward": 1.9230260848999023,
+      "reward_std": 0.017507225275039673,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.92302605509758,
+      "step": 176
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 89.75,
+      "epoch": 0.002203356072299955,
+      "grad_norm": 0.5735234022140503,
+      "kl": 0.12841796875,
+      "learning_rate": 9.9889832196385e-06,
+      "loss": 0.0051,
+      "reward": 1.8215650916099548,
+      "reward_std": 0.08769259601831436,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8215650618076324,
+      "step": 177
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 97.625,
+      "epoch": 0.0022158044116914804,
+      "grad_norm": 0.7775987982749939,
+      "kl": 0.11865234375,
+      "learning_rate": 9.988920977941544e-06,
+      "loss": 0.0047,
+      "reward": 1.8203516602516174,
+      "reward_std": 0.22550363093614578,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.851601630449295,
+      "step": 178
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 84.375,
+      "epoch": 0.0022282527510830057,
+      "grad_norm": 0.5245223045349121,
+      "kl": 0.12744140625,
+      "learning_rate": 9.988858736244585e-06,
+      "loss": 0.0051,
+      "reward": 1.8911280632019043,
+      "reward_std": 0.02437590528279543,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8911280632019043,
+      "step": 179
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 83.4375,
+      "epoch": 0.0022407010904745305,
+      "grad_norm": 0.518882155418396,
+      "kl": 0.1708984375,
+      "learning_rate": 9.988796494547627e-06,
+      "loss": 0.0069,
+      "reward": 1.7126802206039429,
+      "reward_std": 0.05515829473733902,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7126802206039429,
+      "step": 180
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 87.21875,
+      "epoch": 0.002253149429866056,
+      "grad_norm": 0.5248196125030518,
+      "kl": 0.133544921875,
+      "learning_rate": 9.988734252850671e-06,
+      "loss": 0.0053,
+      "reward": 1.7711573839187622,
+      "reward_std": 0.11591519042849541,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.8024073839187622,
+      "step": 181
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 83.78125,
+      "epoch": 0.002265597769257581,
+      "grad_norm": 0.5835661292076111,
+      "kl": 0.11669921875,
+      "learning_rate": 9.988672011153713e-06,
+      "loss": 0.0047,
+      "reward": 1.8540168404579163,
+      "reward_std": 0.07147128880023956,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8540168106555939,
+      "step": 182
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 81.75,
+      "epoch": 0.0022780461086491064,
+      "grad_norm": 0.6301136016845703,
+      "kl": 0.17919921875,
+      "learning_rate": 9.988609769456755e-06,
+      "loss": 0.0072,
+      "reward": 1.529570460319519,
+      "reward_std": 0.40445713698863983,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.560820534825325,
+      "step": 183
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 83.5,
+      "epoch": 0.002290494448040631,
+      "grad_norm": 0.6207419037818909,
+      "kl": 0.14453125,
+      "learning_rate": 9.988547527759798e-06,
+      "loss": 0.0058,
+      "reward": 1.7111443877220154,
+      "reward_std": 0.23457887768745422,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7111444473266602,
+      "step": 184
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 86.1875,
+      "epoch": 0.0023029427874321565,
+      "grad_norm": 0.5986655950546265,
+      "kl": 0.14892578125,
+      "learning_rate": 9.98848528606284e-06,
+      "loss": 0.006,
+      "reward": 1.9187002182006836,
+      "reward_std": 0.06093704979866743,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.9187001585960388,
+      "step": 185
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 87.375,
+      "epoch": 0.0023153911268236817,
+      "grad_norm": 0.6483145952224731,
+      "kl": 0.139892578125,
+      "learning_rate": 9.988423044365882e-06,
+      "loss": 0.0056,
+      "reward": 1.8686339855194092,
+      "reward_std": 0.0916680209338665,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8686340153217316,
+      "step": 186
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 82.71875,
+      "epoch": 0.002327839466215207,
+      "grad_norm": 0.6316834688186646,
+      "kl": 0.142578125,
+      "learning_rate": 9.988360802668925e-06,
+      "loss": 0.0057,
+      "reward": 1.5443827509880066,
+      "reward_std": 0.23182823695242405,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.5443826913833618,
+      "step": 187
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 87.9375,
+      "epoch": 0.002340287805606732,
+      "grad_norm": 0.5704943537712097,
+      "kl": 0.1201171875,
+      "learning_rate": 9.988298560971967e-06,
+      "loss": 0.0048,
+      "reward": 1.7290632724761963,
+      "reward_std": 0.10309558361768723,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7290633618831635,
+      "step": 188
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 93.0,
+      "epoch": 0.002352736144998257,
+      "grad_norm": 0.5360406637191772,
+      "kl": 0.12646484375,
+      "learning_rate": 9.98823631927501e-06,
+      "loss": 0.0051,
+      "reward": 1.6119696497917175,
+      "reward_std": 0.27103206515312195,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.6432196497917175,
+      "step": 189
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 93.71875,
+      "epoch": 0.0023651844843897824,
+      "grad_norm": 0.5768948197364807,
+      "kl": 0.1376953125,
+      "learning_rate": 9.988174077578053e-06,
+      "loss": 0.0055,
+      "reward": 1.7519559860229492,
+      "reward_std": 0.19511063676327467,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.783206045627594,
+      "step": 190
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 87.46875,
+      "epoch": 0.0023776328237813077,
+      "grad_norm": 0.5476570129394531,
+      "kl": 0.1240234375,
+      "learning_rate": 9.988111835881095e-06,
+      "loss": 0.005,
+      "reward": 1.8128422498703003,
+      "reward_std": 0.15785422176122665,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8128422796726227,
+      "step": 191
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 84.8125,
+      "epoch": 0.002390081163172833,
+      "grad_norm": 0.6113950610160828,
+      "kl": 0.10986328125,
+      "learning_rate": 9.988049594184137e-06,
+      "loss": 0.0044,
+      "reward": 1.7782840132713318,
+      "reward_std": 0.10723626054823399,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7782839834690094,
+      "step": 192
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 103.40625,
+      "epoch": 0.002402529502564358,
+      "grad_norm": 0.5610492825508118,
+      "kl": 0.129150390625,
+      "learning_rate": 9.98798735248718e-06,
+      "loss": 0.0052,
+      "reward": 1.8901705145835876,
+      "reward_std": 0.04007100500166416,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.89017054438591,
+      "step": 193
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 101.625,
+      "epoch": 0.002414977841955883,
+      "grad_norm": 0.7355772256851196,
+      "kl": 0.113525390625,
+      "learning_rate": 9.98792511079022e-06,
+      "loss": 0.0046,
+      "reward": 1.762471079826355,
+      "reward_std": 0.02271357737481594,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.762471079826355,
+      "step": 194
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 90.8125,
+      "epoch": 0.0024274261813474083,
+      "grad_norm": 0.6883395314216614,
+      "kl": 0.09521484375,
+      "learning_rate": 9.987862869093264e-06,
+      "loss": 0.0038,
+      "reward": 1.8147326707839966,
+      "reward_std": 0.20457583293318748,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.8459826409816742,
+      "step": 195
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 92.875,
+      "epoch": 0.0024398745207389336,
+      "grad_norm": 0.604902446269989,
+      "kl": 0.15380859375,
+      "learning_rate": 9.987800627396306e-06,
+      "loss": 0.0061,
+      "reward": 1.2101359963417053,
+      "reward_std": 0.16694805398583412,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.21013597398996353,
+      "step": 196
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 100.5625,
+      "epoch": 0.0024523228601304584,
+      "grad_norm": 0.5739537477493286,
+      "kl": 0.1357421875,
+      "learning_rate": 9.987738385699348e-06,
+      "loss": 0.0054,
+      "reward": 1.738525927066803,
+      "reward_std": 0.10700106248259544,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.7385258674621582,
+      "step": 197
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 85.90625,
+      "epoch": 0.0024647711995219837,
+      "grad_norm": 0.5847993493080139,
+      "kl": 0.1318359375,
+      "learning_rate": 9.987676144002391e-06,
+      "loss": 0.0053,
+      "reward": 1.9518784284591675,
+      "reward_std": 0.01839788258075714,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.9518784284591675,
+      "step": 198
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 87.5,
+      "epoch": 0.002477219538913509,
+      "grad_norm": 0.6541993021965027,
+      "kl": 0.100341796875,
+      "learning_rate": 9.987613902305433e-06,
+      "loss": 0.004,
+      "reward": 1.848372757434845,
+      "reward_std": 0.1364253256469965,
+      "rewards/format_reward_rec": 1.0,
+      "rewards/iou_reward": 0.8483727872371674,
+      "step": 199
+    },
+    {
+      "clip_ratio": 0.0,
+      "completion_length": 87.0625,
+      "epoch": 0.0024896678783050343,
+      "grad_norm": 0.7986785173416138,
+      "kl": 0.12353515625,
+      "learning_rate": 9.987551660608475e-06,
+      "loss": 0.005,
+      "reward": 1.69829922914505,
+      "reward_std": 0.1897860188037157,
+      "rewards/format_reward_rec": 0.96875,
+      "rewards/iou_reward": 0.7295492589473724,
+      "step": 200
+    }
+  ],
+  "logging_steps": 1.0,
+  "max_steps": 160664,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:551863c0d96a4b9d1205af0398c8578c24624641a94db62faea5bfdbd4a427be
+size 8312

zero_to_fp32.py

@@ -0,0 +1,674 @@
+#!/usr/bin/env python
+
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: Apache-2.0
+
+# DeepSpeed Team
+
+# This script extracts fp32 consolidated weights from a zero 1, 2 and 3 DeepSpeed checkpoints. It gets
+# copied into the top level checkpoint dir, so the user can easily do the conversion at any point in
+# the future. Once extracted, the weights don't require DeepSpeed and can be used in any
+# application.
+#
+# example:
+#   python zero_to_fp32.py . output_dir/
+#   or
+#   python zero_to_fp32.py . output_dir/ --safe_serialization
+
+import argparse
+import torch
+import glob
+import math
+import os
+import re
+import json
+from tqdm import tqdm
+from collections import OrderedDict
+from dataclasses import dataclass
+
+# while this script doesn't use deepspeed to recover data, since the checkpoints are pickled with
+# DeepSpeed data structures it has to be available in the current python environment.
+from deepspeed.utils import logger
+from deepspeed.checkpoint.constants import (DS_VERSION, OPTIMIZER_STATE_DICT, SINGLE_PARTITION_OF_FP32_GROUPS,
+                                            FP32_FLAT_GROUPS, ZERO_STAGE, PARTITION_COUNT, PARAM_SHAPES, BUFFER_NAMES,
+                                            FROZEN_PARAM_SHAPES, FROZEN_PARAM_FRAGMENTS)
+
+
+@dataclass
+class zero_model_state:
+    buffers: dict()
+    param_shapes: dict()
+    shared_params: list
+    ds_version: int
+    frozen_param_shapes: dict()
+    frozen_param_fragments: dict()
+
+
+debug = 0
+
+# load to cpu
+device = torch.device('cpu')
+
+
+def atoi(text):
+    return int(text) if text.isdigit() else text
+
+
+def natural_keys(text):
+    '''
+    alist.sort(key=natural_keys) sorts in human order
+    http://nedbatchelder.com/blog/200712/human_sorting.html
+    (See Toothy's implementation in the comments)
+    '''
+    return [atoi(c) for c in re.split(r'(\d+)', text)]
+
+
+def get_model_state_file(checkpoint_dir, zero_stage):
+    if not os.path.isdir(checkpoint_dir):
+        raise FileNotFoundError(f"Directory '{checkpoint_dir}' doesn't exist")
+
+    # there should be only one file
+    if zero_stage <= 2:
+        file = os.path.join(checkpoint_dir, "mp_rank_00_model_states.pt")
+    elif zero_stage == 3:
+        file = os.path.join(checkpoint_dir, "zero_pp_rank_0_mp_rank_00_model_states.pt")
+
+    if not os.path.exists(file):
+        raise FileNotFoundError(f"can't find model states file at '{file}'")
+
+    return file
+
+
+def get_checkpoint_files(checkpoint_dir, glob_pattern):
+    # XXX: need to test that this simple glob rule works for multi-node setup too
+    ckpt_files = sorted(glob.glob(os.path.join(checkpoint_dir, glob_pattern)), key=natural_keys)
+
+    if len(ckpt_files) == 0:
+        raise FileNotFoundError(f"can't find {glob_pattern} files in directory '{checkpoint_dir}'")
+
+    return ckpt_files
+
+
+def get_optim_files(checkpoint_dir):
+    return get_checkpoint_files(checkpoint_dir, "*_optim_states.pt")
+
+
+def get_model_state_files(checkpoint_dir):
+    return get_checkpoint_files(checkpoint_dir, "*_model_states.pt")
+
+
+def parse_model_states(files):
+    zero_model_states = []
+    for file in files:
+        state_dict = torch.load(file, map_location=device)
+
+        if BUFFER_NAMES not in state_dict:
+            raise ValueError(f"{file} is not a model state checkpoint")
+        buffer_names = state_dict[BUFFER_NAMES]
+        if debug:
+            print("Found buffers:", buffer_names)
+
+        # recover just the buffers while restoring them to fp32 if they were saved in fp16
+        buffers = {k: v.float() for k, v in state_dict["module"].items() if k in buffer_names}
+        param_shapes = state_dict[PARAM_SHAPES]
+
+        # collect parameters that are included in param_shapes
+        param_names = []
+        for s in param_shapes:
+            for name in s.keys():
+                param_names.append(name)
+
+        # update with frozen parameters
+        frozen_param_shapes = state_dict.get(FROZEN_PARAM_SHAPES, None)
+        if frozen_param_shapes is not None:
+            if debug:
+                print(f"Found frozen_param_shapes: {frozen_param_shapes}")
+            param_names += list(frozen_param_shapes.keys())
+
+        # handle shared params
+        shared_params = [[k, v] for k, v in state_dict["shared_params"].items()]
+
+        ds_version = state_dict.get(DS_VERSION, None)
+
+        frozen_param_fragments = state_dict.get(FROZEN_PARAM_FRAGMENTS, None)
+
+        z_model_state = zero_model_state(buffers=buffers,
+                                         param_shapes=param_shapes,
+                                         shared_params=shared_params,
+                                         ds_version=ds_version,
+                                         frozen_param_shapes=frozen_param_shapes,
+                                         frozen_param_fragments=frozen_param_fragments)
+        zero_model_states.append(z_model_state)
+
+    return zero_model_states
+
+
+def parse_optim_states(files, ds_checkpoint_dir):
+    total_files = len(files)
+    state_dicts = []
+    for f in files:
+        state_dict = torch.load(f, map_location=device)
+        # immediately discard the potentially huge 2 optimizer states as we only care for fp32 master weights
+        # and also handle the case where it was already removed by another helper script
+        state_dict["optimizer_state_dict"].pop("optimizer_state_dict", None)
+        state_dicts.append(state_dict)
+
+    if not ZERO_STAGE in state_dicts[0][OPTIMIZER_STATE_DICT]:
+        raise ValueError(f"{files[0]} is not a zero checkpoint")
+    zero_stage = state_dicts[0][OPTIMIZER_STATE_DICT][ZERO_STAGE]
+    world_size = state_dicts[0][OPTIMIZER_STATE_DICT][PARTITION_COUNT]
+
+    # For ZeRO-2 each param group can have different partition_count as data parallelism for expert
+    # parameters can be different from data parallelism for non-expert parameters. So we can just
+    # use the max of the partition_count to get the dp world_size.
+
+    if type(world_size) is list:
+        world_size = max(world_size)
+
+    if world_size != total_files:
+        raise ValueError(
+            f"Expected {world_size} of '*_optim_states.pt' under '{ds_checkpoint_dir}' but found {total_files} files. "
+            "Possibly due to an overwrite of an old checkpoint, or a checkpoint didn't get saved by one or more processes."
+        )
+
+    # the groups are named differently in each stage
+    if zero_stage <= 2:
+        fp32_groups_key = SINGLE_PARTITION_OF_FP32_GROUPS
+    elif zero_stage == 3:
+        fp32_groups_key = FP32_FLAT_GROUPS
+    else:
+        raise ValueError(f"unknown zero stage {zero_stage}")
+
+    if zero_stage <= 2:
+        fp32_flat_groups = [state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key] for i in range(len(state_dicts))]
+    elif zero_stage == 3:
+        # if there is more than one param group, there will be multiple flattened tensors - one
+        # flattened tensor per group - for simplicity merge them into a single tensor
+        #
+        # XXX: could make the script more memory efficient for when there are multiple groups - it
+        # will require matching the sub-lists of param_shapes for each param group flattened tensor
+
+        fp32_flat_groups = [
+            torch.cat(state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key], 0) for i in range(len(state_dicts))
+        ]
+
+    return zero_stage, world_size, fp32_flat_groups
+
+
+def _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters):
+    """
+    Returns fp32 state_dict reconstructed from ds checkpoint
+
+    Args:
+        - ``ds_checkpoint_dir``: path to the deepspeed checkpoint folder (where the optimizer files are)
+
+    """
+    print(f"Processing zero checkpoint '{ds_checkpoint_dir}'")
+
+    optim_files = get_optim_files(ds_checkpoint_dir)
+    zero_stage, world_size, fp32_flat_groups = parse_optim_states(optim_files, ds_checkpoint_dir)
+    print(f"Detected checkpoint of type zero stage {zero_stage}, world_size: {world_size}")
+
+    model_files = get_model_state_files(ds_checkpoint_dir)
+
+    zero_model_states = parse_model_states(model_files)
+    print(f'Parsing checkpoint created by deepspeed=={zero_model_states[0].ds_version}')
+
+    if zero_stage <= 2:
+        return _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                                          exclude_frozen_parameters)
+    elif zero_stage == 3:
+        return _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                                          exclude_frozen_parameters)
+
+
+def _zero2_merge_frozen_params(state_dict, zero_model_states):
+    if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
+        return
+
+    frozen_param_shapes = zero_model_states[0].frozen_param_shapes
+    frozen_param_fragments = zero_model_states[0].frozen_param_fragments
+
+    if debug:
+        num_elem = sum(s.numel() for s in frozen_param_shapes.values())
+        print(f'rank 0: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
+
+        wanted_params = len(frozen_param_shapes)
+        wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
+        avail_numel = sum([p.numel() for p in frozen_param_fragments.values()])
+        print(f'Frozen params: Have {avail_numel} numels to process.')
+        print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
+
+    total_params = 0
+    total_numel = 0
+    for name, shape in frozen_param_shapes.items():
+        total_params += 1
+        unpartitioned_numel = shape.numel()
+        total_numel += unpartitioned_numel
+
+        state_dict[name] = frozen_param_fragments[name]
+
+        if debug:
+            print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
+
+    print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _has_callable(obj, fn):
+    attr = getattr(obj, fn, None)
+    return callable(attr)
+
+
+def _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
+    param_shapes = zero_model_states[0].param_shapes
+
+    # Reconstruction protocol:
+    #
+    # XXX: document this
+
+    if debug:
+        for i in range(world_size):
+            for j in range(len(fp32_flat_groups[0])):
+                print(f"{FP32_FLAT_GROUPS}[{i}][{j}].shape={fp32_flat_groups[i][j].shape}")
+
+    # XXX: memory usage doubles here (zero2)
+    num_param_groups = len(fp32_flat_groups[0])
+    merged_single_partition_of_fp32_groups = []
+    for i in range(num_param_groups):
+        merged_partitions = [sd[i] for sd in fp32_flat_groups]
+        full_single_fp32_vector = torch.cat(merged_partitions, 0)
+        merged_single_partition_of_fp32_groups.append(full_single_fp32_vector)
+    avail_numel = sum(
+        [full_single_fp32_vector.numel() for full_single_fp32_vector in merged_single_partition_of_fp32_groups])
+
+    if debug:
+        wanted_params = sum([len(shapes) for shapes in param_shapes])
+        wanted_numel = sum([sum(shape.numel() for shape in shapes.values()) for shapes in param_shapes])
+        # not asserting if there is a mismatch due to possible padding
+        print(f"Have {avail_numel} numels to process.")
+        print(f"Need {wanted_numel} numels in {wanted_params} params.")
+
+    # params
+    # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
+    # out-of-core computing solution
+    total_numel = 0
+    total_params = 0
+    for shapes, full_single_fp32_vector in zip(param_shapes, merged_single_partition_of_fp32_groups):
+        offset = 0
+        avail_numel = full_single_fp32_vector.numel()
+        for name, shape in shapes.items():
+
+            unpartitioned_numel = shape.numel() if _has_callable(shape, 'numel') else math.prod(shape)
+            total_numel += unpartitioned_numel
+            total_params += 1
+
+            if debug:
+                print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
+            state_dict[name] = full_single_fp32_vector.narrow(0, offset, unpartitioned_numel).view(shape)
+            offset += unpartitioned_numel
+
+        # Z2 started to align to 2*world_size to improve nccl performance. Therefore both offset and
+        # avail_numel can differ by anywhere between 0..2*world_size. Due to two unrelated complex
+        # paddings performed in the code it's almost impossible to predict the exact numbers w/o the
+        # live optimizer object, so we are checking that the numbers are within the right range
+        align_to = 2 * world_size
+
+        def zero2_align(x):
+            return align_to * math.ceil(x / align_to)
+
+        if debug:
+            print(f"original offset={offset}, avail_numel={avail_numel}")
+
+        offset = zero2_align(offset)
+        avail_numel = zero2_align(avail_numel)
+
+        if debug:
+            print(f"aligned  offset={offset}, avail_numel={avail_numel}")
+
+        # Sanity check
+        if offset != avail_numel:
+            raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
+
+    print(f"Reconstructed fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                               exclude_frozen_parameters):
+    state_dict = OrderedDict()
+
+    # buffers
+    buffers = zero_model_states[0].buffers
+    state_dict.update(buffers)
+    if debug:
+        print(f"added {len(buffers)} buffers")
+
+    if not exclude_frozen_parameters:
+        _zero2_merge_frozen_params(state_dict, zero_model_states)
+
+    _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
+
+    # recover shared parameters
+    for pair in zero_model_states[0].shared_params:
+        if pair[1] in state_dict:
+            state_dict[pair[0]] = state_dict[pair[1]]
+
+    return state_dict
+
+
+def zero3_partitioned_param_info(unpartitioned_numel, world_size):
+    remainder = unpartitioned_numel % world_size
+    padding_numel = (world_size - remainder) if remainder else 0
+    partitioned_numel = math.ceil(unpartitioned_numel / world_size)
+    return partitioned_numel, padding_numel
+
+
+def _zero3_merge_frozen_params(state_dict, world_size, zero_model_states):
+    if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
+        return
+
+    if debug:
+        for i in range(world_size):
+            num_elem = sum(s.numel() for s in zero_model_states[i].frozen_param_fragments.values())
+            print(f'rank {i}: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
+
+        frozen_param_shapes = zero_model_states[0].frozen_param_shapes
+        wanted_params = len(frozen_param_shapes)
+        wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
+        avail_numel = sum([p.numel() for p in zero_model_states[0].frozen_param_fragments.values()]) * world_size
+        print(f'Frozen params: Have {avail_numel} numels to process.')
+        print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
+
+    total_params = 0
+    total_numel = 0
+    for name, shape in zero_model_states[0].frozen_param_shapes.items():
+        total_params += 1
+        unpartitioned_numel = shape.numel()
+        total_numel += unpartitioned_numel
+
+        param_frags = tuple(model_state.frozen_param_fragments[name] for model_state in zero_model_states)
+        state_dict[name] = torch.cat(param_frags, 0).narrow(0, 0, unpartitioned_numel).view(shape)
+
+        partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
+
+        if debug:
+            print(
+                f"Frozen params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
+            )
+
+    print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
+    param_shapes = zero_model_states[0].param_shapes
+    avail_numel = fp32_flat_groups[0].numel() * world_size
+    # Reconstruction protocol: For zero3 we need to zip the partitions together at boundary of each
+    # param, re-consolidating each param, while dealing with padding if any
+
+    # merge list of dicts, preserving order
+    param_shapes = {k: v for d in param_shapes for k, v in d.items()}
+
+    if debug:
+        for i in range(world_size):
+            print(f"{FP32_FLAT_GROUPS}[{i}].shape={fp32_flat_groups[i].shape}")
+
+        wanted_params = len(param_shapes)
+        wanted_numel = sum(shape.numel() for shape in param_shapes.values())
+        # not asserting if there is a mismatch due to possible padding
+        avail_numel = fp32_flat_groups[0].numel() * world_size
+        print(f"Trainable params: Have {avail_numel} numels to process.")
+        print(f"Trainable params: Need {wanted_numel} numels in {wanted_params} params.")
+
+    # params
+    # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
+    # out-of-core computing solution
+    offset = 0
+    total_numel = 0
+    total_params = 0
+    for name, shape in tqdm(param_shapes.items(), desc='Gathering Sharded Weights'):
+        unpartitioned_numel = shape.numel()
+        total_numel += unpartitioned_numel
+        total_params += 1
+        partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
+
+        if debug:
+            print(
+                f"Trainable params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
+            )
+
+        # XXX: memory usage doubles here
+        state_dict[name] = torch.cat(
+            tuple(fp32_flat_groups[i].narrow(0, offset, partitioned_numel) for i in range(world_size)),
+            0).narrow(0, 0, unpartitioned_numel).view(shape)
+        offset += partitioned_numel
+
+    offset *= world_size
+
+    # Sanity check
+    if offset != avail_numel:
+        raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
+
+    print(f"Reconstructed Trainable fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                               exclude_frozen_parameters):
+    state_dict = OrderedDict()
+
+    # buffers
+    buffers = zero_model_states[0].buffers
+    state_dict.update(buffers)
+    if debug:
+        print(f"added {len(buffers)} buffers")
+
+    if not exclude_frozen_parameters:
+        _zero3_merge_frozen_params(state_dict, world_size, zero_model_states)
+
+    _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
+
+    # recover shared parameters
+    for pair in zero_model_states[0].shared_params:
+        if pair[1] in state_dict:
+            state_dict[pair[0]] = state_dict[pair[1]]
+
+    return state_dict
+
+
+def get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag=None, exclude_frozen_parameters=False):
+    """
+    Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated state_dict that can be loaded with
+    ``load_state_dict()`` and used for training without DeepSpeed or shared with others, for example
+    via a model hub.
+
+    Args:
+        - ``checkpoint_dir``: path to the desired checkpoint folder
+        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in 'latest' file. e.g., ``global_step14``
+        - ``exclude_frozen_parameters``: exclude frozen parameters
+
+    Returns:
+        - pytorch ``state_dict``
+
+    Note: this approach may not work if your application doesn't have sufficient free CPU memory and
+    you may need to use the offline approach using the ``zero_to_fp32.py`` script that is saved with
+    the checkpoint.
+
+    A typical usage might be ::
+
+        from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
+        # do the training and checkpoint saving
+        state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir) # already on cpu
+        model = model.cpu() # move to cpu
+        model.load_state_dict(state_dict)
+        # submit to model hub or save the model to share with others
+
+    In this example the ``model`` will no longer be usable in the deepspeed context of the same
+    application. i.e. you will need to re-initialize the deepspeed engine, since
+    ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
+
+    If you want it all done for you, use ``load_state_dict_from_zero_checkpoint`` instead.
+
+    """
+    if tag is None:
+        latest_path = os.path.join(checkpoint_dir, 'latest')
+        if os.path.isfile(latest_path):
+            with open(latest_path, 'r') as fd:
+                tag = fd.read().strip()
+        else:
+            raise ValueError(f"Unable to find 'latest' file at {latest_path}")
+
+    ds_checkpoint_dir = os.path.join(checkpoint_dir, tag)
+
+    if not os.path.isdir(ds_checkpoint_dir):
+        raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
+
+    return _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters)
+
+
+def convert_zero_checkpoint_to_fp32_state_dict(checkpoint_dir,
+                                               output_dir,
+                                               max_shard_size="5GB",
+                                               safe_serialization=False,
+                                               tag=None,
+                                               exclude_frozen_parameters=False):
+    """
+    Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be
+    loaded with ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed.
+
+    Args:
+        - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
+        - ``output_dir``: directory to the pytorch fp32 state_dict output files
+        - ``max_shard_size``: the maximum size for a checkpoint before being sharded, default value is 5GB
+        - ``safe_serialization``:  whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
+        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
+        - ``exclude_frozen_parameters``: exclude frozen parameters
+    """
+    # Dependency pre-check
+    if safe_serialization:
+        try:
+            from safetensors.torch import save_file
+        except ImportError:
+            print('If you want to use `safe_serialization`, please `pip install safetensors`')
+            raise
+    if max_shard_size is not None:
+        try:
+            from huggingface_hub import split_torch_state_dict_into_shards
+        except ImportError:
+            print('If you want to use `max_shard_size`, please `pip install huggingface_hub`')
+            raise
+
+    # Convert zero checkpoint to state_dict
+    state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag, exclude_frozen_parameters)
+
+    # Shard the model if it is too big.
+    weights_name = "model.safetensors" if safe_serialization else "pytorch_model.bin"
+    if max_shard_size is not None:
+        filename_pattern = weights_name.replace(".bin", "{suffix}.bin").replace(".safetensors", "{suffix}.safetensors")
+        state_dict_split = split_torch_state_dict_into_shards(state_dict,
+                                                              filename_pattern=filename_pattern,
+                                                              max_shard_size=max_shard_size)
+    else:
+        from collections import namedtuple
+        StateDictSplit = namedtuple("StateDictSplit", ["is_sharded", "filename_to_tensors"])
+        state_dict_split = StateDictSplit(is_sharded=False,
+                                          filename_to_tensors={weights_name: list(state_dict.keys())})
+
+    # Save the model
+    filename_to_tensors = state_dict_split.filename_to_tensors.items()
+    for shard_file, tensors in tqdm(filename_to_tensors, desc="Saving checkpoint shards"):
+        shard = {tensor: state_dict[tensor].contiguous() for tensor in tensors}
+        output_path = os.path.join(output_dir, shard_file)
+        if safe_serialization:
+            save_file(shard, output_path, metadata={"format": "pt"})
+        else:
+            torch.save(shard, output_path)
+
+    # Save index if sharded
+    if state_dict_split.is_sharded:
+        index = {
+            "metadata": state_dict_split.metadata,
+            "weight_map": state_dict_split.tensor_to_filename,
+        }
+        save_index_file = "model.safetensors.index.json" if safe_serialization else "pytorch_model.bin.index.json"
+        save_index_file = os.path.join(output_dir, save_index_file)
+        with open(save_index_file, "w", encoding="utf-8") as f:
+            content = json.dumps(index, indent=2, sort_keys=True) + "\n"
+            f.write(content)
+
+
+def load_state_dict_from_zero_checkpoint(model, checkpoint_dir, tag=None):
+    """
+    1. Put the provided model to cpu
+    2. Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict``
+    3. Load it into the provided model
+
+    Args:
+        - ``model``: the model object to update
+        - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
+        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
+
+    Returns:
+        - ``model`: modified model
+
+    Make sure you have plenty of CPU memory available before you call this function. If you don't
+    have enough use the ``zero_to_fp32.py`` utility to do the conversion. You will find it
+    conveniently placed for you in the checkpoint folder.
+
+    A typical usage might be ::
+
+        from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint
+        model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
+        # submit to model hub or save the model to share with others
+
+    Note, that once this was run, the ``model`` will no longer be usable in the deepspeed context
+    of the same application. i.e. you will need to re-initialize the deepspeed engine, since
+    ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
+
+    """
+    logger.info(f"Extracting fp32 weights")
+    state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
+
+    logger.info(f"Overwriting model with fp32 weights")
+    model = model.cpu()
+    model.load_state_dict(state_dict, strict=False)
+
+    return model
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("checkpoint_dir",
+                        type=str,
+                        help="path to the desired checkpoint folder, e.g., path/checkpoint-12")
+    parser.add_argument("output_dir",
+                        type=str,
+                        help="directory to the pytorch fp32 state_dict output files"
+                        "(e.g. path/checkpoint-12-output/)")
+    parser.add_argument(
+        "--max_shard_size",
+        type=str,
+        default="5GB",
+        help="The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size"
+        "lower than this size. If expressed as a string, needs to be digits followed by a unit (like `5MB`"
+        "We default it to 5GB in order for models to be able to run easily on free-tier google colab instances"
+        "without CPU OOM issues.")
+    parser.add_argument(
+        "--safe_serialization",
+        default=False,
+        action='store_true',
+        help="Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).")
+    parser.add_argument("-t",
+                        "--tag",
+                        type=str,
+                        default=None,
+                        help="checkpoint tag used as a unique identifier for checkpoint. e.g., global_step1")
+    parser.add_argument("--exclude_frozen_parameters", action='store_true', help="exclude frozen parameters")
+    parser.add_argument("-d", "--debug", action='store_true', help="enable debug")
+    args = parser.parse_args()
+
+    debug = args.debug
+
+    convert_zero_checkpoint_to_fp32_state_dict(args.checkpoint_dir,
+                                               args.output_dir,
+                                               max_shard_size=args.max_shard_size,
+                                               safe_serialization=args.safe_serialization,
+                                               tag=args.tag,
+                                               exclude_frozen_parameters=args.exclude_frozen_parameters)