allenai
/

Llama-3.1-Tulu-3-405B-DPO

Text Generation

text-generation-inference

Model card Files Files and versions

hamishivi commited on Jun 11

Commit

5825f5e

·

verified ·

1 Parent(s): cca45da

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -154,7 +154,7 @@ See the Falcon 180B model card for an example of this.
 DPO:
 - **Learning Rate**: 5 × 10⁻⁷ (8B), 2.0e-7 (70B, 405B)
 - **Learning Rate Schedule**: Linear
-- **Batch Size (effective)**: 32 (8B), 128 (70B), 256(405B)
 - **KL Penalty Coefficient**: 5
 - **Warm-up Ratio**: 0.1
 - **Max Sequence Length**: 2,048

 DPO:
 - **Learning Rate**: 5 × 10⁻⁷ (8B), 2.0e-7 (70B, 405B)
 - **Learning Rate Schedule**: Linear
+- **Batch Size (effective)**: 128 (8B), 128 (70B), 256(405B)
 - **KL Penalty Coefficient**: 5
 - **Warm-up Ratio**: 0.1
 - **Max Sequence Length**: 2,048