Update README.md
Browse files
README.md
CHANGED
|
@@ -60,10 +60,10 @@ We build the Doge by doing Per-Training on [Smollm-Corpus](https://huggingface.c
|
|
| 60 |
|
| 61 |
| Model | Training Data | Steps | Content Length | Tokens | LR | Batch Size | Precision | RTX 4090 GPU hours |
|
| 62 |
|---|---|---|---|---|---|---|---|---|
|
| 63 |
-
| [Doge-20M](https://huggingface.co/SmallDoge/Doge-20M) | [
|
| 64 |
-
| [Doge-60M](https://huggingface.co/SmallDoge/Doge-60M) | [
|
| 65 |
-
| [Doge-160M](https://huggingface.co/SmallDoge/Doge-160M) | [
|
| 66 |
-
| [Doge-320M](https://huggingface.co/SmallDoge/Doge-320M) | [
|
| 67 |
|
| 68 |
**Evaluation**:
|
| 69 |
|
|
|
|
| 60 |
|
| 61 |
| Model | Training Data | Steps | Content Length | Tokens | LR | Batch Size | Precision | RTX 4090 GPU hours |
|
| 62 |
|---|---|---|---|---|---|---|---|---|
|
| 63 |
+
| [Doge-20M](https://huggingface.co/SmallDoge/Doge-20M) | [smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | 8k | 2048 | 4B | 8e-3 | 0.5M | bfloat16 | 14 |
|
| 64 |
+
| [Doge-60M](https://huggingface.co/SmallDoge/Doge-60M) | [smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | 16k | 2048 | 16B | 6e-3 | 1M | bfloat16 | 128 |
|
| 65 |
+
| [Doge-160M](https://huggingface.co/SmallDoge/Doge-160M) | [smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | 24k | 2048 | 32B | 4e-3 | 1.5M | bfloat16 | 522 |
|
| 66 |
+
| [Doge-320M](https://huggingface.co/SmallDoge/Doge-320M) | [smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | 32k | 2048 | 64B | 2e-3 | 2M | bfloat16 | 1856 |
|
| 67 |
|
| 68 |
**Evaluation**:
|
| 69 |
|