Commit
·
598588b
1
Parent(s):
8660a61
Update README.md
Browse files
README.md
CHANGED
|
@@ -3,6 +3,8 @@ license: apache-2.0
|
|
| 3 |
---
|
| 4 |
wandb: https://wandb.ai/open-assistant/supervised-finetuning/runs/pgftwpjx
|
| 5 |
|
|
|
|
|
|
|
| 6 |
datasets:
|
| 7 |
```
|
| 8 |
pretrain:
|
|
@@ -56,4 +58,6 @@ pythia-2.8b-pretrain:
|
|
| 56 |
per_device_eval_batch_size: 12
|
| 57 |
num_train_epochs: 2
|
| 58 |
save_total_limit: 2
|
| 59 |
-
```
|
|
|
|
|
|
|
|
|
| 3 |
---
|
| 4 |
wandb: https://wandb.ai/open-assistant/supervised-finetuning/runs/pgftwpjx
|
| 5 |
|
| 6 |
+
checkpoint: 11k steps
|
| 7 |
+
|
| 8 |
datasets:
|
| 9 |
```
|
| 10 |
pretrain:
|
|
|
|
| 58 |
per_device_eval_batch_size: 12
|
| 59 |
num_train_epochs: 2
|
| 60 |
save_total_limit: 2
|
| 61 |
+
```
|
| 62 |
+
|
| 63 |
+
command: `deepspeed trainer_sft.py --configs defaults pretrain pythia-2.8b-pretrain --cache_dir .cache/ --output_dir .saved_models/pythia-2.8b-pre --residual_dropout 0.0 --deepspeed`
|