llama 3 self-align experiments
Collection
Replicating the pipeline for StarCoder-2 Instruct on Llama-3-8B with some tweaks https://huggingface.co/blog/sc2-instruct
•
4 items
•
Updated
•
6
Weights from the Llama-3-8B Self-Align Experiments
[WEIGHTS TO BE UPLOADED ONCE DONE]
Training Config
The config.yaml should be used during accelerate launch, and run.sh was used to launch the training using the StarCoder2 Self-Align training script.
Some tweaks were performed to get this working on 48GB vRAM:
- FSDP was used
per_device_batch_sizeis2- A learning rate of 3e-6 was used
Environment:
- Trained with 2x4090 GPUs
- 128GB RAM
- Downloads last month
- 8