train_codealpacapy_42_1760638811

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the codealpacapy dataset. It achieves the following results on the evaluation set:

• Loss: 0.4541
• Num Input Tokens Seen: 24887720

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.03
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.4965	1.0	1908	0.4692	1243048
0.3516	2.0	3816	0.4636	2489456
0.4189	3.0	5724	0.4576	3733736
0.5105	4.0	7632	0.4567	4976128
0.5727	5.0	9540	0.4546	6219592
0.5876	6.0	11448	0.4546	7467968
0.4743	7.0	13356	0.4541	8709168
0.4622	8.0	15264	0.4543	9958360
0.4449	9.0	17172	0.4557	11204000
0.4562	10.0	19080	0.4582	12446408
0.3546	11.0	20988	0.4627	13691904
0.3935	12.0	22896	0.4697	14937216
0.5209	13.0	24804	0.4720	16179624
0.3421	14.0	26712	0.4755	17425368
0.2642	15.0	28620	0.4898	18668536
0.3709	16.0	30528	0.4946	19916008
0.1985	17.0	32436	0.5012	21158120
0.3645	18.0	34344	0.5048	22400368
0.3265	19.0	36252	0.5051	23645440
0.2443	20.0	38160	0.5052	24887720

Framework versions

• PEFT 0.15.2
• Transformers 4.51.3
• Pytorch 2.8.0+cu128
• Datasets 3.6.0
• Tokenizers 0.21.1