train_cola_1757340184

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the cola dataset. It achieves the following results on the evaluation set:

• Loss: 0.9521
• Num Input Tokens Seen: 6929680

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 123
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.1268	1.0	3848	0.2820	346872
0.3132	2.0	7696	0.2417	693752
0.2179	3.0	11544	0.2405	1040128
0.2649	4.0	15392	0.2411	1386696
0.2187	5.0	19240	0.2434	1733072
0.1872	6.0	23088	0.2394	2079640
0.2849	7.0	26936	0.2419	2425920
0.1858	8.0	30784	0.2366	2772144
0.2726	9.0	34632	0.2393	3118472
0.2241	10.0	38480	0.2438	3465288
0.2284	11.0	42328	0.2862	3811696
0.0849	12.0	46176	0.2743	4158168
0.1104	13.0	50024	0.3264	4504416
0.1854	14.0	53872	0.3800	4850888
0.1511	15.0	57720	0.4422	5197456
0.0483	16.0	61568	0.5154	5543848
0.1082	17.0	65416	0.6811	5890320
0.2789	18.0	69264	0.7981	6237200
0.3151	19.0	73112	0.9202	6583408
0.0006	20.0	76960	0.9521	6929680

Framework versions

• PEFT 0.15.2
• Transformers 4.51.3
• Pytorch 2.8.0+cu128
• Datasets 3.6.0
• Tokenizers 0.21.1