e98fb5bf-8048-49ba-9174-770aa41afffc

This model is a fine-tuned version of NousResearch/Yarn-Llama-2-13b-128k on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.0008

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.000204
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 50
• training_steps: 463

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0022	1	0.1187
0.0183	0.1080	50	0.0119
0.016	0.2160	100	0.0282
0.0108	0.3240	150	0.0076
0.0051	0.4320	200	0.0169
0.0031	0.5400	250	0.0031
0.0003	0.6479	300	0.0020
0.0029	0.7559	350	0.0015
0.0023	0.8639	400	0.0009
0.0003	0.9719	450	0.0008

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1