instruction_tuned_model

This model is a fine-tuned version of NousResearch/Llama-2-7b-chat-hf on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.7426

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 64
• eval_batch_size: 32
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 2
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.9054	0.1571	100	0.9336
0.7821	0.3142	200	0.8612
0.7367	0.4713	300	0.8301
0.7238	0.6284	400	0.8054
0.6822	0.7855	500	0.7912
0.6511	0.9427	600	0.7823
0.6166	1.0998	700	0.7764
0.5797	1.2569	800	0.7649
0.5902	1.4140	900	0.7541
0.5916	1.5711	1000	0.7562
0.5816	1.7282	1100	0.7468
0.5662	1.8853	1200	0.7448
0.4927	2.0424	1300	0.7501
0.4796	2.1995	1400	0.7540
0.4683	2.3566	1500	0.7472
0.4854	2.5137	1600	0.7453
0.4733	2.6709	1700	0.7455
0.4643	2.8280	1800	0.7431
0.4535	2.9851	1900	0.7426

Framework versions

• PEFT 0.13.2.dev0
• Transformers 4.45.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.1
• Tokenizers 0.20.0