HGU_rulebook-Llama3.2-Bllossom-5B_fine-tuning-QLoRA-16_64_3

This model is a fine-tuned version of Bllossom/llama-3.2-Korean-Bllossom-AICA-5B on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 5.6786

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: Use adamw_bnb_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_ratio: 0.1
• training_steps: 942
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
7.7538	0.2991	47	6.4144
5.7228	0.5982	94	5.7066
5.6895	0.8974	141	5.6889
5.6821	1.1965	188	5.6843
5.6823	1.4956	235	5.6830
5.683	1.7947	282	5.6813
5.6833	2.0939	329	5.6805
5.678	2.3930	376	5.6806
5.6776	2.6921	423	5.6796
5.6752	2.9912	470	5.6791
5.6757	3.2904	517	5.6791
5.6722	3.5895	564	5.6788
5.6735	3.8886	611	5.6786
5.6683	4.1877	658	5.6788
5.6744	4.4869	705	5.6786
5.6738	4.7860	752	5.6785
5.6734	5.0851	799	5.6785
5.6709	5.3842	846	5.6786
5.6698	5.6834	893	5.6786
5.6734	5.9825	940	5.6786

Framework versions

• PEFT 0.12.0
• Transformers 4.46.2
• Pytorch 2.0.1+cu118
• Datasets 3.0.0
• Tokenizers 0.20.1