Llama3-OpenBioLLM-8B-PsyCourse-fold3

This model is a fine-tuned version of aaditya/Llama3-OpenBioLLM-8B on the course-train-fold3 dataset. It achieves the following results on the evaluation set:

• Loss: 0.0350

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 5.0

Training results

Training Loss	Epoch	Step	Validation Loss
0.4566	0.0753	50	0.3151
0.1039	0.1505	100	0.0934
0.0879	0.2258	150	0.0729
0.0684	0.3011	200	0.0644
0.0696	0.3763	250	0.0604
0.064	0.4516	300	0.0538
0.048	0.5269	350	0.0553
0.0568	0.6021	400	0.0504
0.0548	0.6774	450	0.0462
0.0387	0.7527	500	0.0457
0.0454	0.8279	550	0.0439
0.0343	0.9032	600	0.0421
0.0363	0.9785	650	0.0402
0.0272	1.0537	700	0.0415
0.0324	1.1290	750	0.0384
0.0394	1.2043	800	0.0429
0.0297	1.2795	850	0.0411
0.0423	1.3548	900	0.0396
0.0324	1.4300	950	0.0372
0.0311	1.5053	1000	0.0395
0.0309	1.5806	1050	0.0407
0.0233	1.6558	1100	0.0377
0.0455	1.7311	1150	0.0354
0.0329	1.8064	1200	0.0364
0.0352	1.8816	1250	0.0351
0.029	1.9569	1300	0.0350
0.0187	2.0322	1350	0.0363
0.025	2.1074	1400	0.0380
0.0209	2.1827	1450	0.0377
0.0224	2.2580	1500	0.0407
0.0261	2.3332	1550	0.0393
0.0156	2.4085	1600	0.0387
0.0173	2.4838	1650	0.0388
0.0228	2.5590	1700	0.0367
0.0328	2.6343	1750	0.0371
0.0264	2.7096	1800	0.0393
0.0235	2.7848	1850	0.0364
0.0204	2.8601	1900	0.0386
0.0207	2.9354	1950	0.0372
0.01	3.0106	2000	0.0390
0.0103	3.0859	2050	0.0431
0.0096	3.1612	2100	0.0418
0.012	3.2364	2150	0.0431
0.0098	3.3117	2200	0.0443
0.0104	3.3870	2250	0.0466
0.0061	3.4622	2300	0.0481
0.0051	3.5375	2350	0.0463
0.0085	3.6128	2400	0.0458
0.0093	3.6880	2450	0.0467
0.0103	3.7633	2500	0.0477
0.0066	3.8386	2550	0.0487
0.0101	3.9138	2600	0.0477
0.0104	3.9891	2650	0.0455
0.0073	4.0644	2700	0.0460
0.003	4.1396	2750	0.0478
0.0025	4.2149	2800	0.0498
0.0051	4.2901	2850	0.0506
0.0078	4.3654	2900	0.0516
0.0019	4.4407	2950	0.0518
0.0029	4.5159	3000	0.0521
0.0048	4.5912	3050	0.0521
0.0033	4.6665	3100	0.0525
0.002	4.7417	3150	0.0525
0.0017	4.8170	3200	0.0528
0.003	4.8923	3250	0.0529
0.0021	4.9675	3300	0.0526

Framework versions

• PEFT 0.12.0
• Transformers 4.46.1
• Pytorch 2.5.1+cu124
• Datasets 3.1.0
• Tokenizers 0.20.3