Biobert_combo_v1

This model is a fine-tuned version of dmis-lab/biobert-base-cased-v1.1 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.4421
• Accuracy: 0.769
• Auc: 0.867
• Precision: 0.745
• Recall: 0.887
• F1: 0.81
• F1-macro: 0.757
• F1-micro: 0.769
• F1-weighted: 0.763

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 5
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Auc	Precision	Recall	F1	F1-macro	F1-micro	F1-weighted
0.6198	0.1028	500	0.5619	0.693	0.769	0.676	0.858	0.757	0.671	0.693	0.681
0.5401	0.2057	1000	0.5167	0.717	0.807	0.717	0.81	0.761	0.708	0.717	0.713
0.5141	0.3085	1500	0.4933	0.731	0.827	0.75	0.772	0.761	0.726	0.731	0.73
0.4886	0.4114	2000	0.4942	0.736	0.827	0.727	0.841	0.78	0.726	0.736	0.732
0.4882	0.5142	2500	0.4814	0.743	0.838	0.712	0.901	0.795	0.724	0.743	0.732
0.4734	0.6170	3000	0.4718	0.746	0.843	0.724	0.878	0.794	0.733	0.746	0.739
0.4659	0.7199	3500	0.4767	0.748	0.843	0.733	0.859	0.791	0.737	0.748	0.743
0.4632	0.8227	4000	0.4617	0.755	0.852	0.724	0.901	0.803	0.739	0.755	0.746
0.4602	0.9255	4500	0.4639	0.752	0.852	0.717	0.915	0.804	0.734	0.752	0.742
0.457	1.0284	5000	0.4613	0.752	0.85	0.729	0.881	0.798	0.739	0.752	0.745
0.4468	1.1312	5500	0.4541	0.757	0.855	0.731	0.891	0.803	0.743	0.757	0.75
0.4421	1.2341	6000	0.4591	0.755	0.853	0.727	0.892	0.801	0.74	0.755	0.747
0.4373	1.3369	6500	0.4537	0.759	0.856	0.739	0.874	0.801	0.748	0.759	0.753
0.4402	1.4397	7000	0.4552	0.755	0.855	0.74	0.863	0.797	0.745	0.755	0.751
0.4296	1.5426	7500	0.4545	0.76	0.857	0.742	0.87	0.801	0.749	0.76	0.755
0.4407	1.6454	8000	0.4458	0.762	0.86	0.742	0.877	0.804	0.751	0.762	0.757
0.4225	1.7483	8500	0.4472	0.761	0.86	0.735	0.889	0.805	0.748	0.761	0.754
0.4327	1.8511	9000	0.4485	0.758	0.858	0.741	0.867	0.799	0.747	0.758	0.753
0.4311	1.9539	9500	0.4479	0.76	0.859	0.742	0.869	0.801	0.749	0.76	0.755
0.4288	2.0568	10000	0.4527	0.761	0.859	0.742	0.873	0.802	0.75	0.761	0.756
0.4124	2.1596	10500	0.4477	0.762	0.861	0.736	0.891	0.806	0.749	0.762	0.756
0.4181	2.2624	11000	0.4569	0.759	0.857	0.741	0.87	0.8	0.748	0.759	0.754
0.4178	2.3653	11500	0.4469	0.762	0.861	0.741	0.879	0.804	0.751	0.762	0.757
0.4127	2.4681	12000	0.4448	0.764	0.863	0.742	0.881	0.806	0.753	0.764	0.759
0.419	2.5710	12500	0.4454	0.764	0.864	0.734	0.902	0.809	0.75	0.764	0.757
0.4232	2.6738	13000	0.4394	0.766	0.864	0.747	0.873	0.805	0.756	0.766	0.761
0.4226	2.7766	13500	0.4404	0.766	0.864	0.747	0.873	0.805	0.756	0.766	0.761
0.4196	2.8795	14000	0.4477	0.765	0.862	0.758	0.846	0.8	0.757	0.765	0.762
0.408	2.9823	14500	0.4497	0.763	0.862	0.745	0.871	0.803	0.752	0.763	0.758
0.4054	3.0852	15000	0.4404	0.765	0.865	0.749	0.865	0.803	0.755	0.765	0.76
0.4155	3.1880	15500	0.4466	0.764	0.863	0.74	0.885	0.806	0.752	0.764	0.758
0.4155	3.2908	16000	0.4417	0.765	0.864	0.744	0.879	0.806	0.754	0.765	0.76
0.4104	3.3937	16500	0.4407	0.766	0.866	0.742	0.887	0.808	0.755	0.766	0.761
0.4081	3.4965	17000	0.4406	0.765	0.866	0.753	0.859	0.802	0.756	0.765	0.762
0.4046	3.5993	17500	0.4384	0.768	0.868	0.742	0.891	0.81	0.756	0.768	0.762
0.4065	3.7022	18000	0.4443	0.766	0.866	0.742	0.888	0.808	0.754	0.766	0.76
0.4028	3.8050	18500	0.4438	0.768	0.866	0.746	0.882	0.808	0.757	0.768	0.763
0.4035	3.9079	19000	0.4453	0.766	0.865	0.753	0.862	0.804	0.758	0.766	0.763
0.4003	4.0107	19500	0.4426	0.767	0.865	0.75	0.87	0.806	0.757	0.767	0.763
0.4011	4.1135	20000	0.4423	0.767	0.867	0.741	0.892	0.81	0.755	0.767	0.761
0.3924	4.2164	20500	0.4394	0.768	0.867	0.75	0.874	0.807	0.758	0.768	0.764
0.4043	4.3192	21000	0.4421	0.768	0.867	0.744	0.887	0.809	0.757	0.768	0.762
0.395	4.4220	21500	0.4450	0.767	0.865	0.75	0.87	0.805	0.757	0.767	0.763
0.4072	4.5249	22000	0.4392	0.769	0.867	0.751	0.874	0.808	0.759	0.769	0.765
0.4055	4.6277	22500	0.4439	0.768	0.866	0.745	0.885	0.809	0.757	0.768	0.762
0.3987	4.7306	23000	0.4435	0.768	0.866	0.747	0.881	0.808	0.758	0.768	0.763
0.396	4.8334	23500	0.4430	0.768	0.867	0.745	0.885	0.809	0.757	0.768	0.763
0.4017	4.9362	24000	0.4421	0.769	0.867	0.745	0.887	0.81	0.757	0.769	0.763

Framework versions

• Transformers 4.53.1
• Pytorch 2.6.0+cu124
• Datasets 2.14.4
• Tokenizers 0.21.2