Biobert_combo_v2

This model is a fine-tuned version of dmis-lab/biobert-base-cased-v1.1 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.1933
• Accuracy: 0.924
• Auc: 0.978
• Precision: 0.938
• Recall: 0.938
• F1: 0.938
• F1-macro: 0.919
• F1-micro: 0.924
• F1-weighted: 0.924

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 5
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Auc	Precision	Recall	F1	F1-macro	F1-micro	F1-weighted
0.4506	0.2661	500	0.2929	0.883	0.944	0.902	0.91	0.906	0.876	0.883	0.883
0.2846	0.5323	1000	0.2606	0.897	0.957	0.904	0.934	0.918	0.89	0.897	0.897
0.2462	0.7984	1500	0.2316	0.907	0.966	0.915	0.938	0.926	0.901	0.907	0.907
0.2337	1.0644	2000	0.2297	0.91	0.967	0.926	0.928	0.927	0.904	0.91	0.91
0.21	1.3305	2500	0.2212	0.911	0.97	0.934	0.922	0.928	0.906	0.911	0.911
0.2033	1.5967	3000	0.2181	0.913	0.972	0.925	0.935	0.93	0.908	0.913	0.913
0.2029	1.8628	3500	0.2109	0.916	0.974	0.92	0.946	0.933	0.91	0.916	0.915
0.1948	2.1288	4000	0.2030	0.921	0.975	0.94	0.931	0.935	0.916	0.921	0.921
0.1812	2.3949	4500	0.2093	0.918	0.974	0.933	0.935	0.934	0.913	0.918	0.918
0.1822	2.6611	5000	0.1996	0.92	0.976	0.933	0.939	0.936	0.916	0.92	0.92
0.1818	2.9272	5500	0.1945	0.923	0.977	0.936	0.94	0.938	0.918	0.923	0.923
0.1707	3.1932	6000	0.1955	0.923	0.977	0.946	0.929	0.937	0.919	0.923	0.923
0.1635	3.4593	6500	0.2019	0.922	0.977	0.935	0.939	0.937	0.917	0.922	0.922
0.1747	3.7255	7000	0.1983	0.923	0.977	0.931	0.945	0.938	0.918	0.923	0.923
0.1735	3.9916	7500	0.1956	0.923	0.978	0.936	0.941	0.938	0.919	0.923	0.923
0.1646	4.2576	8000	0.1994	0.921	0.977	0.933	0.94	0.937	0.916	0.921	0.921
0.1616	4.5238	8500	0.1925	0.924	0.978	0.942	0.934	0.938	0.919	0.924	0.924
0.1615	4.7899	9000	0.1933	0.924	0.978	0.938	0.938	0.938	0.919	0.924	0.924

Framework versions

• Transformers 4.53.2
• Pytorch 2.6.0+cu124
• Datasets 2.14.4
• Tokenizers 0.21.2