NeoBERT-multiclass-classifier-ICLR

This model is a fine-tuned version of chandar-lab/NeoBERT on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.7258
• F1: 0.5134

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 20
• label_smoothing_factor: 0.1

Training results

Training Loss	Epoch	Step	Validation Loss	F1
No log	1.0	45	1.2609	0.4641
No log	2.0	90	1.3167	0.4641
1.2441	3.0	135	1.1799	0.5405
1.2441	4.0	180	1.2810	0.5386
1.0526	5.0	225	1.2742	0.5098
1.0526	6.0	270	1.4929	0.5030
0.7789	7.0	315	1.5076	0.5425
0.7789	8.0	360	1.6513	0.4908
0.5299	9.0	405	1.6172	0.5476
0.5299	10.0	450	1.7358	0.5389
0.5299	11.0	495	1.8935	0.4847
0.4185	12.0	540	1.8012	0.5152
0.4185	13.0	585	1.7241	0.5337
0.3614	14.0	630	1.7109	0.5257
0.3614	15.0	675	1.7233	0.5024
0.3527	16.0	720	1.7104	0.5147
0.3527	17.0	765	1.7282	0.5134
0.3513	18.0	810	1.7257	0.5134
0.3513	19.0	855	1.7263	0.5134
0.3511	20.0	900	1.7258	0.5134

Framework versions

• Transformers 4.53.0
• Pytorch 2.7.1+cu126
• Datasets 3.6.0
• Tokenizers 0.21.2