AmedeoBonatti/nlp_te_mlm_scibert_tok

mlm_scibert_uncased

This model is a fine-tuned version of allenai/scibert_scivocab_uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.5247

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 16
• eval_batch_size: 8
• seed: 1234
• gradient_accumulation_steps: 16
• total_train_batch_size: 256
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 500
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
5.2151	0.9978	142	3.7157
3.3786	1.9956	284	2.8905
2.8013	2.9934	426	2.5101
2.4892	3.9982	569	2.3025
2.3273	4.9960	711	2.1637
2.2001	5.9939	853	2.0614
2.0949	6.9987	996	1.9968
2.0347	7.9965	1138	1.9339
1.9763	8.9943	1280	1.8893
1.9134	9.9991	1423	1.8491
1.8818	10.9969	1565	1.8097
1.8432	11.9947	1707	1.7851
1.801	12.9996	1850	1.7577
1.7853	13.9974	1992	1.7347
1.7577	14.9952	2134	1.7108
1.7235	16.0	2277	1.7003
1.7098	16.9978	2419	1.6830
1.6891	17.9956	2561	1.6636
1.674	18.9934	2703	1.6488
1.6446	19.9982	2846	1.6392
1.6407	20.9960	2988	1.6268
1.6264	21.9939	3130	1.6132
1.5975	22.9987	3273	1.6025
1.5965	23.9965	3415	1.5991
1.5835	24.9943	3557	1.5914
1.5597	25.9991	3700	1.5725
1.5588	26.9969	3842	1.5693
1.5478	27.9947	3984	1.5709
1.5297	28.9996	4127	1.5611
1.527	29.9974	4269	1.5506
1.5164	30.9952	4411	1.5445
1.4967	32.0	4554	1.5398
1.4982	32.9978	4696	1.5291
1.4906	33.9956	4838	1.5248
1.4827	34.9934	4980	1.5287
1.4645	35.9982	5123	1.5223
1.4657	36.9960	5265	1.5108
1.4538	37.9939	5407	1.5092
1.4365	38.9987	5550	1.5037
1.4419	39.9965	5692	1.4937
1.4327	40.9943	5834	1.4969
1.4153	41.9991	5977	1.4938
1.4168	42.9969	6119	1.4842
1.4126	43.9947	6261	1.4825
1.397	44.9996	6404	1.4800
1.3989	45.9974	6546	1.4805
1.3953	46.9952	6688	1.4748
1.3802	48.0	6831	1.4748
1.3824	48.9978	6973	1.4713
1.3776	49.9956	7115	1.4687
1.3704	50.9934	7257	1.4626
1.3575	51.9982	7400	1.4567
1.3642	52.9960	7542	1.4599
1.3574	53.9939	7684	1.4542
1.3422	54.9987	7827	1.4477
1.3472	55.9965	7969	1.4519
1.3426	56.9943	8111	1.4514
1.3289	57.9991	8254	1.4502
1.3354	58.9969	8396	1.4431
1.3294	59.9947	8538	1.4432
1.3174	60.9996	8681	1.4420
1.3226	61.9974	8823	1.4443
1.3145	62.9952	8965	1.4468
1.3022	64.0	9108	1.4405
1.3071	64.9978	9250	1.4365
1.3021	65.9956	9392	1.4416
1.2996	66.9934	9534	1.4353
1.2854	67.9982	9677	1.4335
1.2891	68.9960	9819	1.4322
1.2892	69.9939	9961	1.4323
1.2731	70.9987	10104	1.4340
1.2806	71.9965	10246	1.4206
1.2771	72.9943	10388	1.4264
1.2628	73.9991	10531	1.4265
1.2682	74.9969	10673	1.4266
1.2634	75.9947	10815	1.4340
1.2529	76.9996	10958	1.4248
1.2594	77.9974	11100	1.4292
1.2532	78.9952	11242	1.4276
1.2428	80.0	11385	1.4246
1.247	80.9978	11527	1.4192
1.2459	81.9956	11669	1.4230
1.2418	82.9934	11811	1.4211
1.2308	83.9982	11954	1.4184
1.2351	84.9960	12096	1.4154
1.2337	85.9939	12238	1.4177
1.2198	86.9987	12381	1.4171
1.2255	87.9965	12523	1.4136
1.2246	88.9943	12665	1.4203
1.2127	89.9991	12808	1.4192
1.2179	90.9969	12950	1.4201
1.213	91.9947	13092	1.4150
1.2023	92.9996	13235	1.4101
1.2107	93.9974	13377	1.4159
1.2056	94.9952	13519	1.4158
1.1942	96.0	13662	1.4145
1.2009	96.9978	13804	1.4134
1.1972	97.9956	13946	1.4118
1.1959	98.9934	14088	1.4151
1.1845	99.9982	14231	1.4124
1.1889	100.9960	14373	1.4092
1.1887	101.9939	14515	1.4157
1.1753	102.9987	14658	1.4129
1.1809	103.9965	14800	1.4196
1.1782	104.9943	14942	1.4104
1.1691	105.9991	15085	1.4115
1.1745	106.9969	15227	1.4093
1.1729	107.9947	15369	1.4077
1.1622	108.9996	15512	1.4136
1.1674	109.9974	15654	1.4075
1.1639	110.9952	15796	1.4104
1.1565	112.0	15939	1.4118
1.1587	112.9978	16081	1.4082
1.1559	113.9956	16223	1.4142
1.1543	114.9934	16365	1.4118
1.1427	115.9982	16508	1.4099
1.1509	116.9960	16650	1.4137
1.1464	117.9939	16792	1.4132
1.1347	118.9987	16935	1.4082
1.1431	119.9965	17077	1.4117
1.1401	120.9943	17219	1.4113
1.1293	121.9991	17362	1.4149
1.1352	122.9969	17504	1.4123
1.1347	123.9947	17646	1.4105
1.1231	124.9996	17789	1.4109
1.1303	125.9974	17931	1.4082
1.1259	126.9952	18073	1.4126
1.1165	128.0	18216	1.4116
1.1236	128.9978	18358	1.4132
1.1202	129.9956	18500	1.4107
1.1192	130.9934	18642	1.4207
1.1108	131.9982	18785	1.4101
1.1137	132.9960	18927	1.4062
1.1125	133.9939	19069	1.4125
1.1028	134.9987	19212	1.4175
1.1098	135.9965	19354	1.4087
1.1076	136.9943	19496	1.4116
1.0984	137.9991	19639	1.4086
1.1005	138.9969	19781	1.4185
1.1013	139.9947	19923	1.4152
1.0919	140.9996	20066	1.4164
1.0949	141.9974	20208	1.4158
1.0968	142.9952	20350	1.4111
1.0857	144.0	20493	1.4114
1.0892	144.9978	20635	1.4157
1.0915	145.9956	20777	1.4197
1.088	146.9934	20919	1.4167
1.0764	147.9982	21062	1.4158
1.0836	148.9960	21204	1.4140
1.0825	149.9939	21346	1.4151
1.0711	150.9987	21489	1.4150
1.0776	151.9965	21631	1.4181
1.0754	152.9943	21773	1.4171
1.0662	153.9991	21916	1.4205
1.0721	154.9969	22058	1.4187
1.0697	155.9947	22200	1.4186
1.0608	156.9996	22343	1.4244
1.0672	157.9974	22485	1.4190
1.0648	158.9952	22627	1.4218
1.0558	160.0	22770	1.4086
1.062	160.9978	22912	1.4184
1.0607	161.9956	23054	1.4171
1.0589	162.9934	23196	1.4271
1.0483	163.9982	23339	1.4142
1.0544	164.9960	23481	1.4198
1.0528	165.9939	23623	1.4182
1.045	166.9987	23766	1.4158
1.0487	167.9965	23908	1.4233
1.0488	168.9943	24050	1.4218
1.0398	169.9991	24193	1.4262
1.045	170.9969	24335	1.4246
1.0416	171.9947	24477	1.4237
1.0352	172.9996	24620	1.4262
1.0392	173.9974	24762	1.4250
1.038	174.9952	24904	1.4237
1.031	176.0	25047	1.4253
1.0357	176.9978	25189	1.4204
1.0342	177.9956	25331	1.4309
1.0321	178.9934	25473	1.4307
1.023	179.9982	25616	1.4303
1.0302	180.9960	25758	1.4310
1.0272	181.9939	25900	1.4289
1.0188	182.9987	26043	1.4257
1.0248	183.9965	26185	1.4298
1.0233	184.9943	26327	1.4326
1.0162	185.9991	26470	1.4299
1.0202	186.9969	26612	1.4259
1.0192	187.9947	26754	1.4268
1.009	188.9996	26897	1.4341
1.0155	189.9974	27039	1.4270
1.0126	190.9952	27181	1.4242
1.0044	192.0	27324	1.4302
1.012	192.9978	27466	1.4267
1.0092	193.9956	27608	1.4296
1.0078	194.9934	27750	1.4312
1.0008	195.9982	27893	1.4307
1.005	196.9960	28035	1.4326
1.0032	197.9939	28177	1.4397
0.9952	198.9987	28320	1.4373
1.0021	199.9965	28462	1.4368
1.0014	200.9943	28604	1.4289
0.9909	201.9991	28747	1.4328
0.9963	202.9969	28889	1.4315
0.997	203.9947	29031	1.4329
0.9891	204.9996	29174	1.4344
0.9939	205.9974	29316	1.4365
0.9907	206.9952	29458	1.4387
0.9841	208.0	29601	1.4436
0.9898	208.9978	29743	1.4360
0.9886	209.9956	29885	1.4373
0.9867	210.9934	30027	1.4378
0.9775	211.9982	30170	1.4367
0.9839	212.9960	30312	1.4341
0.9836	213.9939	30454	1.4371
0.973	214.9987	30597	1.4428
0.9791	215.9965	30739	1.4409
0.9793	216.9943	30881	1.4453
0.9708	217.9991	31024	1.4395
0.9758	218.9969	31166	1.4391
0.9746	219.9947	31308	1.4441
0.9668	220.9996	31451	1.4398
0.9716	221.9974	31593	1.4408
0.9704	222.9952	31735	1.4398
0.9629	224.0	31878	1.4503
0.9686	224.9978	32020	1.4513
0.9647	225.9956	32162	1.4463
0.964	226.9934	32304	1.4491
0.9581	227.9982	32447	1.4490
0.9634	228.9960	32589	1.4481
0.9625	229.9939	32731	1.4511
0.9556	230.9987	32874	1.4545
0.9612	231.9965	33016	1.4493
0.9594	232.9943	33158	1.4517
0.9507	233.9991	33301	1.4513
0.9552	234.9969	33443	1.4500
0.9566	235.9947	33585	1.4497
0.9475	236.9996	33728	1.4490
0.9518	237.9974	33870	1.4476
0.9532	238.9952	34012	1.4544
0.9439	240.0	34155	1.4613
0.9483	240.9978	34297	1.4579
0.9481	241.9956	34439	1.4494
0.9464	242.9934	34581	1.4528
0.9393	243.9982	34724	1.4551
0.9444	244.9960	34866	1.4617
0.9433	245.9939	35008	1.4575
0.9332	246.9987	35151	1.4657
0.94	247.9965	35293	1.4590
0.9395	248.9943	35435	1.4558
0.9322	249.9991	35578	1.4559
0.9369	250.9969	35720	1.4650
0.934	251.9947	35862	1.4630
0.9287	252.9996	36005	1.4572
0.9342	253.9974	36147	1.4567
0.9325	254.9952	36289	1.4602
0.9262	256.0	36432	1.4594
0.9324	256.9978	36574	1.4610
0.9298	257.9956	36716	1.4599
0.9283	258.9934	36858	1.4608
0.9223	259.9982	37001	1.4580
0.9256	260.9960	37143	1.4642
0.9262	261.9939	37285	1.4665
0.9186	262.9987	37428	1.4640
0.9225	263.9965	37570	1.4633
0.9208	264.9943	37712	1.4626
0.913	265.9991	37855	1.4675
0.9218	266.9969	37997	1.4695
0.9187	267.9947	38139	1.4591
0.9114	268.9996	38282	1.4622
0.9168	269.9974	38424	1.4639
0.9154	270.9952	38566	1.4724
0.9097	272.0	38709	1.4701
0.9121	272.9978	38851	1.4727
0.9121	273.9956	38993	1.4696
0.9103	274.9934	39135	1.4684
0.9023	275.9982	39278	1.4634
0.9112	276.9960	39420	1.4674
0.9088	277.9939	39562	1.4666
0.9005	278.9987	39705	1.4700
0.9054	279.9965	39847	1.4749
0.9064	280.9943	39989	1.4635
0.898	281.9991	40132	1.4673
0.9032	282.9969	40274	1.4748
0.9027	283.9947	40416	1.4740
0.8936	284.9996	40559	1.4745
0.9023	285.9974	40701	1.4686
0.9022	286.9952	40843	1.4723
0.8938	288.0	40986	1.4730
0.9005	288.9978	41128	1.4763
0.8981	289.9956	41270	1.4784
0.8955	290.9934	41412	1.4772
0.8899	291.9982	41555	1.4744
0.8952	292.9960	41697	1.4753
0.8929	293.9939	41839	1.4738
0.8858	294.9987	41982	1.4782
0.8914	295.9965	42124	1.4732
0.8924	296.9943	42266	1.4804
0.8833	297.9991	42409	1.4811
0.8877	298.9969	42551	1.4771
0.8885	299.9947	42693	1.4829
0.8782	300.9996	42836	1.4755
0.8864	301.9974	42978	1.4807
0.8875	302.9952	43120	1.4756
0.8791	304.0	43263	1.4779
0.8844	304.9978	43405	1.4849
0.8825	305.9956	43547	1.4844
0.8826	306.9934	43689	1.4803
0.8743	307.9982	43832	1.4794
0.8793	308.9960	43974	1.4846
0.8784	309.9939	44116	1.4855
0.8738	310.9987	44259	1.4817
0.8763	311.9965	44401	1.4848
0.8777	312.9943	44543	1.4890
0.8698	313.9991	44686	1.4901
0.8738	314.9969	44828	1.4855
0.874	315.9947	44970	1.4853
0.8683	316.9996	45113	1.4852
0.8711	317.9974	45255	1.4862
0.872	318.9952	45397	1.4903
0.8644	320.0	45540	1.4822
0.8694	320.9978	45682	1.4955
0.8693	321.9956	45824	1.4904
0.8667	322.9934	45966	1.4900
0.862	323.9982	46109	1.4890
0.8656	324.9960	46251	1.4914
0.8664	325.9939	46393	1.4945
0.8591	326.9987	46536	1.4904
0.8657	327.9965	46678	1.4891
0.8609	328.9943	46820	1.4909
0.8574	329.9991	46963	1.4880
0.8636	330.9969	47105	1.4917
0.8607	331.9947	47247	1.4901
0.8537	332.9996	47390	1.4912
0.8613	333.9974	47532	1.4974
0.8588	334.9952	47674	1.4955
0.8516	336.0	47817	1.4970
0.8565	336.9978	47959	1.4929
0.8553	337.9956	48101	1.4969
0.8569	338.9934	48243	1.4984
0.8485	339.9982	48386	1.5038
0.854	340.9960	48528	1.4965
0.8529	341.9939	48670	1.4978
0.8485	342.9987	48813	1.5020
0.8524	343.9965	48955	1.4966
0.852	344.9943	49097	1.4978
0.845	345.9991	49240	1.4995
0.8491	346.9969	49382	1.4996
0.849	347.9947	49524	1.4993
0.8434	348.9996	49667	1.5005
0.8481	349.9974	49809	1.5011
0.8477	350.9952	49951	1.5016
0.8397	352.0	50094	1.4931
0.8463	352.9978	50236	1.4950
0.8457	353.9956	50378	1.4989
0.8432	354.9934	50520	1.5022
0.839	355.9982	50663	1.5075
0.8423	356.9960	50805	1.4998
0.841	357.9939	50947	1.4941
0.8337	358.9987	51090	1.5071
0.8399	359.9965	51232	1.4993
0.839	360.9943	51374	1.4980
0.8345	361.9991	51517	1.5069
0.8404	362.9969	51659	1.5073
0.8377	363.9947	51801	1.5053
0.8316	364.9996	51944	1.5047
0.8365	365.9974	52086	1.5005
0.8348	366.9952	52228	1.5071
0.8302	368.0	52371	1.5044
0.8359	368.9978	52513	1.5033
0.8345	369.9956	52655	1.5036
0.8331	370.9934	52797	1.5043
0.8283	371.9982	52940	1.5052
0.8324	372.9960	53082	1.5111
0.8314	373.9939	53224	1.5039
0.8264	374.9987	53367	1.5096
0.8314	375.9965	53509	1.5067
0.8304	376.9943	53651	1.5073
0.8233	377.9991	53794	1.5093
0.8301	378.9969	53936	1.5040
0.8279	379.9947	54078	1.5114
0.8207	380.9996	54221	1.5109
0.8262	381.9974	54363	1.5140
0.8252	382.9952	54505	1.5077
0.8215	384.0	54648	1.5126
0.8251	384.9978	54790	1.5118
0.8239	385.9956	54932	1.5160
0.8244	386.9934	55074	1.5170
0.8171	387.9982	55217	1.5175
0.8204	388.9960	55359	1.5128
0.8222	389.9939	55501	1.5137
0.8176	390.9987	55644	1.5144
0.8193	391.9965	55786	1.5187
0.8198	392.9943	55928	1.5097
0.8138	393.9991	56071	1.5152
0.8185	394.9969	56213	1.5116
0.8193	395.9947	56355	1.5135
0.812	396.9996	56498	1.5155
0.8183	397.9974	56640	1.5123
0.817	398.9952	56782	1.5140
0.8109	400.0	56925	1.5172
0.8157	400.9978	57067	1.5103
0.8158	401.9956	57209	1.5197
0.8142	402.9934	57351	1.5216
0.8101	403.9982	57494	1.5114
0.8142	404.9960	57636	1.5184
0.8121	405.9939	57778	1.5158
0.8073	406.9987	57921	1.5166
0.8117	407.9965	58063	1.5177
0.8114	408.9943	58205	1.5145
0.808	409.9991	58348	1.5181
0.8118	410.9969	58490	1.5155
0.8109	411.9947	58632	1.5175
0.8057	412.9996	58775	1.5217
0.8104	413.9974	58917	1.5172
0.8087	414.9952	59059	1.5182
0.8013	416.0	59202	1.5169
0.8062	416.9978	59344	1.5161
0.8077	417.9956	59486	1.5176
0.8049	418.9934	59628	1.5239
0.8015	419.9982	59771	1.5181
0.8066	420.9960	59913	1.5177
0.8037	421.9939	60055	1.5206
0.8002	422.9987	60198	1.5196
0.8043	423.9965	60340	1.5116
0.8053	424.9943	60482	1.5249
0.7982	425.9991	60625	1.5203
0.8018	426.9969	60767	1.5256
0.8028	427.9947	60909	1.5218
0.7972	428.9996	61052	1.5161
0.8012	429.9974	61194	1.5211
0.801	430.9952	61336	1.5185
0.7948	432.0	61479	1.5224
0.799	432.9978	61621	1.5240
0.7996	433.9956	61763	1.5181
0.7989	434.9934	61905	1.5224
0.7951	435.9982	62048	1.5220
0.7978	436.9960	62190	1.5252
0.7981	437.9939	62332	1.5301
0.7903	438.9987	62475	1.5223
0.7975	439.9965	62617	1.5278
0.7972	440.9943	62759	1.5215
0.7891	441.9991	62902	1.5234
0.7954	442.9969	63044	1.5216
0.7951	443.9947	63186	1.5265
0.7889	444.9996	63329	1.5171
0.7959	445.9974	63471	1.5235
0.7945	446.9952	63613	1.5242
0.7899	448.0	63756	1.5200
0.7929	448.9978	63898	1.5213
0.7937	449.9956	64040	1.5215
0.7938	450.9934	64182	1.5279
0.7864	451.9982	64325	1.5258
0.7959	452.9960	64467	1.5217
0.7901	453.9939	64609	1.5254
0.787	454.9987	64752	1.5241
0.7928	455.9965	64894	1.5225
0.7888	456.9943	65036	1.5216
0.7844	457.9991	65179	1.5232
0.7921	458.9969	65321	1.5255
0.7907	459.9947	65463	1.5261
0.7847	460.9996	65606	1.5319
0.7907	461.9974	65748	1.5237
0.7903	462.9952	65890	1.5275
0.7834	464.0	66033	1.5275
0.7887	464.9978	66175	1.5310
0.7887	465.9956	66317	1.5327
0.7872	466.9934	66459	1.5247
0.7811	467.9982	66602	1.5282
0.7866	468.9960	66744	1.5300
0.7874	469.9939	66886	1.5306
0.7808	470.9987	67029	1.5324
0.7864	471.9965	67171	1.5278
0.7871	472.9943	67313	1.5250
0.7816	473.9991	67456	1.5233
0.7829	474.9969	67598	1.5187
0.7863	475.9947	67740	1.5263
0.7801	476.9996	67883	1.5261
0.7825	477.9974	68025	1.5276
0.7854	478.9952	68167	1.5270
0.7792	480.0	68310	1.5248
0.7838	480.9978	68452	1.5295
0.7851	481.9956	68594	1.5269
0.7824	482.9934	68736	1.5314
0.7788	483.9982	68879	1.5308
0.783	484.9960	69021	1.5314
0.7844	485.9939	69163	1.5198
0.7767	486.9987	69306	1.5268
0.7822	487.9965	69448	1.5310
0.7836	488.9943	69590	1.5263
0.777	489.9991	69733	1.5286
0.7835	490.9969	69875	1.5229
0.782	491.9947	70017	1.5306
0.7777	492.9996	70160	1.5275
0.7826	493.9974	70302	1.5286
0.7826	494.9952	70444	1.5286
0.7762	496.0	70587	1.5241
0.7827	496.9978	70729	1.5268
0.78	497.9956	70871	1.5275
0.7756	498.9021	71000	1.5247

Framework versions

• Transformers 4.41.2
• Pytorch 2.2.1
• Datasets 2.19.2
• Tokenizers 0.19.1