chchen commited on
Commit
1d7ba37
·
verified ·
1 Parent(s): 6248113

End of training

Browse files
README.md CHANGED
@@ -4,6 +4,7 @@ license: llama3
4
  base_model: aaditya/Llama3-OpenBioLLM-8B
5
  tags:
6
  - llama-factory
 
7
  - generated_from_trainer
8
  model-index:
9
  - name: Llama3-OpenBioLLM-8B-PsyCourse-info-fold1
@@ -15,9 +16,9 @@ should probably proofread and complete it, then remove this comment. -->
15
 
16
  # Llama3-OpenBioLLM-8B-PsyCourse-info-fold1
17
 
18
- This model is a fine-tuned version of [aaditya/Llama3-OpenBioLLM-8B](https://huggingface.co/aaditya/Llama3-OpenBioLLM-8B) on an unknown dataset.
19
  It achieves the following results on the evaluation set:
20
- - Loss: 0.1527
21
 
22
  ## Model description
23
 
 
4
  base_model: aaditya/Llama3-OpenBioLLM-8B
5
  tags:
6
  - llama-factory
7
+ - lora
8
  - generated_from_trainer
9
  model-index:
10
  - name: Llama3-OpenBioLLM-8B-PsyCourse-info-fold1
 
16
 
17
  # Llama3-OpenBioLLM-8B-PsyCourse-info-fold1
18
 
19
+ This model is a fine-tuned version of [aaditya/Llama3-OpenBioLLM-8B](https://huggingface.co/aaditya/Llama3-OpenBioLLM-8B) on the course-info-train-fold1 dataset.
20
  It achieves the following results on the evaluation set:
21
+ - Loss: 0.1507
22
 
23
  ## Model description
24
 
all_results.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 4.938271604938271,
3
+ "eval_loss": 0.15065895020961761,
4
+ "eval_runtime": 18.9827,
5
+ "eval_samples_per_second": 2.371,
6
+ "eval_steps_per_second": 2.371,
7
+ "total_flos": 1.2628188898630042e+17,
8
+ "train_loss": 0.19368722805380822,
9
+ "train_runtime": 3339.1879,
10
+ "train_samples_per_second": 0.606,
11
+ "train_steps_per_second": 0.037
12
+ }
eval_results.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 4.938271604938271,
3
+ "eval_loss": 0.15065895020961761,
4
+ "eval_runtime": 18.9827,
5
+ "eval_samples_per_second": 2.371,
6
+ "eval_steps_per_second": 2.371
7
+ }
train_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 4.938271604938271,
3
+ "total_flos": 1.2628188898630042e+17,
4
+ "train_loss": 0.19368722805380822,
5
+ "train_runtime": 3339.1879,
6
+ "train_samples_per_second": 0.606,
7
+ "train_steps_per_second": 0.037
8
+ }
trainer_state.json ADDED
@@ -0,0 +1,1013 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": 0.15065895020961761,
3
+ "best_model_checkpoint": "saves/psy-course-info/Llama3-OpenBioLLM-8B/train/fold1/checkpoint-90",
4
+ "epoch": 4.938271604938271,
5
+ "eval_steps": 10,
6
+ "global_step": 125,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.03950617283950617,
13
+ "grad_norm": 3.0266354084014893,
14
+ "learning_rate": 7.692307692307694e-06,
15
+ "loss": 0.8044,
16
+ "step": 1
17
+ },
18
+ {
19
+ "epoch": 0.07901234567901234,
20
+ "grad_norm": 2.7545979022979736,
21
+ "learning_rate": 1.5384615384615387e-05,
22
+ "loss": 1.0543,
23
+ "step": 2
24
+ },
25
+ {
26
+ "epoch": 0.11851851851851852,
27
+ "grad_norm": 3.436570405960083,
28
+ "learning_rate": 2.307692307692308e-05,
29
+ "loss": 0.9637,
30
+ "step": 3
31
+ },
32
+ {
33
+ "epoch": 0.1580246913580247,
34
+ "grad_norm": 3.260969638824463,
35
+ "learning_rate": 3.0769230769230774e-05,
36
+ "loss": 0.9253,
37
+ "step": 4
38
+ },
39
+ {
40
+ "epoch": 0.19753086419753085,
41
+ "grad_norm": 3.48811411857605,
42
+ "learning_rate": 3.846153846153846e-05,
43
+ "loss": 1.0063,
44
+ "step": 5
45
+ },
46
+ {
47
+ "epoch": 0.23703703703703705,
48
+ "grad_norm": 3.1750481128692627,
49
+ "learning_rate": 4.615384615384616e-05,
50
+ "loss": 0.9354,
51
+ "step": 6
52
+ },
53
+ {
54
+ "epoch": 0.2765432098765432,
55
+ "grad_norm": 2.6605045795440674,
56
+ "learning_rate": 5.384615384615385e-05,
57
+ "loss": 0.7956,
58
+ "step": 7
59
+ },
60
+ {
61
+ "epoch": 0.3160493827160494,
62
+ "grad_norm": 1.8658115863800049,
63
+ "learning_rate": 6.153846153846155e-05,
64
+ "loss": 0.7156,
65
+ "step": 8
66
+ },
67
+ {
68
+ "epoch": 0.35555555555555557,
69
+ "grad_norm": 1.9442992210388184,
70
+ "learning_rate": 6.923076923076924e-05,
71
+ "loss": 0.7273,
72
+ "step": 9
73
+ },
74
+ {
75
+ "epoch": 0.3950617283950617,
76
+ "grad_norm": 1.4981564283370972,
77
+ "learning_rate": 7.692307692307693e-05,
78
+ "loss": 0.5198,
79
+ "step": 10
80
+ },
81
+ {
82
+ "epoch": 0.3950617283950617,
83
+ "eval_loss": 0.5015671849250793,
84
+ "eval_runtime": 19.3011,
85
+ "eval_samples_per_second": 2.331,
86
+ "eval_steps_per_second": 2.331,
87
+ "step": 10
88
+ },
89
+ {
90
+ "epoch": 0.4345679012345679,
91
+ "grad_norm": 1.3176920413970947,
92
+ "learning_rate": 8.461538461538461e-05,
93
+ "loss": 0.4448,
94
+ "step": 11
95
+ },
96
+ {
97
+ "epoch": 0.4740740740740741,
98
+ "grad_norm": 1.4542604684829712,
99
+ "learning_rate": 9.230769230769232e-05,
100
+ "loss": 0.5058,
101
+ "step": 12
102
+ },
103
+ {
104
+ "epoch": 0.5135802469135803,
105
+ "grad_norm": 1.2523777484893799,
106
+ "learning_rate": 0.0001,
107
+ "loss": 0.4258,
108
+ "step": 13
109
+ },
110
+ {
111
+ "epoch": 0.5530864197530864,
112
+ "grad_norm": 1.061900019645691,
113
+ "learning_rate": 9.998033131915266e-05,
114
+ "loss": 0.2848,
115
+ "step": 14
116
+ },
117
+ {
118
+ "epoch": 0.5925925925925926,
119
+ "grad_norm": 1.009539008140564,
120
+ "learning_rate": 9.992134075089084e-05,
121
+ "loss": 0.3333,
122
+ "step": 15
123
+ },
124
+ {
125
+ "epoch": 0.6320987654320988,
126
+ "grad_norm": 1.0671191215515137,
127
+ "learning_rate": 9.982307470588098e-05,
128
+ "loss": 0.3718,
129
+ "step": 16
130
+ },
131
+ {
132
+ "epoch": 0.671604938271605,
133
+ "grad_norm": 0.9506099820137024,
134
+ "learning_rate": 9.968561049466214e-05,
135
+ "loss": 0.232,
136
+ "step": 17
137
+ },
138
+ {
139
+ "epoch": 0.7111111111111111,
140
+ "grad_norm": 1.0324467420578003,
141
+ "learning_rate": 9.950905626682228e-05,
142
+ "loss": 0.2587,
143
+ "step": 18
144
+ },
145
+ {
146
+ "epoch": 0.7506172839506173,
147
+ "grad_norm": 0.7604952454566956,
148
+ "learning_rate": 9.92935509259118e-05,
149
+ "loss": 0.2877,
150
+ "step": 19
151
+ },
152
+ {
153
+ "epoch": 0.7901234567901234,
154
+ "grad_norm": 0.9865992069244385,
155
+ "learning_rate": 9.903926402016153e-05,
156
+ "loss": 0.255,
157
+ "step": 20
158
+ },
159
+ {
160
+ "epoch": 0.7901234567901234,
161
+ "eval_loss": 0.24946804344654083,
162
+ "eval_runtime": 19.2039,
163
+ "eval_samples_per_second": 2.343,
164
+ "eval_steps_per_second": 2.343,
165
+ "step": 20
166
+ },
167
+ {
168
+ "epoch": 0.8296296296296296,
169
+ "grad_norm": 0.6814547181129456,
170
+ "learning_rate": 9.874639560909117e-05,
171
+ "loss": 0.2186,
172
+ "step": 21
173
+ },
174
+ {
175
+ "epoch": 0.8691358024691358,
176
+ "grad_norm": 0.6453701853752136,
177
+ "learning_rate": 9.841517610611309e-05,
178
+ "loss": 0.2174,
179
+ "step": 22
180
+ },
181
+ {
182
+ "epoch": 0.908641975308642,
183
+ "grad_norm": 0.6355133056640625,
184
+ "learning_rate": 9.804586609725499e-05,
185
+ "loss": 0.1912,
186
+ "step": 23
187
+ },
188
+ {
189
+ "epoch": 0.9481481481481482,
190
+ "grad_norm": 0.6928957104682922,
191
+ "learning_rate": 9.763875613614482e-05,
192
+ "loss": 0.2412,
193
+ "step": 24
194
+ },
195
+ {
196
+ "epoch": 0.9876543209876543,
197
+ "grad_norm": 0.6595496535301208,
198
+ "learning_rate": 9.719416651541839e-05,
199
+ "loss": 0.2774,
200
+ "step": 25
201
+ },
202
+ {
203
+ "epoch": 1.0271604938271606,
204
+ "grad_norm": 1.4999443292617798,
205
+ "learning_rate": 9.671244701472999e-05,
206
+ "loss": 0.3648,
207
+ "step": 26
208
+ },
209
+ {
210
+ "epoch": 1.0666666666666667,
211
+ "grad_norm": 0.5701374411582947,
212
+ "learning_rate": 9.619397662556435e-05,
213
+ "loss": 0.1664,
214
+ "step": 27
215
+ },
216
+ {
217
+ "epoch": 1.106172839506173,
218
+ "grad_norm": 0.6932143568992615,
219
+ "learning_rate": 9.563916325306594e-05,
220
+ "loss": 0.2712,
221
+ "step": 28
222
+ },
223
+ {
224
+ "epoch": 1.145679012345679,
225
+ "grad_norm": 0.5702258944511414,
226
+ "learning_rate": 9.504844339512095e-05,
227
+ "loss": 0.1666,
228
+ "step": 29
229
+ },
230
+ {
231
+ "epoch": 1.1851851851851851,
232
+ "grad_norm": 0.6305382251739502,
233
+ "learning_rate": 9.442228179894362e-05,
234
+ "loss": 0.2111,
235
+ "step": 30
236
+ },
237
+ {
238
+ "epoch": 1.1851851851851851,
239
+ "eval_loss": 0.20211933553218842,
240
+ "eval_runtime": 19.2511,
241
+ "eval_samples_per_second": 2.338,
242
+ "eval_steps_per_second": 2.338,
243
+ "step": 30
244
+ },
245
+ {
246
+ "epoch": 1.2246913580246914,
247
+ "grad_norm": 0.6781132221221924,
248
+ "learning_rate": 9.376117109543769e-05,
249
+ "loss": 0.1878,
250
+ "step": 31
251
+ },
252
+ {
253
+ "epoch": 1.2641975308641975,
254
+ "grad_norm": 0.7474590539932251,
255
+ "learning_rate": 9.306563141162046e-05,
256
+ "loss": 0.2222,
257
+ "step": 32
258
+ },
259
+ {
260
+ "epoch": 1.3037037037037038,
261
+ "grad_norm": 0.6464303731918335,
262
+ "learning_rate": 9.233620996141421e-05,
263
+ "loss": 0.2205,
264
+ "step": 33
265
+ },
266
+ {
267
+ "epoch": 1.34320987654321,
268
+ "grad_norm": 0.6342009902000427,
269
+ "learning_rate": 9.157348061512727e-05,
270
+ "loss": 0.1516,
271
+ "step": 34
272
+ },
273
+ {
274
+ "epoch": 1.382716049382716,
275
+ "grad_norm": 0.5502495765686035,
276
+ "learning_rate": 9.077804344796302e-05,
277
+ "loss": 0.1929,
278
+ "step": 35
279
+ },
280
+ {
281
+ "epoch": 1.4222222222222223,
282
+ "grad_norm": 0.9800194501876831,
283
+ "learning_rate": 8.995052426791247e-05,
284
+ "loss": 0.1887,
285
+ "step": 36
286
+ },
287
+ {
288
+ "epoch": 1.4617283950617284,
289
+ "grad_norm": 0.4909915328025818,
290
+ "learning_rate": 8.90915741234015e-05,
291
+ "loss": 0.1198,
292
+ "step": 37
293
+ },
294
+ {
295
+ "epoch": 1.5012345679012347,
296
+ "grad_norm": 0.5605656504631042,
297
+ "learning_rate": 8.820186879108038e-05,
298
+ "loss": 0.1327,
299
+ "step": 38
300
+ },
301
+ {
302
+ "epoch": 1.5407407407407407,
303
+ "grad_norm": 1.0153040885925293,
304
+ "learning_rate": 8.728210824415827e-05,
305
+ "loss": 0.2028,
306
+ "step": 39
307
+ },
308
+ {
309
+ "epoch": 1.5802469135802468,
310
+ "grad_norm": 0.6423051953315735,
311
+ "learning_rate": 8.633301610170135e-05,
312
+ "loss": 0.1268,
313
+ "step": 40
314
+ },
315
+ {
316
+ "epoch": 1.5802469135802468,
317
+ "eval_loss": 0.17976433038711548,
318
+ "eval_runtime": 19.2355,
319
+ "eval_samples_per_second": 2.339,
320
+ "eval_steps_per_second": 2.339,
321
+ "step": 40
322
+ },
323
+ {
324
+ "epoch": 1.6197530864197531,
325
+ "grad_norm": 0.7444939613342285,
326
+ "learning_rate": 8.535533905932738e-05,
327
+ "loss": 0.1904,
328
+ "step": 41
329
+ },
330
+ {
331
+ "epoch": 1.6592592592592592,
332
+ "grad_norm": 0.5496087670326233,
333
+ "learning_rate": 8.434984630174509e-05,
334
+ "loss": 0.1364,
335
+ "step": 42
336
+ },
337
+ {
338
+ "epoch": 1.6987654320987655,
339
+ "grad_norm": 0.6592587232589722,
340
+ "learning_rate": 8.33173288976002e-05,
341
+ "loss": 0.1524,
342
+ "step": 43
343
+ },
344
+ {
345
+ "epoch": 1.7382716049382716,
346
+ "grad_norm": 0.6778094172477722,
347
+ "learning_rate": 8.225859917710439e-05,
348
+ "loss": 0.1444,
349
+ "step": 44
350
+ },
351
+ {
352
+ "epoch": 1.7777777777777777,
353
+ "grad_norm": 0.8962506651878357,
354
+ "learning_rate": 8.117449009293668e-05,
355
+ "loss": 0.2192,
356
+ "step": 45
357
+ },
358
+ {
359
+ "epoch": 1.817283950617284,
360
+ "grad_norm": 0.6817356944084167,
361
+ "learning_rate": 8.006585456492029e-05,
362
+ "loss": 0.1513,
363
+ "step": 46
364
+ },
365
+ {
366
+ "epoch": 1.8567901234567903,
367
+ "grad_norm": 0.6316496729850769,
368
+ "learning_rate": 7.89335648089903e-05,
369
+ "loss": 0.1405,
370
+ "step": 47
371
+ },
372
+ {
373
+ "epoch": 1.8962962962962964,
374
+ "grad_norm": 0.6903754472732544,
375
+ "learning_rate": 7.777851165098012e-05,
376
+ "loss": 0.1162,
377
+ "step": 48
378
+ },
379
+ {
380
+ "epoch": 1.9358024691358025,
381
+ "grad_norm": 0.7566717267036438,
382
+ "learning_rate": 7.660160382576683e-05,
383
+ "loss": 0.2169,
384
+ "step": 49
385
+ },
386
+ {
387
+ "epoch": 1.9753086419753085,
388
+ "grad_norm": 0.6865474581718445,
389
+ "learning_rate": 7.540376726232648e-05,
390
+ "loss": 0.122,
391
+ "step": 50
392
+ },
393
+ {
394
+ "epoch": 1.9753086419753085,
395
+ "eval_loss": 0.16953954100608826,
396
+ "eval_runtime": 19.2378,
397
+ "eval_samples_per_second": 2.339,
398
+ "eval_steps_per_second": 2.339,
399
+ "step": 50
400
+ },
401
+ {
402
+ "epoch": 2.0148148148148146,
403
+ "grad_norm": 1.2375051975250244,
404
+ "learning_rate": 7.4185944355262e-05,
405
+ "loss": 0.226,
406
+ "step": 51
407
+ },
408
+ {
409
+ "epoch": 2.054320987654321,
410
+ "grad_norm": 0.6000056862831116,
411
+ "learning_rate": 7.294909322337689e-05,
412
+ "loss": 0.0914,
413
+ "step": 52
414
+ },
415
+ {
416
+ "epoch": 2.093827160493827,
417
+ "grad_norm": 0.5429364442825317,
418
+ "learning_rate": 7.169418695587791e-05,
419
+ "loss": 0.1101,
420
+ "step": 53
421
+ },
422
+ {
423
+ "epoch": 2.1333333333333333,
424
+ "grad_norm": 0.6248661279678345,
425
+ "learning_rate": 7.042221284679982e-05,
426
+ "loss": 0.1528,
427
+ "step": 54
428
+ },
429
+ {
430
+ "epoch": 2.1728395061728394,
431
+ "grad_norm": 0.6743581295013428,
432
+ "learning_rate": 6.91341716182545e-05,
433
+ "loss": 0.1233,
434
+ "step": 55
435
+ },
436
+ {
437
+ "epoch": 2.212345679012346,
438
+ "grad_norm": 0.7042228579521179,
439
+ "learning_rate": 6.783107663311565e-05,
440
+ "loss": 0.1189,
441
+ "step": 56
442
+ },
443
+ {
444
+ "epoch": 2.251851851851852,
445
+ "grad_norm": 0.48001235723495483,
446
+ "learning_rate": 6.651395309775837e-05,
447
+ "loss": 0.0855,
448
+ "step": 57
449
+ },
450
+ {
451
+ "epoch": 2.291358024691358,
452
+ "grad_norm": 0.6637732982635498,
453
+ "learning_rate": 6.518383725548074e-05,
454
+ "loss": 0.0977,
455
+ "step": 58
456
+ },
457
+ {
458
+ "epoch": 2.330864197530864,
459
+ "grad_norm": 0.5641061067581177,
460
+ "learning_rate": 6.384177557124247e-05,
461
+ "loss": 0.0843,
462
+ "step": 59
463
+ },
464
+ {
465
+ "epoch": 2.3703703703703702,
466
+ "grad_norm": 0.6591428518295288,
467
+ "learning_rate": 6.248882390836135e-05,
468
+ "loss": 0.0984,
469
+ "step": 60
470
+ },
471
+ {
472
+ "epoch": 2.3703703703703702,
473
+ "eval_loss": 0.16232524812221527,
474
+ "eval_runtime": 19.2267,
475
+ "eval_samples_per_second": 2.34,
476
+ "eval_steps_per_second": 2.34,
477
+ "step": 60
478
+ },
479
+ {
480
+ "epoch": 2.4098765432098768,
481
+ "grad_norm": 0.6936379075050354,
482
+ "learning_rate": 6.112604669781572e-05,
483
+ "loss": 0.0997,
484
+ "step": 61
485
+ },
486
+ {
487
+ "epoch": 2.449382716049383,
488
+ "grad_norm": 0.8590863943099976,
489
+ "learning_rate": 5.9754516100806423e-05,
490
+ "loss": 0.1849,
491
+ "step": 62
492
+ },
493
+ {
494
+ "epoch": 2.488888888888889,
495
+ "grad_norm": 0.8724762201309204,
496
+ "learning_rate": 5.837531116523682e-05,
497
+ "loss": 0.1388,
498
+ "step": 63
499
+ },
500
+ {
501
+ "epoch": 2.528395061728395,
502
+ "grad_norm": 0.7268065810203552,
503
+ "learning_rate": 5.698951697677498e-05,
504
+ "loss": 0.1579,
505
+ "step": 64
506
+ },
507
+ {
508
+ "epoch": 2.567901234567901,
509
+ "grad_norm": 0.6837067604064941,
510
+ "learning_rate": 5.559822380516539e-05,
511
+ "loss": 0.1277,
512
+ "step": 65
513
+ },
514
+ {
515
+ "epoch": 2.6074074074074076,
516
+ "grad_norm": 0.584983229637146,
517
+ "learning_rate": 5.420252624646238e-05,
518
+ "loss": 0.0922,
519
+ "step": 66
520
+ },
521
+ {
522
+ "epoch": 2.6469135802469137,
523
+ "grad_norm": 0.73332279920578,
524
+ "learning_rate": 5.2803522361859594e-05,
525
+ "loss": 0.1113,
526
+ "step": 67
527
+ },
528
+ {
529
+ "epoch": 2.68641975308642,
530
+ "grad_norm": 0.5340074896812439,
531
+ "learning_rate": 5.140231281379345e-05,
532
+ "loss": 0.0899,
533
+ "step": 68
534
+ },
535
+ {
536
+ "epoch": 2.725925925925926,
537
+ "grad_norm": 0.618689775466919,
538
+ "learning_rate": 5e-05,
539
+ "loss": 0.0749,
540
+ "step": 69
541
+ },
542
+ {
543
+ "epoch": 2.765432098765432,
544
+ "grad_norm": 0.5566624402999878,
545
+ "learning_rate": 4.859768718620656e-05,
546
+ "loss": 0.0732,
547
+ "step": 70
548
+ },
549
+ {
550
+ "epoch": 2.765432098765432,
551
+ "eval_loss": 0.1524766981601715,
552
+ "eval_runtime": 19.2317,
553
+ "eval_samples_per_second": 2.34,
554
+ "eval_steps_per_second": 2.34,
555
+ "step": 70
556
+ },
557
+ {
558
+ "epoch": 2.8049382716049385,
559
+ "grad_norm": 0.571286141872406,
560
+ "learning_rate": 4.7196477638140404e-05,
561
+ "loss": 0.0796,
562
+ "step": 71
563
+ },
564
+ {
565
+ "epoch": 2.8444444444444446,
566
+ "grad_norm": 0.6639549136161804,
567
+ "learning_rate": 4.579747375353763e-05,
568
+ "loss": 0.1408,
569
+ "step": 72
570
+ },
571
+ {
572
+ "epoch": 2.8839506172839506,
573
+ "grad_norm": 0.8859966993331909,
574
+ "learning_rate": 4.4401776194834613e-05,
575
+ "loss": 0.1507,
576
+ "step": 73
577
+ },
578
+ {
579
+ "epoch": 2.9234567901234567,
580
+ "grad_norm": 0.6509288549423218,
581
+ "learning_rate": 4.3010483023225045e-05,
582
+ "loss": 0.1063,
583
+ "step": 74
584
+ },
585
+ {
586
+ "epoch": 2.962962962962963,
587
+ "grad_norm": 0.813045859336853,
588
+ "learning_rate": 4.162468883476319e-05,
589
+ "loss": 0.1217,
590
+ "step": 75
591
+ },
592
+ {
593
+ "epoch": 3.0024691358024693,
594
+ "grad_norm": 1.411219835281372,
595
+ "learning_rate": 4.0245483899193595e-05,
596
+ "loss": 0.218,
597
+ "step": 76
598
+ },
599
+ {
600
+ "epoch": 3.0419753086419754,
601
+ "grad_norm": 0.49604693055152893,
602
+ "learning_rate": 3.887395330218429e-05,
603
+ "loss": 0.0683,
604
+ "step": 77
605
+ },
606
+ {
607
+ "epoch": 3.0814814814814815,
608
+ "grad_norm": 0.6909966468811035,
609
+ "learning_rate": 3.7511176091638653e-05,
610
+ "loss": 0.0829,
611
+ "step": 78
612
+ },
613
+ {
614
+ "epoch": 3.1209876543209876,
615
+ "grad_norm": 0.6037319302558899,
616
+ "learning_rate": 3.6158224428757535e-05,
617
+ "loss": 0.1044,
618
+ "step": 79
619
+ },
620
+ {
621
+ "epoch": 3.1604938271604937,
622
+ "grad_norm": 0.5923836827278137,
623
+ "learning_rate": 3.4816162744519263e-05,
624
+ "loss": 0.0726,
625
+ "step": 80
626
+ },
627
+ {
628
+ "epoch": 3.1604938271604937,
629
+ "eval_loss": 0.1539728194475174,
630
+ "eval_runtime": 19.2384,
631
+ "eval_samples_per_second": 2.339,
632
+ "eval_steps_per_second": 2.339,
633
+ "step": 80
634
+ },
635
+ {
636
+ "epoch": 3.2,
637
+ "grad_norm": 0.55913907289505,
638
+ "learning_rate": 3.3486046902241664e-05,
639
+ "loss": 0.0828,
640
+ "step": 81
641
+ },
642
+ {
643
+ "epoch": 3.2395061728395063,
644
+ "grad_norm": 0.5650818943977356,
645
+ "learning_rate": 3.216892336688435e-05,
646
+ "loss": 0.0742,
647
+ "step": 82
648
+ },
649
+ {
650
+ "epoch": 3.2790123456790123,
651
+ "grad_norm": 0.7615413069725037,
652
+ "learning_rate": 3.086582838174551e-05,
653
+ "loss": 0.072,
654
+ "step": 83
655
+ },
656
+ {
657
+ "epoch": 3.3185185185185184,
658
+ "grad_norm": 0.5601615905761719,
659
+ "learning_rate": 2.9577787153200197e-05,
660
+ "loss": 0.0921,
661
+ "step": 84
662
+ },
663
+ {
664
+ "epoch": 3.3580246913580245,
665
+ "grad_norm": 0.7153237462043762,
666
+ "learning_rate": 2.8305813044122097e-05,
667
+ "loss": 0.0752,
668
+ "step": 85
669
+ },
670
+ {
671
+ "epoch": 3.397530864197531,
672
+ "grad_norm": 0.6471256017684937,
673
+ "learning_rate": 2.705090677662311e-05,
674
+ "loss": 0.0871,
675
+ "step": 86
676
+ },
677
+ {
678
+ "epoch": 3.437037037037037,
679
+ "grad_norm": 0.6173242926597595,
680
+ "learning_rate": 2.581405564473801e-05,
681
+ "loss": 0.0874,
682
+ "step": 87
683
+ },
684
+ {
685
+ "epoch": 3.476543209876543,
686
+ "grad_norm": 0.5788525938987732,
687
+ "learning_rate": 2.459623273767354e-05,
688
+ "loss": 0.0781,
689
+ "step": 88
690
+ },
691
+ {
692
+ "epoch": 3.5160493827160493,
693
+ "grad_norm": 0.5925896763801575,
694
+ "learning_rate": 2.3398396174233178e-05,
695
+ "loss": 0.0736,
696
+ "step": 89
697
+ },
698
+ {
699
+ "epoch": 3.5555555555555554,
700
+ "grad_norm": 0.5686795711517334,
701
+ "learning_rate": 2.2221488349019903e-05,
702
+ "loss": 0.0705,
703
+ "step": 90
704
+ },
705
+ {
706
+ "epoch": 3.5555555555555554,
707
+ "eval_loss": 0.15065895020961761,
708
+ "eval_runtime": 19.2412,
709
+ "eval_samples_per_second": 2.339,
710
+ "eval_steps_per_second": 2.339,
711
+ "step": 90
712
+ },
713
+ {
714
+ "epoch": 3.595061728395062,
715
+ "grad_norm": 0.715358555316925,
716
+ "learning_rate": 2.1066435191009715e-05,
717
+ "loss": 0.1066,
718
+ "step": 91
719
+ },
720
+ {
721
+ "epoch": 3.634567901234568,
722
+ "grad_norm": 0.6285521388053894,
723
+ "learning_rate": 1.9934145435079702e-05,
724
+ "loss": 0.0742,
725
+ "step": 92
726
+ },
727
+ {
728
+ "epoch": 3.674074074074074,
729
+ "grad_norm": 0.7287279367446899,
730
+ "learning_rate": 1.8825509907063327e-05,
731
+ "loss": 0.076,
732
+ "step": 93
733
+ },
734
+ {
735
+ "epoch": 3.71358024691358,
736
+ "grad_norm": 0.584709107875824,
737
+ "learning_rate": 1.774140082289563e-05,
738
+ "loss": 0.0787,
739
+ "step": 94
740
+ },
741
+ {
742
+ "epoch": 3.753086419753086,
743
+ "grad_norm": 0.5803191661834717,
744
+ "learning_rate": 1.6682671102399805e-05,
745
+ "loss": 0.0657,
746
+ "step": 95
747
+ },
748
+ {
749
+ "epoch": 3.7925925925925927,
750
+ "grad_norm": 0.6027998924255371,
751
+ "learning_rate": 1.5650153698254916e-05,
752
+ "loss": 0.0774,
753
+ "step": 96
754
+ },
755
+ {
756
+ "epoch": 3.832098765432099,
757
+ "grad_norm": 0.6973051428794861,
758
+ "learning_rate": 1.4644660940672627e-05,
759
+ "loss": 0.0817,
760
+ "step": 97
761
+ },
762
+ {
763
+ "epoch": 3.871604938271605,
764
+ "grad_norm": 0.6387346386909485,
765
+ "learning_rate": 1.3666983898298657e-05,
766
+ "loss": 0.0588,
767
+ "step": 98
768
+ },
769
+ {
770
+ "epoch": 3.911111111111111,
771
+ "grad_norm": 0.6541065573692322,
772
+ "learning_rate": 1.2717891755841722e-05,
773
+ "loss": 0.0782,
774
+ "step": 99
775
+ },
776
+ {
777
+ "epoch": 3.950617283950617,
778
+ "grad_norm": 0.6163211464881897,
779
+ "learning_rate": 1.1798131208919627e-05,
780
+ "loss": 0.097,
781
+ "step": 100
782
+ },
783
+ {
784
+ "epoch": 3.950617283950617,
785
+ "eval_loss": 0.15166276693344116,
786
+ "eval_runtime": 19.2442,
787
+ "eval_samples_per_second": 2.338,
788
+ "eval_steps_per_second": 2.338,
789
+ "step": 100
790
+ },
791
+ {
792
+ "epoch": 3.9901234567901236,
793
+ "grad_norm": 0.834725558757782,
794
+ "learning_rate": 1.090842587659851e-05,
795
+ "loss": 0.0834,
796
+ "step": 101
797
+ },
798
+ {
799
+ "epoch": 4.029629629629629,
800
+ "grad_norm": 1.2114893198013306,
801
+ "learning_rate": 1.004947573208756e-05,
802
+ "loss": 0.1065,
803
+ "step": 102
804
+ },
805
+ {
806
+ "epoch": 4.069135802469136,
807
+ "grad_norm": 0.4666559398174286,
808
+ "learning_rate": 9.221956552036992e-06,
809
+ "loss": 0.0553,
810
+ "step": 103
811
+ },
812
+ {
813
+ "epoch": 4.108641975308642,
814
+ "grad_norm": 0.8054733872413635,
815
+ "learning_rate": 8.426519384872733e-06,
816
+ "loss": 0.0724,
817
+ "step": 104
818
+ },
819
+ {
820
+ "epoch": 4.148148148148148,
821
+ "grad_norm": 0.45573511719703674,
822
+ "learning_rate": 7.663790038585793e-06,
823
+ "loss": 0.046,
824
+ "step": 105
825
+ },
826
+ {
827
+ "epoch": 4.187654320987654,
828
+ "grad_norm": 0.6370751857757568,
829
+ "learning_rate": 6.934368588379553e-06,
830
+ "loss": 0.0771,
831
+ "step": 106
832
+ },
833
+ {
834
+ "epoch": 4.22716049382716,
835
+ "grad_norm": 0.5112573504447937,
836
+ "learning_rate": 6.238828904562316e-06,
837
+ "loss": 0.0621,
838
+ "step": 107
839
+ },
840
+ {
841
+ "epoch": 4.266666666666667,
842
+ "grad_norm": 0.5506611466407776,
843
+ "learning_rate": 5.577718201056392e-06,
844
+ "loss": 0.0742,
845
+ "step": 108
846
+ },
847
+ {
848
+ "epoch": 4.306172839506173,
849
+ "grad_norm": 0.6445054411888123,
850
+ "learning_rate": 4.951556604879048e-06,
851
+ "loss": 0.0732,
852
+ "step": 109
853
+ },
854
+ {
855
+ "epoch": 4.345679012345679,
856
+ "grad_norm": 0.5202900171279907,
857
+ "learning_rate": 4.360836746934055e-06,
858
+ "loss": 0.0495,
859
+ "step": 110
860
+ },
861
+ {
862
+ "epoch": 4.345679012345679,
863
+ "eval_loss": 0.15176202356815338,
864
+ "eval_runtime": 19.2575,
865
+ "eval_samples_per_second": 2.337,
866
+ "eval_steps_per_second": 2.337,
867
+ "step": 110
868
+ },
869
+ {
870
+ "epoch": 4.385185185185185,
871
+ "grad_norm": 0.5771532654762268,
872
+ "learning_rate": 3.8060233744356633e-06,
873
+ "loss": 0.0736,
874
+ "step": 111
875
+ },
876
+ {
877
+ "epoch": 4.424691358024692,
878
+ "grad_norm": 0.5677194595336914,
879
+ "learning_rate": 3.2875529852700147e-06,
880
+ "loss": 0.0663,
881
+ "step": 112
882
+ },
883
+ {
884
+ "epoch": 4.4641975308641975,
885
+ "grad_norm": 0.42305389046669006,
886
+ "learning_rate": 2.8058334845816213e-06,
887
+ "loss": 0.0411,
888
+ "step": 113
889
+ },
890
+ {
891
+ "epoch": 4.503703703703704,
892
+ "grad_norm": 0.5345879197120667,
893
+ "learning_rate": 2.361243863855184e-06,
894
+ "loss": 0.059,
895
+ "step": 114
896
+ },
897
+ {
898
+ "epoch": 4.54320987654321,
899
+ "grad_norm": 0.4867326617240906,
900
+ "learning_rate": 1.9541339027450256e-06,
901
+ "loss": 0.0513,
902
+ "step": 115
903
+ },
904
+ {
905
+ "epoch": 4.582716049382716,
906
+ "grad_norm": 0.5048791766166687,
907
+ "learning_rate": 1.584823893886933e-06,
908
+ "loss": 0.0545,
909
+ "step": 116
910
+ },
911
+ {
912
+ "epoch": 4.622222222222222,
913
+ "grad_norm": 0.5188657641410828,
914
+ "learning_rate": 1.2536043909088191e-06,
915
+ "loss": 0.0515,
916
+ "step": 117
917
+ },
918
+ {
919
+ "epoch": 4.661728395061728,
920
+ "grad_norm": 0.49358126521110535,
921
+ "learning_rate": 9.607359798384785e-07,
922
+ "loss": 0.0576,
923
+ "step": 118
924
+ },
925
+ {
926
+ "epoch": 4.701234567901235,
927
+ "grad_norm": 0.5929109454154968,
928
+ "learning_rate": 7.064490740882057e-07,
929
+ "loss": 0.0709,
930
+ "step": 119
931
+ },
932
+ {
933
+ "epoch": 4.7407407407407405,
934
+ "grad_norm": 0.6397348642349243,
935
+ "learning_rate": 4.909437331777179e-07,
936
+ "loss": 0.0504,
937
+ "step": 120
938
+ },
939
+ {
940
+ "epoch": 4.7407407407407405,
941
+ "eval_loss": 0.15273258090019226,
942
+ "eval_runtime": 19.2601,
943
+ "eval_samples_per_second": 2.336,
944
+ "eval_steps_per_second": 2.336,
945
+ "step": 120
946
+ },
947
+ {
948
+ "epoch": 4.780246913580247,
949
+ "grad_norm": 0.6079633235931396,
950
+ "learning_rate": 3.143895053378698e-07,
951
+ "loss": 0.0519,
952
+ "step": 121
953
+ },
954
+ {
955
+ "epoch": 4.8197530864197535,
956
+ "grad_norm": 0.5618969798088074,
957
+ "learning_rate": 1.7692529411904578e-07,
958
+ "loss": 0.0844,
959
+ "step": 122
960
+ },
961
+ {
962
+ "epoch": 4.859259259259259,
963
+ "grad_norm": 0.738714337348938,
964
+ "learning_rate": 7.865924910916977e-08,
965
+ "loss": 0.0745,
966
+ "step": 123
967
+ },
968
+ {
969
+ "epoch": 4.898765432098766,
970
+ "grad_norm": 0.6914772391319275,
971
+ "learning_rate": 1.9668680847356735e-08,
972
+ "loss": 0.0692,
973
+ "step": 124
974
+ },
975
+ {
976
+ "epoch": 4.938271604938271,
977
+ "grad_norm": 0.6855793595314026,
978
+ "learning_rate": 0.0,
979
+ "loss": 0.0743,
980
+ "step": 125
981
+ },
982
+ {
983
+ "epoch": 4.938271604938271,
984
+ "step": 125,
985
+ "total_flos": 1.2628188898630042e+17,
986
+ "train_loss": 0.19368722805380822,
987
+ "train_runtime": 3339.1879,
988
+ "train_samples_per_second": 0.606,
989
+ "train_steps_per_second": 0.037
990
+ }
991
+ ],
992
+ "logging_steps": 1,
993
+ "max_steps": 125,
994
+ "num_input_tokens_seen": 0,
995
+ "num_train_epochs": 5,
996
+ "save_steps": 10,
997
+ "stateful_callbacks": {
998
+ "TrainerControl": {
999
+ "args": {
1000
+ "should_epoch_stop": false,
1001
+ "should_evaluate": false,
1002
+ "should_log": false,
1003
+ "should_save": true,
1004
+ "should_training_stop": true
1005
+ },
1006
+ "attributes": {}
1007
+ }
1008
+ },
1009
+ "total_flos": 1.2628188898630042e+17,
1010
+ "train_batch_size": 1,
1011
+ "trial_name": null,
1012
+ "trial_params": null
1013
+ }
training_eval_loss.png ADDED
training_loss.png ADDED