Hugging Face's logo

eaddario
/
Watt-Tool-8B-GGUF

Text Generation
GGUF
English
quant
experimental
conversational
Model card Files Community
2
eaddario commited on
Commit
a5d4592
·
verified ·
1 Parent(s): 408a83c

Regenerate Perplexity, KLD, ARC, HellaSwag, MMLU, Truthful QA and WinoGrande scores

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. scores/Watt-Tool-8B-F16.arc +10 -760
  2. scores/Watt-Tool-8B-F16.hsw +10 -758
  3. scores/Watt-Tool-8B-F16.mmlu +10 -760
  4. scores/Watt-Tool-8B-F16.tqa +10 -760
  5. scores/Watt-Tool-8B-F16.wng +9 -758
  6. scores/Watt-Tool-8B-Q4_K_M-naive.arc +13 -0
  7. scores/Watt-Tool-8B-Q4_K_M-naive.hsw +12 -0
  8. scores/Watt-Tool-8B-Q4_K_M-naive.mmlu +13 -0
  9. scores/Watt-Tool-8B-Q4_K_M-naive.ppx +37 -0
  10. scores/Watt-Tool-8B-Q4_K_M-naive.tqa +13 -0
  11. scores/Watt-Tool-8B-Q4_K_M-naive.wng +11 -0
  12. scores/Watt-Tool-8B-iq3_m.arc +10 -760
  13. scores/Watt-Tool-8B-iq3_m.hsw +10 -758
  14. scores/Watt-Tool-8B-iq3_m.mmlu +10 -760
  15. scores/Watt-Tool-8B-iq3_m.ppx +31 -31
  16. scores/Watt-Tool-8B-iq3_m.tqa +10 -760
  17. scores/Watt-Tool-8B-iq3_m.wng +9 -758
  18. scores/Watt-Tool-8B-iq3_s.arc +10 -760
  19. scores/Watt-Tool-8B-iq3_s.hsw +10 -758
  20. scores/Watt-Tool-8B-iq3_s.mmlu +10 -760
  21. scores/Watt-Tool-8B-iq3_s.ppx +32 -32
  22. scores/Watt-Tool-8B-iq3_s.tqa +10 -760
  23. scores/Watt-Tool-8B-iq3_s.wng +9 -758
  24. scores/Watt-Tool-8B-iq4_nl.arc +10 -760
  25. scores/Watt-Tool-8B-iq4_nl.hsw +10 -758
  26. scores/Watt-Tool-8B-iq4_nl.mmlu +10 -760
  27. scores/Watt-Tool-8B-iq4_nl.ppx +32 -32
  28. scores/Watt-Tool-8B-iq4_nl.tqa +10 -760
  29. scores/Watt-Tool-8B-iq4_nl.wng +9 -758
  30. scores/Watt-Tool-8B-q3_k_l.arc +10 -760
  31. scores/Watt-Tool-8B-q3_k_l.hsw +10 -758
  32. scores/Watt-Tool-8B-q3_k_l.mmlu +10 -760
  33. scores/Watt-Tool-8B-q3_k_l.ppx +32 -32
  34. scores/Watt-Tool-8B-q3_k_l.tqa +10 -760
  35. scores/Watt-Tool-8B-q3_k_l.wng +9 -758
  36. scores/Watt-Tool-8B-q3_k_m.arc +10 -760
  37. scores/Watt-Tool-8B-q3_k_m.hsw +10 -758
  38. scores/Watt-Tool-8B-q3_k_m.mmlu +10 -760
  39. scores/Watt-Tool-8B-q3_k_m.ppx +32 -32
  40. scores/Watt-Tool-8B-q3_k_m.tqa +10 -760
  41. scores/Watt-Tool-8B-q3_k_m.wng +9 -758
  42. scores/Watt-Tool-8B-q3_k_s.arc +10 -760
  43. scores/Watt-Tool-8B-q3_k_s.hsw +10 -758
  44. scores/Watt-Tool-8B-q3_k_s.mmlu +10 -760
  45. scores/Watt-Tool-8B-q3_k_s.ppx +32 -32
  46. scores/Watt-Tool-8B-q3_k_s.tqa +10 -760
  47. scores/Watt-Tool-8B-q3_k_s.wng +9 -758
  48. scores/Watt-Tool-8B-q4_k_m-naive.arc +0 -763
  49. scores/Watt-Tool-8B-q4_k_m-naive.hsw +0 -760
  50. scores/Watt-Tool-8B-q4_k_m-naive.mmlu +0 -763
scores/Watt-Tool-8B-F16.arc CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 869 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 869 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 100.00000000
13
- 3 100.00000000
14
- 4 75.00000000
15
- 5 80.00000000
16
- 6 83.33333333
17
- 7 71.42857143
18
- 8 75.00000000
19
- 9 66.66666667
20
- 10 70.00000000
21
- 11 63.63636364
22
- 12 66.66666667
23
- 13 69.23076923
24
- 14 71.42857143
25
- 15 73.33333333
26
- 16 75.00000000
27
- 17 76.47058824
28
- 18 72.22222222
29
- 19 68.42105263
30
- 20 65.00000000
31
- 21 66.66666667
32
- 22 68.18181818
33
- 23 65.21739130
34
- 24 66.66666667
35
- 25 64.00000000
36
- 26 61.53846154
37
- 27 62.96296296
38
- 28 60.71428571
39
- 29 62.06896552
40
- 30 63.33333333
41
- 31 61.29032258
42
- 32 62.50000000
43
- 33 60.60606061
44
- 34 58.82352941
45
- 35 60.00000000
46
- 36 58.33333333
47
- 37 59.45945946
48
- 38 60.52631579
49
- 39 61.53846154
50
- 40 62.50000000
51
- 41 63.41463415
52
- 42 61.90476190
53
- 43 60.46511628
54
- 44 59.09090909
55
- 45 60.00000000
56
- 46 60.86956522
57
- 47 61.70212766
58
- 48 62.50000000
59
- 49 63.26530612
60
- 50 62.00000000
61
- 51 60.78431373
62
- 52 61.53846154
63
- 53 60.37735849
64
- 54 59.25925926
65
- 55 58.18181818
66
- 56 58.92857143
67
- 57 59.64912281
68
- 58 60.34482759
69
- 59 61.01694915
70
- 60 61.66666667
71
- 61 62.29508197
72
- 62 61.29032258
73
- 63 61.90476190
74
- 64 62.50000000
75
- 65 61.53846154
76
- 66 62.12121212
77
- 67 62.68656716
78
- 68 63.23529412
79
- 69 63.76811594
80
- 70 62.85714286
81
- 71 61.97183099
82
- 72 61.11111111
83
- 73 61.64383562
84
- 74 62.16216216
85
- 75 62.66666667
86
- 76 61.84210526
87
- 77 62.33766234
88
- 78 62.82051282
89
- 79 62.02531646
90
- 80 62.50000000
91
- 81 61.72839506
92
- 82 62.19512195
93
- 83 61.44578313
94
- 84 61.90476190
95
- 85 62.35294118
96
- 86 62.79069767
97
- 87 63.21839080
98
- 88 63.63636364
99
- 89 64.04494382
100
- 90 64.44444444
101
- 91 64.83516484
102
- 92 65.21739130
103
- 93 64.51612903
104
- 94 63.82978723
105
- 95 64.21052632
106
- 96 64.58333333
107
- 97 63.91752577
108
- 98 64.28571429
109
- 99 64.64646465
110
- 100 65.00000000
111
- 101 64.35643564
112
- 102 64.70588235
113
- 103 64.07766990
114
- 104 63.46153846
115
- 105 63.80952381
116
- 106 63.20754717
117
- 107 63.55140187
118
- 108 62.96296296
119
- 109 62.38532110
120
- 110 62.72727273
121
- 111 63.06306306
122
- 112 63.39285714
123
- 113 63.71681416
124
- 114 64.03508772
125
- 115 64.34782609
126
- 116 64.65517241
127
- 117 64.10256410
128
- 118 64.40677966
129
- 119 63.86554622
130
- 120 64.16666667
131
- 121 63.63636364
132
- 122 63.93442623
133
- 123 64.22764228
134
- 124 63.70967742
135
- 125 64.00000000
136
- 126 64.28571429
137
- 127 64.56692913
138
- 128 64.84375000
139
- 129 64.34108527
140
- 130 64.61538462
141
- 131 64.88549618
142
- 132 65.15151515
143
- 133 65.41353383
144
- 134 65.67164179
145
- 135 65.18518519
146
- 136 64.70588235
147
- 137 64.96350365
148
- 138 65.21739130
149
- 139 65.46762590
150
- 140 65.71428571
151
- 141 65.24822695
152
- 142 65.49295775
153
- 143 65.73426573
154
- 144 65.97222222
155
- 145 66.20689655
156
- 146 65.75342466
157
- 147 65.98639456
158
- 148 66.21621622
159
- 149 66.44295302
160
- 150 66.66666667
161
- 151 66.88741722
162
- 152 67.10526316
163
- 153 67.32026144
164
- 154 66.88311688
165
- 155 67.09677419
166
- 156 67.30769231
167
- 157 67.51592357
168
- 158 67.08860759
169
- 159 67.29559748
170
- 160 66.87500000
171
- 161 67.08074534
172
- 162 67.28395062
173
- 163 66.87116564
174
- 164 67.07317073
175
- 165 66.66666667
176
- 166 66.86746988
177
- 167 66.46706587
178
- 168 66.66666667
179
- 169 66.27218935
180
- 170 66.47058824
181
- 171 66.66666667
182
- 172 66.86046512
183
- 173 67.05202312
184
- 174 67.24137931
185
- 175 66.85714286
186
- 176 67.04545455
187
- 177 67.23163842
188
- 178 67.41573034
189
- 179 67.59776536
190
- 180 67.77777778
191
- 181 67.40331492
192
- 182 67.58241758
193
- 183 67.21311475
194
- 184 67.39130435
195
- 185 67.56756757
196
- 186 67.74193548
197
- 187 67.37967914
198
- 188 67.02127660
199
- 189 66.66666667
200
- 190 66.31578947
201
- 191 66.49214660
202
- 192 66.66666667
203
- 193 66.32124352
204
- 194 66.49484536
205
- 195 66.66666667
206
- 196 66.83673469
207
- 197 67.00507614
208
- 198 66.66666667
209
- 199 66.33165829
210
- 200 66.50000000
211
- 201 66.66666667
212
- 202 66.83168317
213
- 203 66.99507389
214
- 204 67.15686275
215
- 205 66.82926829
216
- 206 66.50485437
217
- 207 66.66666667
218
- 208 66.82692308
219
- 209 66.98564593
220
- 210 67.14285714
221
- 211 67.29857820
222
- 212 67.45283019
223
- 213 67.60563380
224
- 214 67.75700935
225
- 215 67.90697674
226
- 216 67.59259259
227
- 217 67.74193548
228
- 218 67.43119266
229
- 219 67.57990868
230
- 220 67.27272727
231
- 221 67.42081448
232
- 222 67.56756757
233
- 223 67.26457399
234
- 224 67.41071429
235
- 225 67.11111111
236
- 226 67.25663717
237
- 227 67.40088106
238
- 228 67.54385965
239
- 229 67.68558952
240
- 230 67.82608696
241
- 231 67.53246753
242
- 232 67.24137931
243
- 233 67.38197425
244
- 234 67.52136752
245
- 235 67.23404255
246
- 236 66.94915254
247
- 237 66.66666667
248
- 238 66.80672269
249
- 239 66.94560669
250
- 240 67.08333333
251
- 241 67.21991701
252
- 242 67.35537190
253
- 243 67.48971193
254
- 244 67.21311475
255
- 245 67.34693878
256
- 246 67.47967480
257
- 247 67.61133603
258
- 248 67.74193548
259
- 249 67.46987952
260
- 250 67.60000000
261
- 251 67.72908367
262
- 252 67.85714286
263
- 253 67.98418972
264
- 254 67.71653543
265
- 255 67.84313725
266
- 256 67.57812500
267
- 257 67.70428016
268
- 258 67.82945736
269
- 259 67.56756757
270
- 260 67.69230769
271
- 261 67.81609195
272
- 262 67.93893130
273
- 263 68.06083650
274
- 264 68.18181818
275
- 265 68.30188679
276
- 266 68.04511278
277
- 267 68.16479401
278
- 268 68.28358209
279
- 269 68.40148699
280
- 270 68.51851852
281
- 271 68.63468635
282
- 272 68.38235294
283
- 273 68.13186813
284
- 274 67.88321168
285
- 275 67.63636364
286
- 276 67.75362319
287
- 277 67.87003610
288
- 278 67.62589928
289
- 279 67.74193548
290
- 280 67.85714286
291
- 281 67.97153025
292
- 282 68.08510638
293
- 283 67.84452297
294
- 284 67.95774648
295
- 285 68.07017544
296
- 286 68.18181818
297
- 287 68.29268293
298
- 288 68.40277778
299
- 289 68.16608997
300
- 290 68.27586207
301
- 291 68.38487973
302
- 292 68.49315068
303
- 293 68.60068259
304
- 294 68.36734694
305
- 295 68.47457627
306
- 296 68.58108108
307
- 297 68.68686869
308
- 298 68.79194631
309
- 299 68.89632107
310
- 300 69.00000000
311
- 301 68.77076412
312
- 302 68.87417219
313
- 303 68.97689769
314
- 304 69.07894737
315
- 305 69.18032787
316
- 306 68.95424837
317
- 307 69.05537459
318
- 308 69.15584416
319
- 309 69.25566343
320
- 310 69.35483871
321
- 311 69.45337621
322
- 312 69.23076923
323
- 313 69.32907348
324
- 314 69.10828025
325
- 315 69.20634921
326
- 316 69.30379747
327
- 317 69.40063091
328
- 318 69.18238994
329
- 319 68.96551724
330
- 320 68.75000000
331
- 321 68.84735202
332
- 322 68.94409938
333
- 323 69.04024768
334
- 324 68.82716049
335
- 325 68.92307692
336
- 326 69.01840491
337
- 327 68.80733945
338
- 328 68.90243902
339
- 329 68.99696049
340
- 330 68.78787879
341
- 331 68.88217523
342
- 332 68.97590361
343
- 333 69.06906907
344
- 334 69.16167665
345
- 335 68.95522388
346
- 336 69.04761905
347
- 337 69.13946588
348
- 338 68.93491124
349
- 339 69.02654867
350
- 340 69.11764706
351
- 341 69.20821114
352
- 342 69.29824561
353
- 343 69.38775510
354
- 344 69.47674419
355
- 345 69.27536232
356
- 346 69.36416185
357
- 347 69.16426513
358
- 348 68.96551724
359
- 349 69.05444126
360
- 350 68.85714286
361
- 351 68.94586895
362
- 352 69.03409091
363
- 353 69.12181303
364
- 354 68.92655367
365
- 355 68.73239437
366
- 356 68.82022472
367
- 357 68.90756303
368
- 358 68.99441341
369
- 359 68.80222841
370
- 360 68.88888889
371
- 361 68.97506925
372
- 362 69.06077348
373
- 363 68.87052342
374
- 364 68.95604396
375
- 365 69.04109589
376
- 366 68.85245902
377
- 367 68.66485014
378
- 368 68.75000000
379
- 369 68.83468835
380
- 370 68.91891892
381
- 371 69.00269542
382
- 372 69.08602151
383
- 373 69.16890080
384
- 374 69.25133690
385
- 375 69.06666667
386
- 376 69.14893617
387
- 377 68.96551724
388
- 378 69.04761905
389
- 379 68.86543536
390
- 380 68.94736842
391
- 381 69.02887139
392
- 382 68.84816754
393
- 383 68.66840731
394
- 384 68.75000000
395
- 385 68.83116883
396
- 386 68.91191710
397
- 387 68.99224806
398
- 388 68.81443299
399
- 389 68.89460154
400
- 390 68.71794872
401
- 391 68.79795396
402
- 392 68.87755102
403
- 393 68.95674300
404
- 394 69.03553299
405
- 395 69.11392405
406
- 396 68.93939394
407
- 397 69.01763224
408
- 398 68.84422111
409
- 399 68.92230576
410
- 400 68.75000000
411
- 401 68.82793017
412
- 402 68.90547264
413
- 403 68.98263027
414
- 404 69.05940594
415
- 405 69.13580247
416
- 406 69.21182266
417
- 407 69.28746929
418
- 408 69.36274510
419
- 409 69.19315403
420
- 410 69.26829268
421
- 411 69.34306569
422
- 412 69.41747573
423
- 413 69.49152542
424
- 414 69.56521739
425
- 415 69.39759036
426
- 416 69.47115385
427
- 417 69.30455635
428
- 418 69.37799043
429
- 419 69.21241050
430
- 420 69.28571429
431
- 421 69.12114014
432
- 422 69.19431280
433
- 423 69.26713948
434
- 424 69.10377358
435
- 425 69.17647059
436
- 426 69.01408451
437
- 427 69.08665105
438
- 428 68.92523364
439
- 429 68.99766900
440
- 430 68.83720930
441
- 431 68.90951276
442
- 432 68.98148148
443
- 433 68.82217090
444
- 434 68.66359447
445
- 435 68.73563218
446
- 436 68.80733945
447
- 437 68.87871854
448
- 438 68.72146119
449
- 439 68.79271071
450
- 440 68.63636364
451
- 441 68.48072562
452
- 442 68.32579186
453
- 443 68.17155756
454
- 444 68.01801802
455
- 445 68.08988764
456
- 446 68.16143498
457
- 447 68.23266219
458
- 448 68.30357143
459
- 449 68.37416481
460
- 450 68.22222222
461
- 451 68.29268293
462
- 452 68.36283186
463
- 453 68.43267108
464
- 454 68.50220264
465
- 455 68.35164835
466
- 456 68.42105263
467
- 457 68.49015317
468
- 458 68.55895197
469
- 459 68.62745098
470
- 460 68.69565217
471
- 461 68.76355748
472
- 462 68.83116883
473
- 463 68.68250540
474
- 464 68.53448276
475
- 465 68.60215054
476
- 466 68.66952790
477
- 467 68.73661670
478
- 468 68.80341880
479
- 469 68.65671642
480
- 470 68.72340426
481
- 471 68.78980892
482
- 472 68.85593220
483
- 473 68.71035941
484
- 474 68.77637131
485
- 475 68.63157895
486
- 476 68.69747899
487
- 477 68.76310273
488
- 478 68.82845188
489
- 479 68.89352818
490
- 480 68.95833333
491
- 481 69.02286902
492
- 482 69.08713693
493
- 483 69.15113872
494
- 484 69.21487603
495
- 485 69.27835052
496
- 486 69.34156379
497
- 487 69.19917864
498
- 488 69.26229508
499
- 489 69.32515337
500
- 490 69.18367347
501
- 491 69.04276986
502
- 492 69.10569106
503
- 493 68.96551724
504
- 494 69.02834008
505
- 495 68.88888889
506
- 496 68.95161290
507
- 497 69.01408451
508
- 498 69.07630522
509
- 499 69.13827655
510
- 500 69.20000000
511
- 501 69.06187625
512
- 502 69.12350598
513
- 503 69.18489066
514
- 504 69.24603175
515
- 505 69.30693069
516
- 506 69.16996047
517
- 507 69.03353057
518
- 508 69.09448819
519
- 509 68.95874263
520
- 510 69.01960784
521
- 511 68.88454012
522
- 512 68.94531250
523
- 513 68.81091618
524
- 514 68.87159533
525
- 515 68.93203883
526
- 516 68.99224806
527
- 517 69.05222437
528
- 518 69.11196911
529
- 519 69.17148362
530
- 520 69.03846154
531
- 521 69.09788868
532
- 522 69.15708812
533
- 523 69.02485660
534
- 524 68.89312977
535
- 525 68.76190476
536
- 526 68.63117871
537
- 527 68.50094877
538
- 528 68.37121212
539
- 529 68.24196597
540
- 530 68.30188679
541
- 531 68.17325800
542
- 532 68.23308271
543
- 533 68.29268293
544
- 534 68.16479401
545
- 535 68.03738318
546
- 536 68.09701493
547
- 537 68.15642458
548
- 538 68.21561338
549
- 539 68.27458256
550
- 540 68.14814815
551
- 541 68.20702403
552
- 542 68.26568266
553
- 543 68.13996317
554
- 544 68.19852941
555
- 545 68.25688073
556
- 546 68.13186813
557
- 547 68.00731261
558
- 548 68.06569343
559
- 549 68.12386157
560
- 550 68.18181818
561
- 551 68.05807623
562
- 552 67.93478261
563
- 553 67.99276673
564
- 554 68.05054152
565
- 555 68.10810811
566
- 556 67.98561151
567
- 557 68.04308797
568
- 558 67.92114695
569
- 559 67.97853309
570
- 560 68.03571429
571
- 561 68.09269162
572
- 562 68.14946619
573
- 563 68.20603908
574
- 564 68.26241135
575
- 565 68.14159292
576
- 566 68.19787986
577
- 567 68.07760141
578
- 568 68.13380282
579
- 569 68.01405975
580
- 570 67.89473684
581
- 571 67.95096322
582
- 572 67.83216783
583
- 573 67.88830716
584
- 574 67.94425087
585
- 575 68.00000000
586
- 576 68.05555556
587
- 577 68.11091854
588
- 578 68.16608997
589
- 579 68.22107081
590
- 580 68.27586207
591
- 581 68.15834768
592
- 582 68.04123711
593
- 583 68.09605489
594
- 584 68.15068493
595
- 585 68.20512821
596
- 586 68.08873720
597
- 587 68.14310051
598
- 588 68.19727891
599
- 589 68.25127334
600
- 590 68.30508475
601
- 591 68.35871404
602
- 592 68.24324324
603
- 593 68.12816189
604
- 594 68.18181818
605
- 595 68.23529412
606
- 596 68.12080537
607
- 597 68.00670017
608
- 598 67.89297659
609
- 599 67.94657763
610
- 600 68.00000000
611
- 601 68.05324459
612
- 602 67.94019934
613
- 603 67.82752902
614
- 604 67.88079470
615
- 605 67.76859504
616
- 606 67.82178218
617
- 607 67.87479407
618
- 608 67.92763158
619
- 609 67.81609195
620
- 610 67.86885246
621
- 611 67.75777414
622
- 612 67.81045752
623
- 613 67.69983687
624
- 614 67.75244300
625
- 615 67.64227642
626
- 616 67.53246753
627
- 617 67.58508914
628
- 618 67.63754045
629
- 619 67.52827141
630
- 620 67.58064516
631
- 621 67.63285024
632
- 622 67.52411576
633
- 623 67.57624398
634
- 624 67.46794872
635
- 625 67.36000000
636
- 626 67.25239617
637
- 627 67.30462520
638
- 628 67.35668790
639
- 629 67.24960254
640
- 630 67.30158730
641
- 631 67.35340729
642
- 632 67.24683544
643
- 633 67.14060032
644
- 634 67.19242902
645
- 635 67.24409449
646
- 636 67.29559748
647
- 637 67.18995290
648
- 638 67.24137931
649
- 639 67.29264476
650
- 640 67.34375000
651
- 641 67.39469579
652
- 642 67.28971963
653
- 643 67.18506998
654
- 644 67.23602484
655
- 645 67.13178295
656
- 646 67.18266254
657
- 647 67.07882535
658
- 648 66.97530864
659
- 649 66.87211094
660
- 650 66.92307692
661
- 651 66.97388633
662
- 652 67.02453988
663
- 653 66.92189893
664
- 654 66.81957187
665
- 655 66.71755725
666
- 656 66.61585366
667
- 657 66.51445967
668
- 658 66.56534954
669
- 659 66.46433991
670
- 660 66.51515152
671
- 661 66.41452345
672
- 662 66.46525680
673
- 663 66.51583710
674
- 664 66.56626506
675
- 665 66.61654135
676
- 666 66.51651652
677
- 667 66.56671664
678
- 668 66.46706587
679
- 669 66.36771300
680
- 670 66.41791045
681
- 671 66.31892697
682
- 672 66.36904762
683
- 673 66.27043091
684
- 674 66.32047478
685
- 675 66.22222222
686
- 676 66.12426036
687
- 677 66.17429838
688
- 678 66.22418879
689
- 679 66.27393225
690
- 680 66.32352941
691
- 681 66.22613803
692
- 682 66.12903226
693
- 683 66.03221083
694
- 684 66.08187135
695
- 685 66.13138686
696
- 686 66.03498542
697
- 687 65.93886463
698
- 688 65.98837209
699
- 689 65.89259797
700
- 690 65.94202899
701
- 691 65.99131693
702
- 692 66.04046243
703
- 693 66.08946609
704
- 694 66.13832853
705
- 695 66.04316547
706
- 696 65.94827586
707
- 697 65.85365854
708
- 698 65.75931232
709
- 699 65.80829757
710
- 700 65.85714286
711
- 701 65.90584879
712
- 702 65.95441595
713
- 703 66.00284495
714
- 704 66.05113636
715
- 705 66.09929078
716
- 706 66.14730878
717
- 707 66.19519095
718
- 708 66.24293785
719
- 709 66.29055007
720
- 710 66.33802817
721
- 711 66.38537271
722
- 712 66.43258427
723
- 713 66.47966339
724
- 714 66.38655462
725
- 715 66.29370629
726
- 716 66.34078212
727
- 717 66.24825662
728
- 718 66.15598886
729
- 719 66.06397775
730
- 720 66.11111111
731
- 721 66.01941748
732
- 722 66.06648199
733
- 723 66.11341632
734
- 724 66.02209945
735
- 725 66.06896552
736
- 726 65.97796143
737
- 727 65.88720770
738
- 728 65.93406593
739
- 729 65.98079561
740
- 730 66.02739726
741
- 731 66.07387141
742
- 732 66.12021858
743
- 733 66.16643929
744
- 734 66.21253406
745
- 735 66.25850340
746
- 736 66.30434783
747
- 737 66.21438263
748
- 738 66.26016260
749
- 739 66.17050068
750
- 740 66.08108108
751
- 741 66.12685560
752
- 742 66.03773585
753
- 743 66.08344549
754
- 744 65.99462366
755
- 745 66.04026846
756
- 746 66.08579088
757
- 747 65.99732262
758
- 748 65.90909091
759
- 749 65.82109479
760
- 750 65.86666667
761
 
762
- Final result: 65.8667 +/- 1.7325
763
- Random chance: 25.0083 +/- 1.5824
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 36 key-value pairs and 292 tensors from ./Watt-Tool-8B-F16.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 65.2870 +/- 1.7406
6
+ Random chance: 25.0334 +/- 1.5840
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 7009.00 ms
10
+ llama_perf_context_print: prompt eval time = 157202.97 ms / 36703 tokens ( 4.28 ms per token, 233.48 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 159569.20 ms / 36704 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-F16.hsw CHANGED
@@ -1,760 +1,12 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- hellaswag_score : loaded 10042 tasks from prompt.
6
- ================================= is_spm = 0
7
- hellaswag_score : selecting 750 randomized tasks.
8
- hellaswag_score : calculating hellaswag score over selected tasks.
9
 
10
- task acc_norm
11
- 1 0.00000000
12
- 2 50.00000000
13
- 3 66.66666667
14
- 4 75.00000000
15
- 5 80.00000000
16
- 6 83.33333333
17
- 7 85.71428571
18
- 8 87.50000000
19
- 9 88.88888889
20
- 10 90.00000000
21
- 11 90.90909091
22
- 12 83.33333333
23
- 13 84.61538462
24
- 14 85.71428571
25
- 15 86.66666667
26
- 16 87.50000000
27
- 17 82.35294118
28
- 18 83.33333333
29
- 19 78.94736842
30
- 20 80.00000000
31
- 21 80.95238095
32
- 22 81.81818182
33
- 23 82.60869565
34
- 24 79.16666667
35
- 25 76.00000000
36
- 26 73.07692308
37
- 27 70.37037037
38
- 28 71.42857143
39
- 29 72.41379310
40
- 30 70.00000000
41
- 31 70.96774194
42
- 32 68.75000000
43
- 33 69.69696970
44
- 34 70.58823529
45
- 35 71.42857143
46
- 36 72.22222222
47
- 37 70.27027027
48
- 38 71.05263158
49
- 39 71.79487179
50
- 40 70.00000000
51
- 41 70.73170732
52
- 42 69.04761905
53
- 43 69.76744186
54
- 44 70.45454545
55
- 45 68.88888889
56
- 46 69.56521739
57
- 47 70.21276596
58
- 48 70.83333333
59
- 49 71.42857143
60
- 50 72.00000000
61
- 51 72.54901961
62
- 52 71.15384615
63
- 53 69.81132075
64
- 54 68.51851852
65
- 55 69.09090909
66
- 56 67.85714286
67
- 57 68.42105263
68
- 58 68.96551724
69
- 59 69.49152542
70
- 60 70.00000000
71
- 61 70.49180328
72
- 62 69.35483871
73
- 63 69.84126984
74
- 64 70.31250000
75
- 65 70.76923077
76
- 66 69.69696970
77
- 67 70.14925373
78
- 68 70.58823529
79
- 69 71.01449275
80
- 70 71.42857143
81
- 71 70.42253521
82
- 72 70.83333333
83
- 73 71.23287671
84
- 74 71.62162162
85
- 75 70.66666667
86
- 76 71.05263158
87
- 77 71.42857143
88
- 78 71.79487179
89
- 79 72.15189873
90
- 80 72.50000000
91
- 81 72.83950617
92
- 82 73.17073171
93
- 83 73.49397590
94
- 84 73.80952381
95
- 85 74.11764706
96
- 86 74.41860465
97
- 87 74.71264368
98
- 88 75.00000000
99
- 89 74.15730337
100
- 90 74.44444444
101
- 91 74.72527473
102
- 92 73.91304348
103
- 93 74.19354839
104
- 94 74.46808511
105
- 95 74.73684211
106
- 96 75.00000000
107
- 97 75.25773196
108
- 98 75.51020408
109
- 99 75.75757576
110
- 100 75.00000000
111
- 101 75.24752475
112
- 102 74.50980392
113
- 103 73.78640777
114
- 104 74.03846154
115
- 105 74.28571429
116
- 106 74.52830189
117
- 107 74.76635514
118
- 108 75.00000000
119
- 109 74.31192661
120
- 110 74.54545455
121
- 111 73.87387387
122
- 112 74.10714286
123
- 113 74.33628319
124
- 114 74.56140351
125
- 115 74.78260870
126
- 116 74.13793103
127
- 117 73.50427350
128
- 118 73.72881356
129
- 119 73.94957983
130
- 120 73.33333333
131
- 121 73.55371901
132
- 122 73.77049180
133
- 123 73.98373984
134
- 124 74.19354839
135
- 125 74.40000000
136
- 126 74.60317460
137
- 127 74.80314961
138
- 128 74.21875000
139
- 129 74.41860465
140
- 130 73.84615385
141
- 131 74.04580153
142
- 132 74.24242424
143
- 133 74.43609023
144
- 134 74.62686567
145
- 135 74.81481481
146
- 136 75.00000000
147
- 137 75.18248175
148
- 138 75.36231884
149
- 139 74.82014388
150
- 140 75.00000000
151
- 141 75.17730496
152
- 142 75.35211268
153
- 143 75.52447552
154
- 144 75.69444444
155
- 145 75.86206897
156
- 146 76.02739726
157
- 147 76.19047619
158
- 148 76.35135135
159
- 149 76.51006711
160
- 150 76.66666667
161
- 151 76.15894040
162
- 152 75.65789474
163
- 153 75.81699346
164
- 154 75.97402597
165
- 155 76.12903226
166
- 156 75.64102564
167
- 157 75.15923567
168
- 158 75.31645570
169
- 159 74.84276730
170
- 160 75.00000000
171
- 161 75.15527950
172
- 162 75.30864198
173
- 163 75.46012270
174
- 164 75.60975610
175
- 165 75.75757576
176
- 166 75.90361446
177
- 167 76.04790419
178
- 168 76.19047619
179
- 169 75.73964497
180
- 170 75.88235294
181
- 171 76.02339181
182
- 172 76.16279070
183
- 173 76.30057803
184
- 174 76.43678161
185
- 175 76.00000000
186
- 176 76.13636364
187
- 177 76.27118644
188
- 178 76.40449438
189
- 179 75.97765363
190
- 180 76.11111111
191
- 181 75.69060773
192
- 182 75.82417582
193
- 183 75.95628415
194
- 184 76.08695652
195
- 185 76.21621622
196
- 186 75.80645161
197
- 187 75.40106952
198
- 188 75.00000000
199
- 189 74.60317460
200
- 190 74.73684211
201
- 191 74.86910995
202
- 192 75.00000000
203
- 193 74.61139896
204
- 194 74.74226804
205
- 195 74.35897436
206
- 196 74.48979592
207
- 197 74.11167513
208
- 198 73.73737374
209
- 199 73.86934673
210
- 200 74.00000000
211
- 201 74.12935323
212
- 202 74.25742574
213
- 203 73.89162562
214
- 204 74.01960784
215
- 205 74.14634146
216
- 206 74.27184466
217
- 207 74.39613527
218
- 208 74.03846154
219
- 209 74.16267943
220
- 210 74.28571429
221
- 211 74.40758294
222
- 212 74.52830189
223
- 213 74.64788732
224
- 214 74.76635514
225
- 215 74.41860465
226
- 216 74.53703704
227
- 217 74.19354839
228
- 218 74.31192661
229
- 219 74.42922374
230
- 220 74.54545455
231
- 221 74.20814480
232
- 222 74.32432432
233
- 223 74.43946188
234
- 224 74.55357143
235
- 225 74.66666667
236
- 226 74.77876106
237
- 227 74.44933921
238
- 228 74.56140351
239
- 229 74.67248908
240
- 230 74.78260870
241
- 231 74.45887446
242
- 232 74.13793103
243
- 233 74.24892704
244
- 234 74.35897436
245
- 235 74.04255319
246
- 236 73.72881356
247
- 237 73.83966245
248
- 238 73.94957983
249
- 239 74.05857741
250
- 240 74.16666667
251
- 241 74.27385892
252
- 242 74.38016529
253
- 243 74.48559671
254
- 244 74.59016393
255
- 245 74.28571429
256
- 246 74.39024390
257
- 247 74.49392713
258
- 248 74.59677419
259
- 249 74.69879518
260
- 250 74.80000000
261
- 251 74.90039841
262
- 252 75.00000000
263
- 253 74.70355731
264
- 254 74.80314961
265
- 255 74.90196078
266
- 256 75.00000000
267
- 257 74.70817121
268
- 258 74.80620155
269
- 259 74.90347490
270
- 260 75.00000000
271
- 261 75.09578544
272
- 262 74.80916031
273
- 263 74.90494297
274
- 264 75.00000000
275
- 265 75.09433962
276
- 266 75.18796992
277
- 267 75.28089888
278
- 268 75.37313433
279
- 269 75.46468401
280
- 270 75.55555556
281
- 271 75.27675277
282
- 272 75.36764706
283
- 273 75.09157509
284
- 274 75.18248175
285
- 275 74.90909091
286
- 276 75.00000000
287
- 277 75.09025271
288
- 278 75.17985612
289
- 279 75.26881720
290
- 280 75.35714286
291
- 281 75.44483986
292
- 282 75.53191489
293
- 283 75.61837456
294
- 284 75.70422535
295
- 285 75.43859649
296
- 286 75.52447552
297
- 287 75.60975610
298
- 288 75.69444444
299
- 289 75.77854671
300
- 290 75.86206897
301
- 291 75.94501718
302
- 292 75.68493151
303
- 293 75.76791809
304
- 294 75.85034014
305
- 295 75.93220339
306
- 296 76.01351351
307
- 297 76.09427609
308
- 298 76.17449664
309
- 299 75.91973244
310
- 300 75.66666667
311
- 301 75.74750831
312
- 302 75.82781457
313
- 303 75.57755776
314
- 304 75.32894737
315
- 305 75.40983607
316
- 306 75.49019608
317
- 307 75.24429967
318
- 308 75.32467532
319
- 309 75.40453074
320
- 310 75.48387097
321
- 311 75.56270096
322
- 312 75.64102564
323
- 313 75.71884984
324
- 314 75.79617834
325
- 315 75.87301587
326
- 316 75.94936709
327
- 317 76.02523659
328
- 318 75.78616352
329
- 319 75.54858934
330
- 320 75.62500000
331
- 321 75.38940810
332
- 322 75.15527950
333
- 323 75.23219814
334
- 324 75.30864198
335
- 325 75.38461538
336
- 326 75.46012270
337
- 327 75.22935780
338
- 328 75.30487805
339
- 329 75.37993921
340
- 330 75.45454545
341
- 331 75.52870091
342
- 332 75.60240964
343
- 333 75.67567568
344
- 334 75.44910180
345
- 335 75.52238806
346
- 336 75.59523810
347
- 337 75.37091988
348
- 338 75.14792899
349
- 339 75.22123894
350
- 340 75.29411765
351
- 341 75.36656891
352
- 342 75.43859649
353
- 343 75.51020408
354
- 344 75.29069767
355
- 345 75.36231884
356
- 346 75.14450867
357
- 347 75.21613833
358
- 348 75.00000000
359
- 349 74.78510029
360
- 350 74.85714286
361
- 351 74.92877493
362
- 352 75.00000000
363
- 353 75.07082153
364
- 354 75.14124294
365
- 355 75.21126761
366
- 356 75.28089888
367
- 357 75.35014006
368
- 358 75.41899441
369
- 359 75.48746518
370
- 360 75.27777778
371
- 361 75.34626039
372
- 362 75.41436464
373
- 363 75.48209366
374
- 364 75.54945055
375
- 365 75.61643836
376
- 366 75.68306011
377
- 367 75.74931880
378
- 368 75.81521739
379
- 369 75.88075881
380
- 370 75.94594595
381
- 371 76.01078167
382
- 372 76.07526882
383
- 373 75.87131367
384
- 374 75.93582888
385
- 375 76.00000000
386
- 376 76.06382979
387
- 377 75.86206897
388
- 378 75.92592593
389
- 379 75.98944591
390
- 380 76.05263158
391
- 381 76.11548556
392
- 382 76.17801047
393
- 383 76.24020888
394
- 384 76.30208333
395
- 385 76.36363636
396
- 386 76.42487047
397
- 387 76.48578811
398
- 388 76.54639175
399
- 389 76.60668380
400
- 390 76.66666667
401
- 391 76.47058824
402
- 392 76.53061224
403
- 393 76.59033079
404
- 394 76.64974619
405
- 395 76.70886076
406
- 396 76.76767677
407
- 397 76.82619647
408
- 398 76.88442211
409
- 399 76.94235589
410
- 400 77.00000000
411
- 401 77.05735661
412
- 402 77.11442786
413
- 403 76.92307692
414
- 404 76.98019802
415
- 405 76.79012346
416
- 406 76.84729064
417
- 407 76.90417690
418
- 408 76.96078431
419
- 409 77.01711491
420
- 410 76.82926829
421
- 411 76.64233577
422
- 412 76.69902913
423
- 413 76.51331719
424
- 414 76.57004831
425
- 415 76.38554217
426
- 416 76.44230769
427
- 417 76.49880096
428
- 418 76.55502392
429
- 419 76.61097852
430
- 420 76.66666667
431
- 421 76.72209026
432
- 422 76.77725118
433
- 423 76.83215130
434
- 424 76.88679245
435
- 425 76.94117647
436
- 426 76.99530516
437
- 427 76.81498829
438
- 428 76.86915888
439
- 429 76.68997669
440
- 430 76.74418605
441
- 431 76.56612529
442
- 432 76.62037037
443
- 433 76.67436490
444
- 434 76.72811060
445
- 435 76.55172414
446
- 436 76.60550459
447
- 437 76.43020595
448
- 438 76.48401826
449
- 439 76.53758542
450
- 440 76.59090909
451
- 441 76.64399093
452
- 442 76.69683258
453
- 443 76.74943567
454
- 444 76.80180180
455
- 445 76.85393258
456
- 446 76.90582960
457
- 447 76.95749441
458
- 448 77.00892857
459
- 449 77.06013363
460
- 450 77.11111111
461
- 451 76.94013304
462
- 452 76.99115044
463
- 453 77.04194260
464
- 454 77.09251101
465
- 455 77.14285714
466
- 456 77.19298246
467
- 457 77.24288840
468
- 458 77.29257642
469
- 459 77.34204793
470
- 460 77.39130435
471
- 461 77.44034707
472
- 462 77.48917749
473
- 463 77.53779698
474
- 464 77.58620690
475
- 465 77.41935484
476
- 466 77.46781116
477
- 467 77.51605996
478
- 468 77.35042735
479
- 469 77.39872068
480
- 470 77.23404255
481
- 471 77.28237792
482
- 472 77.33050847
483
- 473 77.37843552
484
- 474 77.42616034
485
- 475 77.47368421
486
- 476 77.52100840
487
- 477 77.35849057
488
- 478 77.40585774
489
- 479 77.45302714
490
- 480 77.50000000
491
- 481 77.33887734
492
- 482 77.17842324
493
- 483 77.22567288
494
- 484 77.06611570
495
- 485 76.90721649
496
- 486 76.95473251
497
- 487 77.00205339
498
- 488 76.84426230
499
- 489 76.89161554
500
- 490 76.93877551
501
- 491 76.98574338
502
- 492 77.03252033
503
- 493 77.07910751
504
- 494 77.12550607
505
- 495 77.17171717
506
- 496 77.21774194
507
- 497 77.26358149
508
- 498 77.30923695
509
- 499 77.15430862
510
- 500 77.20000000
511
- 501 77.04590818
512
- 502 77.09163347
513
- 503 77.13717694
514
- 504 77.18253968
515
- 505 77.22772277
516
- 506 77.27272727
517
- 507 77.31755424
518
- 508 77.36220472
519
- 509 77.40667976
520
- 510 77.45098039
521
- 511 77.49510763
522
- 512 77.34375000
523
- 513 77.38791423
524
- 514 77.43190661
525
- 515 77.47572816
526
- 516 77.32558140
527
- 517 77.17601547
528
- 518 77.22007722
529
- 519 77.26396917
530
- 520 77.30769231
531
- 521 77.15930902
532
- 522 77.20306513
533
- 523 77.24665392
534
- 524 77.29007634
535
- 525 77.33333333
536
- 526 77.18631179
537
- 527 77.03984820
538
- 528 77.08333333
539
- 529 77.12665406
540
- 530 77.16981132
541
- 531 77.21280603
542
- 532 77.25563910
543
- 533 77.29831144
544
- 534 77.34082397
545
- 535 77.38317757
546
- 536 77.42537313
547
- 537 77.46741155
548
- 538 77.50929368
549
- 539 77.36549165
550
- 540 77.40740741
551
- 541 77.44916821
552
- 542 77.49077491
553
- 543 77.53222836
554
- 544 77.57352941
555
- 545 77.61467890
556
- 546 77.65567766
557
- 547 77.69652651
558
- 548 77.73722628
559
- 549 77.77777778
560
- 550 77.63636364
561
- 551 77.67695100
562
- 552 77.53623188
563
- 553 77.57685353
564
- 554 77.61732852
565
- 555 77.47747748
566
- 556 77.51798561
567
- 557 77.37881508
568
- 558 77.24014337
569
- 559 77.28085868
570
- 560 77.32142857
571
- 561 77.36185383
572
- 562 77.22419929
573
- 563 77.26465364
574
- 564 77.30496454
575
- 565 77.16814159
576
- 566 77.03180212
577
- 567 77.07231041
578
- 568 77.11267606
579
- 569 77.15289982
580
- 570 77.19298246
581
- 571 77.23292469
582
- 572 77.27272727
583
- 573 77.31239092
584
- 574 77.35191638
585
- 575 77.39130435
586
- 576 77.43055556
587
- 577 77.29636049
588
- 578 77.33564014
589
- 579 77.37478411
590
- 580 77.41379310
591
- 581 77.28055077
592
- 582 77.31958763
593
- 583 77.35849057
594
- 584 77.39726027
595
- 585 77.43589744
596
- 586 77.47440273
597
- 587 77.51277683
598
- 588 77.38095238
599
- 589 77.24957555
600
- 590 77.28813559
601
- 591 77.32656514
602
- 592 77.36486486
603
- 593 77.23440135
604
- 594 77.10437710
605
- 595 77.14285714
606
- 596 77.18120805
607
- 597 77.21943049
608
- 598 77.25752508
609
- 599 77.12854758
610
- 600 77.16666667
611
- 601 77.20465890
612
- 602 77.24252492
613
- 603 77.28026534
614
- 604 77.31788079
615
- 605 77.35537190
616
- 606 77.22772277
617
- 607 77.26523888
618
- 608 77.13815789
619
- 609 77.17569787
620
- 610 77.21311475
621
- 611 77.25040917
622
- 612 77.12418301
623
- 613 76.99836868
624
- 614 77.03583062
625
- 615 77.07317073
626
- 616 76.94805195
627
- 617 76.98541329
628
- 618 77.02265372
629
- 619 77.05977383
630
- 620 77.09677419
631
- 621 77.13365539
632
- 622 77.17041801
633
- 623 77.20706260
634
- 624 77.24358974
635
- 625 77.28000000
636
- 626 77.31629393
637
- 627 77.35247209
638
- 628 77.38853503
639
- 629 77.26550079
640
- 630 77.30158730
641
- 631 77.33755943
642
- 632 77.21518987
643
- 633 77.25118483
644
- 634 77.28706625
645
- 635 77.32283465
646
- 636 77.35849057
647
- 637 77.39403454
648
- 638 77.42946708
649
- 639 77.46478873
650
- 640 77.34375000
651
- 641 77.37909516
652
- 642 77.41433022
653
- 643 77.44945568
654
- 644 77.48447205
655
- 645 77.36434109
656
- 646 77.39938080
657
- 647 77.43431221
658
- 648 77.46913580
659
- 649 77.34976888
660
- 650 77.38461538
661
- 651 77.41935484
662
- 652 77.45398773
663
- 653 77.48851455
664
- 654 77.52293578
665
- 655 77.55725191
666
- 656 77.59146341
667
- 657 77.62557078
668
- 658 77.65957447
669
- 659 77.54172989
670
- 660 77.42424242
671
- 661 77.45839637
672
- 662 77.49244713
673
- 663 77.52639517
674
- 664 77.56024096
675
- 665 77.59398496
676
- 666 77.62762763
677
- 667 77.66116942
678
- 668 77.69461078
679
- 669 77.72795217
680
- 670 77.76119403
681
- 671 77.79433681
682
- 672 77.82738095
683
- 673 77.86032689
684
- 674 77.89317507
685
- 675 77.92592593
686
- 676 77.95857988
687
- 677 77.99113737
688
- 678 78.02359882
689
- 679 78.05596465
690
- 680 78.08823529
691
- 681 77.97356828
692
- 682 78.00586510
693
- 683 78.03806735
694
- 684 78.07017544
695
- 685 78.10218978
696
- 686 78.13411079
697
- 687 78.16593886
698
- 688 78.19767442
699
- 689 78.22931785
700
- 690 78.26086957
701
- 691 78.29232996
702
- 692 78.32369942
703
- 693 78.21067821
704
- 694 78.24207493
705
- 695 78.27338129
706
- 696 78.30459770
707
- 697 78.33572453
708
- 698 78.36676218
709
- 699 78.39771102
710
- 700 78.42857143
711
- 701 78.45934379
712
- 702 78.49002849
713
- 703 78.52062589
714
- 704 78.55113636
715
- 705 78.58156028
716
- 706 78.61189802
717
- 707 78.64214993
718
- 708 78.67231638
719
- 709 78.70239774
720
- 710 78.73239437
721
- 711 78.76230661
722
- 712 78.79213483
723
- 713 78.82187938
724
- 714 78.85154062
725
- 715 78.74125874
726
- 716 78.77094972
727
- 717 78.80055788
728
- 718 78.83008357
729
- 719 78.85952712
730
- 720 78.88888889
731
- 721 78.77947295
732
- 722 78.80886427
733
- 723 78.83817427
734
- 724 78.86740331
735
- 725 78.89655172
736
- 726 78.92561983
737
- 727 78.95460798
738
- 728 78.98351648
739
- 729 79.01234568
740
- 730 79.04109589
741
- 731 79.06976744
742
- 732 78.96174863
743
- 733 78.85402456
744
- 734 78.88283379
745
- 735 78.77551020
746
- 736 78.80434783
747
- 737 78.69742198
748
- 738 78.59078591
749
- 739 78.48443843
750
- 740 78.51351351
751
- 741 78.54251012
752
- 742 78.57142857
753
- 743 78.60026918
754
- 744 78.62903226
755
- 745 78.65771812
756
- 746 78.68632708
757
- 747 78.58099063
758
- 748 78.60962567
759
- 749 78.63818425
760
- 750 78.66666667
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 36 key-value pairs and 292 tensors from ./Watt-Tool-8B-F16.gguf (version GGUF V3 (latest))
4
 
5
+ 750 80.93333333
 
 
 
 
6
 
7
+
8
+ llama_perf_context_print: load time = 622.45 ms
9
+ llama_perf_context_print: prompt eval time = 412508.17 ms / 125702 tokens ( 3.28 ms per token, 304.73 tokens per second)
10
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
11
+ llama_perf_context_print: total time = 418728.78 ms / 125703 tokens
12
+ ggml_metal_free: deallocating
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scores/Watt-Tool-8B-F16.mmlu CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 1548 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 1548 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 50.00000000
13
- 3 33.33333333
14
- 4 50.00000000
15
- 5 40.00000000
16
- 6 33.33333333
17
- 7 42.85714286
18
- 8 50.00000000
19
- 9 55.55555556
20
- 10 60.00000000
21
- 11 54.54545455
22
- 12 50.00000000
23
- 13 53.84615385
24
- 14 50.00000000
25
- 15 53.33333333
26
- 16 56.25000000
27
- 17 58.82352941
28
- 18 55.55555556
29
- 19 57.89473684
30
- 20 60.00000000
31
- 21 61.90476190
32
- 22 59.09090909
33
- 23 56.52173913
34
- 24 58.33333333
35
- 25 60.00000000
36
- 26 57.69230769
37
- 27 59.25925926
38
- 28 60.71428571
39
- 29 62.06896552
40
- 30 63.33333333
41
- 31 64.51612903
42
- 32 65.62500000
43
- 33 63.63636364
44
- 34 64.70588235
45
- 35 65.71428571
46
- 36 66.66666667
47
- 37 64.86486486
48
- 38 65.78947368
49
- 39 64.10256410
50
- 40 62.50000000
51
- 41 60.97560976
52
- 42 59.52380952
53
- 43 60.46511628
54
- 44 59.09090909
55
- 45 60.00000000
56
- 46 58.69565217
57
- 47 57.44680851
58
- 48 58.33333333
59
- 49 59.18367347
60
- 50 58.00000000
61
- 51 56.86274510
62
- 52 55.76923077
63
- 53 54.71698113
64
- 54 53.70370370
65
- 55 54.54545455
66
- 56 55.35714286
67
- 57 56.14035088
68
- 58 55.17241379
69
- 59 55.93220339
70
- 60 55.00000000
71
- 61 55.73770492
72
- 62 54.83870968
73
- 63 53.96825397
74
- 64 53.12500000
75
- 65 52.30769231
76
- 66 51.51515152
77
- 67 50.74626866
78
- 68 50.00000000
79
- 69 49.27536232
80
- 70 50.00000000
81
- 71 50.70422535
82
- 72 50.00000000
83
- 73 49.31506849
84
- 74 50.00000000
85
- 75 50.66666667
86
- 76 51.31578947
87
- 77 51.94805195
88
- 78 52.56410256
89
- 79 51.89873418
90
- 80 51.25000000
91
- 81 51.85185185
92
- 82 51.21951220
93
- 83 51.80722892
94
- 84 52.38095238
95
- 85 51.76470588
96
- 86 51.16279070
97
- 87 51.72413793
98
- 88 52.27272727
99
- 89 51.68539326
100
- 90 51.11111111
101
- 91 50.54945055
102
- 92 50.00000000
103
- 93 49.46236559
104
- 94 50.00000000
105
- 95 49.47368421
106
- 96 50.00000000
107
- 97 50.51546392
108
- 98 50.00000000
109
- 99 50.50505051
110
- 100 50.00000000
111
- 101 49.50495050
112
- 102 49.01960784
113
- 103 48.54368932
114
- 104 49.03846154
115
- 105 49.52380952
116
- 106 50.00000000
117
- 107 49.53271028
118
- 108 50.00000000
119
- 109 50.45871560
120
- 110 50.90909091
121
- 111 51.35135135
122
- 112 51.78571429
123
- 113 51.32743363
124
- 114 51.75438596
125
- 115 52.17391304
126
- 116 52.58620690
127
- 117 52.13675214
128
- 118 51.69491525
129
- 119 52.10084034
130
- 120 51.66666667
131
- 121 52.06611570
132
- 122 51.63934426
133
- 123 51.21951220
134
- 124 51.61290323
135
- 125 52.00000000
136
- 126 51.58730159
137
- 127 51.18110236
138
- 128 50.78125000
139
- 129 50.38759690
140
- 130 50.00000000
141
- 131 50.38167939
142
- 132 50.75757576
143
- 133 50.37593985
144
- 134 50.00000000
145
- 135 49.62962963
146
- 136 49.26470588
147
- 137 49.63503650
148
- 138 49.27536232
149
- 139 49.64028777
150
- 140 50.00000000
151
- 141 50.35460993
152
- 142 50.70422535
153
- 143 51.04895105
154
- 144 51.38888889
155
- 145 51.72413793
156
- 146 52.05479452
157
- 147 51.70068027
158
- 148 52.02702703
159
- 149 51.67785235
160
- 150 51.33333333
161
- 151 50.99337748
162
- 152 51.31578947
163
- 153 51.63398693
164
- 154 51.29870130
165
- 155 50.96774194
166
- 156 50.64102564
167
- 157 50.95541401
168
- 158 51.26582278
169
- 159 50.94339623
170
- 160 51.25000000
171
- 161 50.93167702
172
- 162 50.61728395
173
- 163 50.30674847
174
- 164 50.00000000
175
- 165 50.30303030
176
- 166 50.00000000
177
- 167 50.29940120
178
- 168 50.00000000
179
- 169 50.29585799
180
- 170 50.00000000
181
- 171 49.70760234
182
- 172 49.41860465
183
- 173 49.13294798
184
- 174 48.85057471
185
- 175 48.57142857
186
- 176 48.29545455
187
- 177 48.02259887
188
- 178 47.75280899
189
- 179 48.04469274
190
- 180 47.77777778
191
- 181 47.51381215
192
- 182 47.25274725
193
- 183 47.54098361
194
- 184 47.28260870
195
- 185 47.56756757
196
- 186 47.84946237
197
- 187 47.59358289
198
- 188 47.87234043
199
- 189 48.14814815
200
- 190 47.89473684
201
- 191 48.16753927
202
- 192 47.91666667
203
- 193 47.66839378
204
- 194 47.93814433
205
- 195 48.20512821
206
- 196 48.46938776
207
- 197 48.22335025
208
- 198 48.48484848
209
- 199 48.24120603
210
- 200 48.00000000
211
- 201 47.76119403
212
- 202 47.52475248
213
- 203 47.29064039
214
- 204 47.05882353
215
- 205 47.31707317
216
- 206 47.08737864
217
- 207 47.34299517
218
- 208 47.11538462
219
- 209 46.88995215
220
- 210 46.66666667
221
- 211 46.91943128
222
- 212 46.69811321
223
- 213 46.47887324
224
- 214 46.26168224
225
- 215 46.51162791
226
- 216 46.75925926
227
- 217 46.54377880
228
- 218 46.78899083
229
- 219 47.03196347
230
- 220 47.27272727
231
- 221 47.05882353
232
- 222 46.84684685
233
- 223 47.08520179
234
- 224 47.32142857
235
- 225 47.55555556
236
- 226 47.34513274
237
- 227 47.13656388
238
- 228 47.36842105
239
- 229 47.16157205
240
- 230 46.95652174
241
- 231 46.75324675
242
- 232 46.98275862
243
- 233 47.21030043
244
- 234 47.43589744
245
- 235 47.23404255
246
- 236 47.03389831
247
- 237 46.83544304
248
- 238 46.63865546
249
- 239 46.44351464
250
- 240 46.66666667
251
- 241 46.47302905
252
- 242 46.69421488
253
- 243 46.91358025
254
- 244 46.72131148
255
- 245 46.93877551
256
- 246 46.74796748
257
- 247 46.96356275
258
- 248 47.17741935
259
- 249 46.98795181
260
- 250 46.80000000
261
- 251 46.61354582
262
- 252 46.42857143
263
- 253 46.24505929
264
- 254 46.06299213
265
- 255 45.88235294
266
- 256 46.09375000
267
- 257 46.30350195
268
- 258 46.12403101
269
- 259 45.94594595
270
- 260 46.15384615
271
- 261 45.97701149
272
- 262 45.80152672
273
- 263 45.62737643
274
- 264 45.45454545
275
- 265 45.66037736
276
- 266 45.86466165
277
- 267 45.69288390
278
- 268 45.52238806
279
- 269 45.72490706
280
- 270 45.92592593
281
- 271 45.75645756
282
- 272 45.95588235
283
- 273 45.78754579
284
- 274 45.62043796
285
- 275 45.45454545
286
- 276 45.65217391
287
- 277 45.84837545
288
- 278 45.68345324
289
- 279 45.51971326
290
- 280 45.35714286
291
- 281 45.19572954
292
- 282 45.03546099
293
- 283 44.87632509
294
- 284 44.71830986
295
- 285 44.91228070
296
- 286 44.75524476
297
- 287 44.94773519
298
- 288 44.79166667
299
- 289 44.63667820
300
- 290 44.82758621
301
- 291 45.01718213
302
- 292 45.20547945
303
- 293 45.39249147
304
- 294 45.23809524
305
- 295 45.08474576
306
- 296 44.93243243
307
- 297 44.78114478
308
- 298 44.96644295
309
- 299 45.15050167
310
- 300 45.33333333
311
- 301 45.18272425
312
- 302 45.36423841
313
- 303 45.21452145
314
- 304 45.06578947
315
- 305 44.91803279
316
- 306 44.77124183
317
- 307 44.95114007
318
- 308 44.80519481
319
- 309 44.66019417
320
- 310 44.51612903
321
- 311 44.69453376
322
- 312 44.87179487
323
- 313 44.72843450
324
- 314 44.90445860
325
- 315 44.76190476
326
- 316 44.62025316
327
- 317 44.79495268
328
- 318 44.65408805
329
- 319 44.82758621
330
- 320 44.68750000
331
- 321 44.54828660
332
- 322 44.72049689
333
- 323 44.58204334
334
- 324 44.44444444
335
- 325 44.61538462
336
- 326 44.47852761
337
- 327 44.64831804
338
- 328 44.51219512
339
- 329 44.37689970
340
- 330 44.24242424
341
- 331 44.10876133
342
- 332 44.27710843
343
- 333 44.44444444
344
- 334 44.31137725
345
- 335 44.47761194
346
- 336 44.64285714
347
- 337 44.51038576
348
- 338 44.37869822
349
- 339 44.24778761
350
- 340 44.41176471
351
- 341 44.28152493
352
- 342 44.44444444
353
- 343 44.31486880
354
- 344 44.18604651
355
- 345 44.05797101
356
- 346 44.21965318
357
- 347 44.38040346
358
- 348 44.25287356
359
- 349 44.12607450
360
- 350 44.28571429
361
- 351 44.15954416
362
- 352 44.03409091
363
- 353 44.19263456
364
- 354 44.06779661
365
- 355 43.94366197
366
- 356 44.10112360
367
- 357 43.97759104
368
- 358 43.85474860
369
- 359 43.73259053
370
- 360 43.88888889
371
- 361 44.04432133
372
- 362 44.19889503
373
- 363 44.07713499
374
- 364 43.95604396
375
- 365 43.83561644
376
- 366 43.98907104
377
- 367 43.86920981
378
- 368 43.75000000
379
- 369 43.63143631
380
- 370 43.78378378
381
- 371 43.66576819
382
- 372 43.54838710
383
- 373 43.43163539
384
- 374 43.31550802
385
- 375 43.46666667
386
- 376 43.35106383
387
- 377 43.50132626
388
- 378 43.65079365
389
- 379 43.53562005
390
- 380 43.42105263
391
- 381 43.56955381
392
- 382 43.71727749
393
- 383 43.86422977
394
- 384 43.75000000
395
- 385 43.89610390
396
- 386 43.78238342
397
- 387 43.66925065
398
- 388 43.55670103
399
- 389 43.70179949
400
- 390 43.84615385
401
- 391 43.98976982
402
- 392 44.13265306
403
- 393 44.02035623
404
- 394 43.90862944
405
- 395 44.05063291
406
- 396 44.19191919
407
- 397 44.08060453
408
- 398 43.96984925
409
- 399 44.11027569
410
- 400 44.00000000
411
- 401 44.13965087
412
- 402 44.27860697
413
- 403 44.16873449
414
- 404 44.30693069
415
- 405 44.44444444
416
- 406 44.33497537
417
- 407 44.22604423
418
- 408 44.11764706
419
- 409 44.25427873
420
- 410 44.14634146
421
- 411 44.03892944
422
- 412 43.93203883
423
- 413 43.82566586
424
- 414 43.71980676
425
- 415 43.61445783
426
- 416 43.75000000
427
- 417 43.64508393
428
- 418 43.54066986
429
- 419 43.67541766
430
- 420 43.80952381
431
- 421 43.94299287
432
- 422 43.83886256
433
- 423 43.73522459
434
- 424 43.86792453
435
- 425 43.76470588
436
- 426 43.89671362
437
- 427 43.79391101
438
- 428 43.69158879
439
- 429 43.82284382
440
- 430 43.72093023
441
- 431 43.85150812
442
- 432 43.75000000
443
- 433 43.64896074
444
- 434 43.54838710
445
- 435 43.44827586
446
- 436 43.34862385
447
- 437 43.24942792
448
- 438 43.37899543
449
- 439 43.50797267
450
- 440 43.40909091
451
- 441 43.31065760
452
- 442 43.21266968
453
- 443 43.11512415
454
- 444 43.01801802
455
- 445 42.92134831
456
- 446 42.82511211
457
- 447 42.72930649
458
- 448 42.63392857
459
- 449 42.53897550
460
- 450 42.66666667
461
- 451 42.79379157
462
- 452 42.69911504
463
- 453 42.82560706
464
- 454 42.73127753
465
- 455 42.63736264
466
- 456 42.76315789
467
- 457 42.88840263
468
- 458 42.79475983
469
- 459 42.70152505
470
- 460 42.82608696
471
- 461 42.73318872
472
- 462 42.64069264
473
- 463 42.76457883
474
- 464 42.88793103
475
- 465 42.79569892
476
- 466 42.70386266
477
- 467 42.82655246
478
- 468 42.94871795
479
- 469 43.07036247
480
- 470 42.97872340
481
- 471 42.88747346
482
- 472 42.79661017
483
- 473 42.91754757
484
- 474 42.82700422
485
- 475 42.73684211
486
- 476 42.85714286
487
- 477 42.76729560
488
- 478 42.88702929
489
- 479 42.79749478
490
- 480 42.91666667
491
- 481 42.82744283
492
- 482 42.73858921
493
- 483 42.65010352
494
- 484 42.56198347
495
- 485 42.47422680
496
- 486 42.59259259
497
- 487 42.71047228
498
- 488 42.62295082
499
- 489 42.53578732
500
- 490 42.65306122
501
- 491 42.56619145
502
- 492 42.47967480
503
- 493 42.39350913
504
- 494 42.51012146
505
- 495 42.62626263
506
- 496 42.74193548
507
- 497 42.65593561
508
- 498 42.57028112
509
- 499 42.48496994
510
- 500 42.60000000
511
- 501 42.51497006
512
- 502 42.62948207
513
- 503 42.54473161
514
- 504 42.46031746
515
- 505 42.37623762
516
- 506 42.49011858
517
- 507 42.40631164
518
- 508 42.32283465
519
- 509 42.23968566
520
- 510 42.35294118
521
- 511 42.27005871
522
- 512 42.18750000
523
- 513 42.10526316
524
- 514 42.02334630
525
- 515 42.13592233
526
- 516 42.24806202
527
- 517 42.35976789
528
- 518 42.47104247
529
- 519 42.38921002
530
- 520 42.50000000
531
- 521 42.41842610
532
- 522 42.52873563
533
- 523 42.44741874
534
- 524 42.55725191
535
- 525 42.47619048
536
- 526 42.39543726
537
- 527 42.31499051
538
- 528 42.23484848
539
- 529 42.15500945
540
- 530 42.26415094
541
- 531 42.18455744
542
- 532 42.10526316
543
- 533 42.02626642
544
- 534 42.13483146
545
- 535 42.24299065
546
- 536 42.16417910
547
- 537 42.27188082
548
- 538 42.37918216
549
- 539 42.30055659
550
- 540 42.22222222
551
- 541 42.14417745
552
- 542 42.25092251
553
- 543 42.17311234
554
- 544 42.27941176
555
- 545 42.20183486
556
- 546 42.12454212
557
- 547 42.04753199
558
- 548 42.15328467
559
- 549 42.07650273
560
- 550 42.00000000
561
- 551 42.10526316
562
- 552 42.02898551
563
- 553 41.95298373
564
- 554 42.05776173
565
- 555 41.98198198
566
- 556 42.08633094
567
- 557 42.19030521
568
- 558 42.29390681
569
- 559 42.21824687
570
- 560 42.14285714
571
- 561 42.06773619
572
- 562 42.17081851
573
- 563 42.27353464
574
- 564 42.19858156
575
- 565 42.30088496
576
- 566 42.22614841
577
- 567 42.15167549
578
- 568 42.07746479
579
- 569 42.00351494
580
- 570 41.92982456
581
- 571 41.85639229
582
- 572 41.95804196
583
- 573 41.88481675
584
- 574 41.81184669
585
- 575 41.73913043
586
- 576 41.84027778
587
- 577 41.94107452
588
- 578 42.04152249
589
- 579 41.96891192
590
- 580 42.06896552
591
- 581 42.16867470
592
- 582 42.09621993
593
- 583 42.19554031
594
- 584 42.29452055
595
- 585 42.22222222
596
- 586 42.32081911
597
- 587 42.24872232
598
- 588 42.17687075
599
- 589 42.10526316
600
- 590 42.03389831
601
- 591 41.96277496
602
- 592 41.89189189
603
- 593 41.98988196
604
- 594 42.08754209
605
- 595 42.01680672
606
- 596 42.11409396
607
- 597 42.04355109
608
- 598 41.97324415
609
- 599 42.07011686
610
- 600 42.00000000
611
- 601 41.93011647
612
- 602 41.86046512
613
- 603 41.95688226
614
- 604 41.88741722
615
- 605 41.81818182
616
- 606 41.74917492
617
- 607 41.84514003
618
- 608 41.77631579
619
- 609 41.70771757
620
- 610 41.63934426
621
- 611 41.73486088
622
- 612 41.66666667
623
- 613 41.76182708
624
- 614 41.85667752
625
- 615 41.78861789
626
- 616 41.88311688
627
- 617 41.97730956
628
- 618 41.90938511
629
- 619 41.84168013
630
- 620 41.77419355
631
- 621 41.70692432
632
- 622 41.80064309
633
- 623 41.73354735
634
- 624 41.66666667
635
- 625 41.60000000
636
- 626 41.69329073
637
- 627 41.62679426
638
- 628 41.56050955
639
- 629 41.65341812
640
- 630 41.74603175
641
- 631 41.83835182
642
- 632 41.93037975
643
- 633 41.86413902
644
- 634 41.79810726
645
- 635 41.73228346
646
- 636 41.66666667
647
- 637 41.60125589
648
- 638 41.53605016
649
- 639 41.62754304
650
- 640 41.71875000
651
- 641 41.65366615
652
- 642 41.58878505
653
- 643 41.52410575
654
- 644 41.61490683
655
- 645 41.55038760
656
- 646 41.48606811
657
- 647 41.42194745
658
- 648 41.35802469
659
- 649 41.29429892
660
- 650 41.23076923
661
- 651 41.16743472
662
- 652 41.10429448
663
- 653 41.04134763
664
- 654 41.13149847
665
- 655 41.06870229
666
- 656 41.15853659
667
- 657 41.09589041
668
- 658 41.18541033
669
- 659 41.12291351
670
- 660 41.06060606
671
- 661 40.99848714
672
- 662 40.93655589
673
- 663 40.87481146
674
- 664 40.96385542
675
- 665 41.05263158
676
- 666 40.99099099
677
- 667 41.07946027
678
- 668 41.16766467
679
- 669 41.25560538
680
- 670 41.34328358
681
- 671 41.28166915
682
- 672 41.22023810
683
- 673 41.15898960
684
- 674 41.24629080
685
- 675 41.18518519
686
- 676 41.12426036
687
- 677 41.06351551
688
- 678 41.15044248
689
- 679 41.08983800
690
- 680 41.17647059
691
- 681 41.26284875
692
- 682 41.20234604
693
- 683 41.14202050
694
- 684 41.22807018
695
- 685 41.16788321
696
- 686 41.10787172
697
- 687 41.19359534
698
- 688 41.13372093
699
- 689 41.07402032
700
- 690 41.01449275
701
- 691 41.09985528
702
- 692 41.04046243
703
- 693 40.98124098
704
- 694 41.06628242
705
- 695 41.00719424
706
- 696 41.09195402
707
- 697 41.03299857
708
- 698 41.11747851
709
- 699 41.20171674
710
- 700 41.14285714
711
- 701 41.22681883
712
- 702 41.31054131
713
- 703 41.39402560
714
- 704 41.33522727
715
- 705 41.27659574
716
- 706 41.21813031
717
- 707 41.15983027
718
- 708 41.10169492
719
- 709 41.04372355
720
- 710 40.98591549
721
- 711 41.06891702
722
- 712 41.15168539
723
- 713 41.23422160
724
- 714 41.31652661
725
- 715 41.25874126
726
- 716 41.20111732
727
- 717 41.14365411
728
- 718 41.08635097
729
- 719 41.02920723
730
- 720 40.97222222
731
- 721 40.91539528
732
- 722 40.85872576
733
- 723 40.94052559
734
- 724 40.88397790
735
- 725 40.96551724
736
- 726 40.90909091
737
- 727 40.85281981
738
- 728 40.79670330
739
- 729 40.74074074
740
- 730 40.68493151
741
- 731 40.62927497
742
- 732 40.57377049
743
- 733 40.65484311
744
- 734 40.73569482
745
- 735 40.81632653
746
- 736 40.76086957
747
- 737 40.84124830
748
- 738 40.92140921
749
- 739 41.00135318
750
- 740 40.94594595
751
- 741 40.89068826
752
- 742 40.97035040
753
- 743 41.04979812
754
- 744 41.12903226
755
- 745 41.07382550
756
- 746 41.01876676
757
- 747 40.96385542
758
- 748 40.90909091
759
- 749 40.85447263
760
- 750 40.93333333
761
 
762
- Final result: 40.9333 +/- 1.7967
763
- Random chance: 25.0000 +/- 1.5822
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 36 key-value pairs and 292 tensors from ./Watt-Tool-8B-F16.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 42.1333 +/- 1.8042
6
+ Random chance: 25.0000 +/- 1.5822
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 625.31 ms
10
+ llama_perf_context_print: prompt eval time = 245071.99 ms / 69227 tokens ( 3.54 ms per token, 282.48 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 247990.59 ms / 69228 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-F16.tqa CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 817 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 817 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 50.00000000
13
- 3 33.33333333
14
- 4 25.00000000
15
- 5 20.00000000
16
- 6 16.66666667
17
- 7 14.28571429
18
- 8 12.50000000
19
- 9 11.11111111
20
- 10 10.00000000
21
- 11 9.09090909
22
- 12 16.66666667
23
- 13 23.07692308
24
- 14 21.42857143
25
- 15 26.66666667
26
- 16 31.25000000
27
- 17 29.41176471
28
- 18 27.77777778
29
- 19 31.57894737
30
- 20 30.00000000
31
- 21 28.57142857
32
- 22 27.27272727
33
- 23 26.08695652
34
- 24 29.16666667
35
- 25 28.00000000
36
- 26 26.92307692
37
- 27 25.92592593
38
- 28 25.00000000
39
- 29 27.58620690
40
- 30 26.66666667
41
- 31 25.80645161
42
- 32 28.12500000
43
- 33 30.30303030
44
- 34 29.41176471
45
- 35 28.57142857
46
- 36 27.77777778
47
- 37 29.72972973
48
- 38 28.94736842
49
- 39 30.76923077
50
- 40 30.00000000
51
- 41 29.26829268
52
- 42 28.57142857
53
- 43 30.23255814
54
- 44 31.81818182
55
- 45 33.33333333
56
- 46 32.60869565
57
- 47 34.04255319
58
- 48 35.41666667
59
- 49 34.69387755
60
- 50 36.00000000
61
- 51 35.29411765
62
- 52 34.61538462
63
- 53 33.96226415
64
- 54 33.33333333
65
- 55 34.54545455
66
- 56 33.92857143
67
- 57 33.33333333
68
- 58 32.75862069
69
- 59 32.20338983
70
- 60 33.33333333
71
- 61 32.78688525
72
- 62 33.87096774
73
- 63 33.33333333
74
- 64 34.37500000
75
- 65 33.84615385
76
- 66 34.84848485
77
- 67 34.32835821
78
- 68 33.82352941
79
- 69 34.78260870
80
- 70 34.28571429
81
- 71 33.80281690
82
- 72 34.72222222
83
- 73 35.61643836
84
- 74 35.13513514
85
- 75 34.66666667
86
- 76 34.21052632
87
- 77 33.76623377
88
- 78 33.33333333
89
- 79 32.91139241
90
- 80 33.75000000
91
- 81 33.33333333
92
- 82 34.14634146
93
- 83 33.73493976
94
- 84 33.33333333
95
- 85 32.94117647
96
- 86 32.55813953
97
- 87 32.18390805
98
- 88 31.81818182
99
- 89 32.58426966
100
- 90 32.22222222
101
- 91 31.86813187
102
- 92 31.52173913
103
- 93 31.18279570
104
- 94 30.85106383
105
- 95 30.52631579
106
- 96 31.25000000
107
- 97 31.95876289
108
- 98 31.63265306
109
- 99 32.32323232
110
- 100 33.00000000
111
- 101 33.66336634
112
- 102 33.33333333
113
- 103 33.00970874
114
- 104 32.69230769
115
- 105 32.38095238
116
- 106 32.07547170
117
- 107 32.71028037
118
- 108 33.33333333
119
- 109 33.02752294
120
- 110 33.63636364
121
- 111 33.33333333
122
- 112 33.92857143
123
- 113 33.62831858
124
- 114 34.21052632
125
- 115 33.91304348
126
- 116 33.62068966
127
- 117 33.33333333
128
- 118 33.05084746
129
- 119 32.77310924
130
- 120 32.50000000
131
- 121 33.05785124
132
- 122 32.78688525
133
- 123 32.52032520
134
- 124 33.06451613
135
- 125 32.80000000
136
- 126 32.53968254
137
- 127 32.28346457
138
- 128 32.03125000
139
- 129 31.78294574
140
- 130 32.30769231
141
- 131 32.06106870
142
- 132 31.81818182
143
- 133 31.57894737
144
- 134 31.34328358
145
- 135 31.85185185
146
- 136 31.61764706
147
- 137 32.11678832
148
- 138 32.60869565
149
- 139 33.09352518
150
- 140 32.85714286
151
- 141 32.62411348
152
- 142 32.39436620
153
- 143 32.16783217
154
- 144 32.63888889
155
- 145 32.41379310
156
- 146 32.19178082
157
- 147 31.97278912
158
- 148 31.75675676
159
- 149 31.54362416
160
- 150 31.33333333
161
- 151 31.78807947
162
- 152 31.57894737
163
- 153 31.37254902
164
- 154 31.16883117
165
- 155 30.96774194
166
- 156 31.41025641
167
- 157 31.21019108
168
- 158 31.64556962
169
- 159 31.44654088
170
- 160 31.25000000
171
- 161 31.67701863
172
- 162 32.09876543
173
- 163 31.90184049
174
- 164 31.70731707
175
- 165 32.12121212
176
- 166 32.53012048
177
- 167 32.33532934
178
- 168 32.14285714
179
- 169 31.95266272
180
- 170 31.76470588
181
- 171 31.57894737
182
- 172 31.39534884
183
- 173 31.21387283
184
- 174 31.60919540
185
- 175 32.00000000
186
- 176 32.38636364
187
- 177 32.20338983
188
- 178 32.02247191
189
- 179 32.40223464
190
- 180 32.77777778
191
- 181 32.59668508
192
- 182 32.41758242
193
- 183 32.24043716
194
- 184 32.06521739
195
- 185 31.89189189
196
- 186 32.25806452
197
- 187 32.08556150
198
- 188 31.91489362
199
- 189 32.27513228
200
- 190 32.63157895
201
- 191 32.46073298
202
- 192 32.29166667
203
- 193 32.12435233
204
- 194 32.47422680
205
- 195 32.30769231
206
- 196 32.14285714
207
- 197 32.48730964
208
- 198 32.82828283
209
- 199 32.66331658
210
- 200 32.50000000
211
- 201 32.33830846
212
- 202 32.67326733
213
- 203 32.51231527
214
- 204 32.35294118
215
- 205 32.68292683
216
- 206 32.52427184
217
- 207 32.85024155
218
- 208 32.69230769
219
- 209 32.53588517
220
- 210 32.38095238
221
- 211 32.22748815
222
- 212 32.07547170
223
- 213 31.92488263
224
- 214 31.77570093
225
- 215 31.62790698
226
- 216 31.94444444
227
- 217 32.25806452
228
- 218 32.56880734
229
- 219 32.42009132
230
- 220 32.27272727
231
- 221 32.12669683
232
- 222 31.98198198
233
- 223 31.83856502
234
- 224 32.14285714
235
- 225 32.00000000
236
- 226 32.30088496
237
- 227 32.59911894
238
- 228 32.45614035
239
- 229 32.31441048
240
- 230 32.17391304
241
- 231 32.03463203
242
- 232 31.89655172
243
- 233 32.18884120
244
- 234 32.47863248
245
- 235 32.34042553
246
- 236 32.62711864
247
- 237 32.91139241
248
- 238 33.19327731
249
- 239 33.05439331
250
- 240 32.91666667
251
- 241 32.78008299
252
- 242 32.64462810
253
- 243 32.51028807
254
- 244 32.37704918
255
- 245 32.24489796
256
- 246 32.11382114
257
- 247 32.38866397
258
- 248 32.25806452
259
- 249 32.53012048
260
- 250 32.40000000
261
- 251 32.66932271
262
- 252 32.53968254
263
- 253 32.41106719
264
- 254 32.28346457
265
- 255 32.15686275
266
- 256 32.03125000
267
- 257 31.90661479
268
- 258 31.78294574
269
- 259 31.66023166
270
- 260 31.53846154
271
- 261 31.80076628
272
- 262 31.67938931
273
- 263 31.55893536
274
- 264 31.81818182
275
- 265 32.07547170
276
- 266 31.95488722
277
- 267 31.83520599
278
- 268 31.71641791
279
- 269 31.59851301
280
- 270 31.48148148
281
- 271 31.36531365
282
- 272 31.25000000
283
- 273 31.50183150
284
- 274 31.38686131
285
- 275 31.27272727
286
- 276 31.15942029
287
- 277 31.04693141
288
- 278 30.93525180
289
- 279 30.82437276
290
- 280 31.07142857
291
- 281 31.31672598
292
- 282 31.20567376
293
- 283 31.44876325
294
- 284 31.69014085
295
- 285 31.57894737
296
- 286 31.81818182
297
- 287 32.05574913
298
- 288 32.29166667
299
- 289 32.17993080
300
- 290 32.06896552
301
- 291 32.30240550
302
- 292 32.19178082
303
- 293 32.42320819
304
- 294 32.31292517
305
- 295 32.54237288
306
- 296 32.77027027
307
- 297 32.65993266
308
- 298 32.55033557
309
- 299 32.44147157
310
- 300 32.33333333
311
- 301 32.22591362
312
- 302 32.11920530
313
- 303 32.01320132
314
- 304 31.90789474
315
- 305 31.80327869
316
- 306 31.69934641
317
- 307 31.59609121
318
- 308 31.81818182
319
- 309 31.71521036
320
- 310 31.61290323
321
- 311 31.51125402
322
- 312 31.73076923
323
- 313 31.62939297
324
- 314 31.52866242
325
- 315 31.42857143
326
- 316 31.32911392
327
- 317 31.23028391
328
- 318 31.13207547
329
- 319 31.03448276
330
- 320 30.93750000
331
- 321 30.84112150
332
- 322 31.05590062
333
- 323 30.95975232
334
- 324 30.86419753
335
- 325 30.76923077
336
- 326 30.67484663
337
- 327 30.88685015
338
- 328 31.09756098
339
- 329 31.00303951
340
- 330 31.21212121
341
- 331 31.11782477
342
- 332 31.32530120
343
- 333 31.23123123
344
- 334 31.13772455
345
- 335 31.34328358
346
- 336 31.54761905
347
- 337 31.45400593
348
- 338 31.36094675
349
- 339 31.56342183
350
- 340 31.47058824
351
- 341 31.67155425
352
- 342 31.87134503
353
- 343 31.77842566
354
- 344 31.97674419
355
- 345 32.17391304
356
- 346 32.08092486
357
- 347 32.27665706
358
- 348 32.18390805
359
- 349 32.09169054
360
- 350 32.00000000
361
- 351 31.90883191
362
- 352 31.81818182
363
- 353 31.72804533
364
- 354 31.63841808
365
- 355 31.83098592
366
- 356 31.74157303
367
- 357 31.65266106
368
- 358 31.84357542
369
- 359 31.75487465
370
- 360 31.66666667
371
- 361 31.85595568
372
- 362 31.76795580
373
- 363 31.95592287
374
- 364 31.86813187
375
- 365 31.78082192
376
- 366 31.69398907
377
- 367 31.88010899
378
- 368 31.79347826
379
- 369 31.70731707
380
- 370 31.62162162
381
- 371 31.53638814
382
- 372 31.72043011
383
- 373 31.63538874
384
- 374 31.81818182
385
- 375 31.73333333
386
- 376 31.64893617
387
- 377 31.83023873
388
- 378 31.74603175
389
- 379 31.66226913
390
- 380 31.84210526
391
- 381 31.75853018
392
- 382 31.67539267
393
- 383 31.59268930
394
- 384 31.51041667
395
- 385 31.42857143
396
- 386 31.34715026
397
- 387 31.26614987
398
- 388 31.44329897
399
- 389 31.36246787
400
- 390 31.28205128
401
- 391 31.45780051
402
- 392 31.37755102
403
- 393 31.55216285
404
- 394 31.72588832
405
- 395 31.64556962
406
- 396 31.56565657
407
- 397 31.73803526
408
- 398 31.65829146
409
- 399 31.57894737
410
- 400 31.50000000
411
- 401 31.67082294
412
- 402 31.84079602
413
- 403 31.76178660
414
- 404 31.68316832
415
- 405 31.60493827
416
- 406 31.52709360
417
- 407 31.44963145
418
- 408 31.37254902
419
- 409 31.29584352
420
- 410 31.21951220
421
- 411 31.14355231
422
- 412 31.31067961
423
- 413 31.47699758
424
- 414 31.64251208
425
- 415 31.80722892
426
- 416 31.73076923
427
- 417 31.65467626
428
- 418 31.57894737
429
- 419 31.50357995
430
- 420 31.66666667
431
- 421 31.82897862
432
- 422 31.75355450
433
- 423 31.67848700
434
- 424 31.60377358
435
- 425 31.52941176
436
- 426 31.45539906
437
- 427 31.38173302
438
- 428 31.30841121
439
- 429 31.23543124
440
- 430 31.16279070
441
- 431 31.09048724
442
- 432 31.25000000
443
- 433 31.40877598
444
- 434 31.56682028
445
- 435 31.49425287
446
- 436 31.42201835
447
- 437 31.35011442
448
- 438 31.50684932
449
- 439 31.66287016
450
- 440 31.59090909
451
- 441 31.51927438
452
- 442 31.44796380
453
- 443 31.37697517
454
- 444 31.53153153
455
- 445 31.46067416
456
- 446 31.39013453
457
- 447 31.31991051
458
- 448 31.25000000
459
- 449 31.40311804
460
- 450 31.55555556
461
- 451 31.48558758
462
- 452 31.41592920
463
- 453 31.56732892
464
- 454 31.49779736
465
- 455 31.42857143
466
- 456 31.35964912
467
- 457 31.29102845
468
- 458 31.22270742
469
- 459 31.15468410
470
- 460 31.08695652
471
- 461 31.01952278
472
- 462 31.16883117
473
- 463 31.31749460
474
- 464 31.46551724
475
- 465 31.61290323
476
- 466 31.54506438
477
- 467 31.47751606
478
- 468 31.62393162
479
- 469 31.55650320
480
- 470 31.48936170
481
- 471 31.63481953
482
- 472 31.77966102
483
- 473 31.71247357
484
- 474 31.85654008
485
- 475 32.00000000
486
- 476 31.93277311
487
- 477 32.07547170
488
- 478 32.00836820
489
- 479 31.94154489
490
- 480 31.87500000
491
- 481 31.80873181
492
- 482 31.95020747
493
- 483 31.88405797
494
- 484 32.02479339
495
- 485 32.16494845
496
- 486 32.09876543
497
- 487 32.23819302
498
- 488 32.17213115
499
- 489 32.31083845
500
- 490 32.44897959
501
- 491 32.58655804
502
- 492 32.72357724
503
- 493 32.65720081
504
- 494 32.79352227
505
- 495 32.72727273
506
- 496 32.66129032
507
- 497 32.59557344
508
- 498 32.53012048
509
- 499 32.46492986
510
- 500 32.40000000
511
- 501 32.53493014
512
- 502 32.47011952
513
- 503 32.40556660
514
- 504 32.53968254
515
- 505 32.47524752
516
- 506 32.41106719
517
- 507 32.54437870
518
- 508 32.67716535
519
- 509 32.80943026
520
- 510 32.94117647
521
- 511 32.87671233
522
- 512 32.81250000
523
- 513 32.74853801
524
- 514 32.68482490
525
- 515 32.81553398
526
- 516 32.75193798
527
- 517 32.88201161
528
- 518 32.81853282
529
- 519 32.75529865
530
- 520 32.69230769
531
- 521 32.62955854
532
- 522 32.56704981
533
- 523 32.50478011
534
- 524 32.44274809
535
- 525 32.38095238
536
- 526 32.31939163
537
- 527 32.25806452
538
- 528 32.19696970
539
- 529 32.32514178
540
- 530 32.45283019
541
- 531 32.39171375
542
- 532 32.33082707
543
- 533 32.45778612
544
- 534 32.39700375
545
- 535 32.33644860
546
- 536 32.27611940
547
- 537 32.40223464
548
- 538 32.34200743
549
- 539 32.28200371
550
- 540 32.22222222
551
- 541 32.16266174
552
- 542 32.10332103
553
- 543 32.04419890
554
- 544 31.98529412
555
- 545 31.92660550
556
- 546 31.86813187
557
- 547 31.99268739
558
- 548 32.11678832
559
- 549 32.05828780
560
- 550 32.00000000
561
- 551 31.94192377
562
- 552 32.06521739
563
- 553 32.18806510
564
- 554 32.12996390
565
- 555 32.07207207
566
- 556 32.01438849
567
- 557 31.95691203
568
- 558 31.89964158
569
- 559 31.84257603
570
- 560 31.96428571
571
- 561 32.08556150
572
- 562 32.02846975
573
- 563 31.97158082
574
- 564 31.91489362
575
- 565 32.03539823
576
- 566 32.15547703
577
- 567 32.09876543
578
- 568 32.21830986
579
- 569 32.33743409
580
- 570 32.28070175
581
- 571 32.39929947
582
- 572 32.34265734
583
- 573 32.28621291
584
- 574 32.40418118
585
- 575 32.52173913
586
- 576 32.63888889
587
- 577 32.75563258
588
- 578 32.87197232
589
- 579 32.98791019
590
- 580 33.10344828
591
- 581 33.04647160
592
- 582 33.16151203
593
- 583 33.10463122
594
- 584 33.21917808
595
- 585 33.16239316
596
- 586 33.27645051
597
- 587 33.21976150
598
- 588 33.16326531
599
- 589 33.10696095
600
- 590 33.05084746
601
- 591 32.99492386
602
- 592 32.93918919
603
- 593 32.88364250
604
- 594 32.99663300
605
- 595 32.94117647
606
- 596 32.88590604
607
- 597 32.83082077
608
- 598 32.77591973
609
- 599 32.72120200
610
- 600 32.66666667
611
- 601 32.77870216
612
- 602 32.72425249
613
- 603 32.83582090
614
- 604 32.78145695
615
- 605 32.72727273
616
- 606 32.83828383
617
- 607 32.78418451
618
- 608 32.73026316
619
- 609 32.84072250
620
- 610 32.78688525
621
- 611 32.73322422
622
- 612 32.84313725
623
- 613 32.78955954
624
- 614 32.73615635
625
- 615 32.68292683
626
- 616 32.79220779
627
- 617 32.90113452
628
- 618 32.84789644
629
- 619 32.95638126
630
- 620 32.90322581
631
- 621 32.85024155
632
- 622 32.95819936
633
- 623 32.90529695
634
- 624 32.85256410
635
- 625 32.80000000
636
- 626 32.74760383
637
- 627 32.85486443
638
- 628 32.96178344
639
- 629 32.90937997
640
- 630 32.85714286
641
- 631 32.96354992
642
- 632 32.91139241
643
- 633 32.85939968
644
- 634 32.96529968
645
- 635 32.91338583
646
- 636 32.86163522
647
- 637 32.81004710
648
- 638 32.75862069
649
- 639 32.70735524
650
- 640 32.81250000
651
- 641 32.76131045
652
- 642 32.71028037
653
- 643 32.65940902
654
- 644 32.60869565
655
- 645 32.71317829
656
- 646 32.66253870
657
- 647 32.61205564
658
- 648 32.71604938
659
- 649 32.66563945
660
- 650 32.76923077
661
- 651 32.87250384
662
- 652 32.97546012
663
- 653 33.07810107
664
- 654 33.02752294
665
- 655 32.97709924
666
- 656 33.07926829
667
- 657 33.18112633
668
- 658 33.13069909
669
- 659 33.23216995
670
- 660 33.18181818
671
- 661 33.13161876
672
- 662 33.08157100
673
- 663 33.03167421
674
- 664 32.98192771
675
- 665 32.93233083
676
- 666 33.03303303
677
- 667 33.13343328
678
- 668 33.08383234
679
- 669 33.03437967
680
- 670 33.13432836
681
- 671 33.08494784
682
- 672 33.03571429
683
- 673 32.98662704
684
- 674 32.93768546
685
- 675 33.03703704
686
- 676 33.13609467
687
- 677 33.08714919
688
- 678 33.03834808
689
- 679 32.98969072
690
- 680 32.94117647
691
- 681 32.89280470
692
- 682 32.99120235
693
- 683 32.94289898
694
- 684 33.04093567
695
- 685 32.99270073
696
- 686 32.94460641
697
- 687 32.89665211
698
- 688 32.84883721
699
- 689 32.94629898
700
- 690 32.89855072
701
- 691 32.99565847
702
- 692 33.09248555
703
- 693 33.04473304
704
- 694 32.99711816
705
- 695 32.94964029
706
- 696 33.04597701
707
- 697 32.99856528
708
- 698 32.95128940
709
- 699 32.90414878
710
- 700 33.00000000
711
- 701 32.95292439
712
- 702 32.90598291
713
- 703 32.85917496
714
- 704 32.81250000
715
- 705 32.76595745
716
- 706 32.86118980
717
- 707 32.95615276
718
- 708 32.90960452
719
- 709 32.86318759
720
- 710 32.81690141
721
- 711 32.77074543
722
- 712 32.86516854
723
- 713 32.81907433
724
- 714 32.77310924
725
- 715 32.86713287
726
- 716 32.82122905
727
- 717 32.91492329
728
- 718 32.86908078
729
- 719 32.96244784
730
- 720 32.91666667
731
- 721 32.87101248
732
- 722 32.82548476
733
- 723 32.78008299
734
- 724 32.73480663
735
- 725 32.68965517
736
- 726 32.78236915
737
- 727 32.73727648
738
- 728 32.69230769
739
- 729 32.78463649
740
- 730 32.87671233
741
- 731 32.83173735
742
- 732 32.92349727
743
- 733 32.87858117
744
- 734 32.83378747
745
- 735 32.78911565
746
- 736 32.74456522
747
- 737 32.83582090
748
- 738 32.79132791
749
- 739 32.74695535
750
- 740 32.70270270
751
- 741 32.65856950
752
- 742 32.61455526
753
- 743 32.57065949
754
- 744 32.66129032
755
- 745 32.61744966
756
- 746 32.70777480
757
- 747 32.79785810
758
- 748 32.75401070
759
- 749 32.84379172
760
- 750 32.93333333
761
 
762
- Final result: 32.9333 +/- 1.7172
763
- Random chance: 19.8992 +/- 1.4588
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 36 key-value pairs and 292 tensors from ./Watt-Tool-8B-F16.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 36.1963 +/- 2.6657
6
+ Random chance: 28.6467 +/- 2.5079
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 621.37 ms
10
+ llama_perf_context_print: prompt eval time = 74638.90 ms / 17686 tokens ( 4.22 ms per token, 236.95 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 75960.59 ms / 17687 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-F16.wng CHANGED
@@ -1,760 +1,11 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- winogrande_score : loaded 1267 tasks from prompt.
6
- winogrande_score : selecting 750 random tasks
7
- winogrande_score : tokenizing selected tasks
8
- winogrande_score : calculating winogrande score over selected tasks.
9
- 1 100.0000 -2.808372 -3.986347 1 1
10
- 2 100.0000 -1.862125 -1.968489 1 1
11
- 3 100.0000 -3.200459 -2.638400 2 2
12
- 4 75.0000 -4.936352 -4.783675 2 1
13
- 5 60.0000 -2.813185 -3.249257 1 2
14
- 6 50.0000 -4.577247 -4.545535 2 1
15
- 7 57.1429 -4.100353 -2.614538 2 2
16
- 8 62.5000 -2.749729 -2.160588 2 2
17
- 9 66.6667 -3.616816 -5.485690 1 1
18
- 10 70.0000 -1.817691 -1.596229 2 2
19
- 11 72.7273 -9.966570 -5.174712 2 2
20
- 12 75.0000 -2.930354 -1.772044 2 2
21
- 13 76.9231 -2.865421 -5.015953 1 1
22
- 14 78.5714 -3.110883 -3.734874 1 1
23
- 15 73.3333 -4.140598 -5.992033 1 2
24
- 16 75.0000 -3.761336 -3.183792 2 2
25
- 17 70.5882 -6.534601 -6.656921 1 2
26
- 18 72.2222 -0.869111 -2.805132 1 1
27
- 19 73.6842 -3.153991 -5.016741 1 1
28
- 20 75.0000 -2.835330 -3.031811 1 1
29
- 21 76.1905 -1.812435 -3.258714 1 1
30
- 22 77.2727 -2.505112 -2.464662 2 2
31
- 23 78.2609 -4.307917 -4.551782 1 1
32
- 24 75.0000 -2.476264 -2.363936 2 1
33
- 25 72.0000 -2.017823 -2.085149 1 2
34
- 26 73.0769 -3.233939 -4.219592 1 1
35
- 27 70.3704 -4.997873 -5.091943 1 2
36
- 28 71.4286 -4.907565 -5.330077 1 1
37
- 29 72.4138 -3.779872 -3.885241 1 1
38
- 30 73.3333 -2.045996 -1.807404 2 2
39
- 31 74.1935 -4.698597 -5.016974 1 1
40
- 32 75.0000 -4.268658 -5.922236 1 1
41
- 33 75.7576 -4.725372 -5.976323 1 1
42
- 34 73.5294 -2.471132 -2.460032 2 1
43
- 35 74.2857 -4.460015 -4.406233 2 2
44
- 36 75.0000 -3.543887 -3.537315 2 2
45
- 37 75.6757 -3.301446 -3.491268 1 1
46
- 38 73.6842 -3.760706 -3.742477 2 1
47
- 39 74.3590 -5.269673 -4.526429 2 2
48
- 40 72.5000 -5.007502 -5.491180 1 2
49
- 41 73.1707 -1.935971 -2.468723 1 1
50
- 42 73.8095 -1.820763 -3.149451 1 1
51
- 43 72.0930 -3.139908 -3.159038 1 2
52
- 44 72.7273 -2.205203 -1.483413 2 2
53
- 45 73.3333 -0.146027 -12.741934 1 1
54
- 46 71.7391 -2.676498 -2.624206 2 1
55
- 47 70.2128 -3.260029 -3.363434 1 2
56
- 48 70.8333 -3.896943 -3.675448 2 2
57
- 49 69.3878 -2.956736 -1.572451 2 1
58
- 50 68.0000 -2.416464 -2.552233 1 2
59
- 51 68.6275 -2.139358 -2.792195 1 1
60
- 52 69.2308 -3.426427 -2.167424 2 2
61
- 53 69.8113 -3.140648 -3.863662 1 1
62
- 54 70.3704 -2.478319 -1.789429 2 2
63
- 55 70.9091 -2.744639 -2.392722 2 2
64
- 56 69.6429 -3.658513 -4.799203 1 2
65
- 57 70.1754 -3.133277 -2.710656 2 2
66
- 58 70.6897 -3.306060 -4.317593 1 1
67
- 59 71.1864 -3.187873 -3.184092 2 2
68
- 60 71.6667 -4.250255 -4.159523 2 2
69
- 61 72.1311 -3.398939 -3.778073 1 1
70
- 62 72.5806 -5.690569 -6.257550 1 1
71
- 63 71.4286 -4.470813 -4.739361 1 2
72
- 64 70.3125 -3.871334 -3.770645 2 1
73
- 65 70.7692 -5.293809 -4.922169 2 2
74
- 66 71.2121 -2.531059 -3.208329 1 1
75
- 67 71.6418 -3.192625 -2.570860 2 2
76
- 68 70.5882 -4.997354 -5.235201 1 2
77
- 69 71.0145 -5.618177 -7.037840 1 1
78
- 70 71.4286 -3.570706 -3.277066 2 2
79
- 71 70.4225 -3.031544 -2.259567 2 1
80
- 72 70.8333 -3.887293 -3.542439 2 2
81
- 73 71.2329 -2.541998 -2.667740 1 1
82
- 74 71.6216 -2.299958 -2.849231 1 1
83
- 75 72.0000 -1.726187 -2.468893 1 1
84
- 76 72.3684 -1.854226 -4.343611 1 1
85
- 77 72.7273 -3.213614 -2.529167 2 2
86
- 78 73.0769 -0.804867 -1.568819 1 1
87
- 79 72.1519 -2.222689 -2.702102 1 2
88
- 80 72.5000 -2.322448 -3.624953 1 1
89
- 81 72.8395 -5.031424 -3.781725 2 2
90
- 82 71.9512 -3.516279 -3.770876 1 2
91
- 83 72.2892 -5.517694 -4.273431 2 2
92
- 84 72.6190 -2.358482 -2.188434 2 2
93
- 85 72.9412 -5.122569 -5.523038 1 1
94
- 86 72.0930 -5.965085 -6.197050 1 2
95
- 87 71.2644 -1.593245 -1.523731 2 1
96
- 88 70.4545 -4.389425 -3.906376 2 1
97
- 89 70.7865 -2.976207 -3.281713 1 1
98
- 90 71.1111 -4.737138 -3.921072 2 2
99
- 91 70.3297 -4.724197 -4.437488 2 1
100
- 92 70.6522 -4.946042 -3.962291 2 2
101
- 93 70.9677 -2.255734 -2.168020 2 2
102
- 94 71.2766 -4.057702 -3.587712 2 2
103
- 95 71.5789 -3.951796 -4.432115 1 1
104
- 96 71.8750 -1.691705 -2.686609 1 1
105
- 97 71.1340 -5.434227 -5.071898 2 1
106
- 98 71.4286 -3.717277 -3.457841 2 2
107
- 99 70.7071 -2.747042 -2.576545 2 1
108
- 100 71.0000 -5.637707 -5.045768 2 2
109
- 101 70.2970 -4.689617 -4.763712 1 2
110
- 102 70.5882 -3.524121 -3.660753 1 1
111
- 103 70.8738 -2.723515 -2.218321 2 2
112
- 104 71.1538 -7.432077 -7.307415 2 2
113
- 105 71.4286 -1.201535 -2.212591 1 1
114
- 106 71.6981 -2.676145 -2.384315 2 2
115
- 107 71.9626 -2.730828 -2.519785 2 2
116
- 108 72.2222 -3.868581 -2.860814 2 2
117
- 109 71.5596 -4.958840 -5.430970 1 2
118
- 110 71.8182 -4.649810 -3.849894 2 2
119
- 111 72.0721 -3.187225 -2.770141 2 2
120
- 112 71.4286 -3.250098 -2.892128 2 1
121
- 113 70.7965 -4.835470 -6.073565 1 2
122
- 114 71.0526 -4.053928 -4.936009 1 1
123
- 115 71.3043 -1.471064 -2.264510 1 1
124
- 116 71.5517 -2.896025 -2.299990 2 2
125
- 117 70.9402 -4.375414 -4.800037 1 2
126
- 118 71.1864 -1.359103 -2.238704 1 1
127
- 119 71.4286 -1.941206 -2.906118 1 1
128
- 120 70.8333 -3.115091 -2.738097 2 1
129
- 121 71.0744 -1.088561 -0.739258 2 2
130
- 122 71.3115 -6.384457 -5.062609 2 2
131
- 123 71.5447 -3.625156 -4.049012 1 1
132
- 124 71.7742 -2.840488 -2.657875 2 2
133
- 125 72.0000 -2.371050 -2.283054 2 2
134
- 126 72.2222 -6.464376 -6.250141 2 2
135
- 127 72.4409 -2.763931 -1.603638 2 2
136
- 128 72.6562 -1.802445 -1.941710 1 1
137
- 129 72.8682 -2.250193 -1.362510 2 2
138
- 130 73.0769 -5.171524 -4.220255 2 2
139
- 131 73.2824 -3.554552 -3.351983 2 2
140
- 132 73.4848 -2.537911 -1.841431 2 2
141
- 133 73.6842 -1.657771 -3.901824 1 1
142
- 134 73.8806 -5.188384 -4.363622 2 2
143
- 135 74.0741 -4.503179 -5.538940 1 1
144
- 136 74.2647 -4.293422 -2.890682 2 2
145
- 137 74.4526 -5.230302 -5.473861 1 1
146
- 138 74.6377 -4.539020 -4.140951 2 2
147
- 139 74.8201 -4.382773 -4.890900 1 1
148
- 140 75.0000 -1.286452 -0.992643 2 2
149
- 141 75.1773 -5.401156 -4.039779 2 2
150
- 142 75.3521 -3.126006 -2.053194 2 2
151
- 143 75.5245 -3.229954 -6.902375 1 1
152
- 144 75.0000 -1.990657 -1.977646 2 1
153
- 145 74.4828 -1.333280 -13.556462 1 2
154
- 146 73.9726 -3.731478 -3.835951 1 2
155
- 147 74.1497 -4.635980 -3.884509 2 2
156
- 148 74.3243 -1.825972 -2.444355 1 1
157
- 149 73.8255 -3.816110 -3.860672 1 2
158
- 150 74.0000 -4.175235 -3.187042 2 2
159
- 151 73.5099 -1.919140 -1.848312 2 1
160
- 152 73.0263 -3.627994 -4.110334 1 2
161
- 153 73.2026 -2.492730 -3.083698 1 1
162
- 154 73.3766 -3.261436 -3.483662 1 1
163
- 155 72.9032 -2.456748 -2.774251 1 2
164
- 156 73.0769 -2.089134 -2.307138 1 1
165
- 157 73.2484 -2.702117 -3.987688 1 1
166
- 158 73.4177 -4.605155 -4.164978 2 2
167
- 159 73.5849 -3.266766 -4.452752 1 1
168
- 160 73.7500 -3.257624 -3.153017 2 2
169
- 161 73.9130 -2.090913 -2.440199 1 1
170
- 162 73.4568 -0.061929 -11.650084 1 2
171
- 163 73.6196 -1.673144 -1.892737 1 1
172
- 164 73.7805 -5.461616 -4.128358 2 2
173
- 165 73.9394 -3.985791 -3.930915 2 2
174
- 166 74.0964 -4.440895 -3.749005 2 2
175
- 167 73.6527 -4.310969 -3.775348 2 1
176
- 168 73.8095 -5.494232 -4.512979 2 2
177
- 169 73.3728 -3.552193 -4.073780 1 2
178
- 170 73.5294 -2.491734 -2.995286 1 1
179
- 171 73.6842 -3.869573 -4.684419 1 1
180
- 172 73.8372 -3.431870 -4.824991 1 1
181
- 173 73.9884 -2.893759 -2.867301 2 2
182
- 174 74.1379 -5.455872 -5.173553 2 2
183
- 175 74.2857 -3.510733 -3.950287 1 1
184
- 176 74.4318 -2.081499 -4.001548 1 1
185
- 177 74.5763 -5.709090 -5.537027 2 2
186
- 178 74.1573 -2.741448 -2.567935 2 1
187
- 179 74.3017 -2.458099 -2.811735 1 1
188
- 180 74.4444 -5.381778 -5.046999 2 2
189
- 181 74.5856 -3.573293 -2.577988 2 2
190
- 182 74.7253 -3.810875 -2.725847 2 2
191
- 183 74.8634 -5.260901 -5.162382 2 2
192
- 184 75.0000 -1.548311 -2.685604 1 1
193
- 185 74.5946 -4.166282 -4.386950 1 2
194
- 186 74.7312 -2.774298 -3.426739 1 1
195
- 187 74.8663 -4.770831 -4.186534 2 2
196
- 188 74.4681 -4.326640 -3.493172 2 1
197
- 189 74.0741 -4.064426 -4.534026 1 2
198
- 190 73.6842 -4.811060 -4.834938 1 2
199
- 191 73.8220 -2.796808 -3.334131 1 1
200
- 192 73.9583 -5.031643 -4.308503 2 2
201
- 193 73.5751 -3.904570 -3.937568 1 2
202
- 194 73.7113 -1.842030 -1.468940 2 2
203
- 195 73.3333 -3.166573 -3.032718 2 1
204
- 196 72.9592 -3.420272 -3.574049 1 2
205
- 197 73.0964 -0.947065 -3.801325 1 1
206
- 198 73.2323 -3.938889 -7.509238 1 1
207
- 199 73.3668 -4.923169 -4.083604 2 2
208
- 200 73.5000 -3.153655 -2.844061 2 2
209
- 201 73.6318 -2.818398 -3.140370 1 1
210
- 202 73.7624 -8.319080 -7.707882 2 2
211
- 203 73.8916 -3.530376 -3.070027 2 2
212
- 204 74.0196 -2.389137 -3.099433 1 1
213
- 205 74.1463 -3.631398 -2.508176 2 2
214
- 206 73.7864 -4.337655 -4.892370 1 2
215
- 207 73.9130 -2.871725 -3.496886 1 1
216
- 208 74.0385 -3.333477 -2.513934 2 2
217
- 209 74.1627 -4.025947 -3.199840 2 2
218
- 210 74.2857 -6.777838 -6.654049 2 2
219
- 211 73.9336 -5.271577 -5.067123 2 1
220
- 212 74.0566 -5.051704 -4.194228 2 2
221
- 213 73.7089 -3.693595 -5.043519 1 2
222
- 214 73.3645 -2.875313 -2.792535 2 1
223
- 215 73.4884 -4.803906 -4.663526 2 2
224
- 216 73.1481 -3.258569 -3.158849 2 1
225
- 217 73.2719 -0.967518 -1.428769 1 1
226
- 218 73.3945 -3.019980 -2.613474 2 2
227
- 219 73.0594 -3.129693 -3.004189 2 1
228
- 220 72.7273 -3.274084 -3.411078 1 2
229
- 221 72.8507 -4.545965 -8.964200 1 1
230
- 222 72.5225 -0.345264 -8.722687 1 2
231
- 223 72.6457 -3.726919 -3.131508 2 2
232
- 224 72.7679 -5.925332 -4.538466 2 2
233
- 225 72.4444 -2.799486 -2.822364 1 2
234
- 226 72.5664 -2.580324 -3.492037 1 1
235
- 227 72.6872 -3.111263 -3.197424 1 1
236
- 228 72.8070 -1.385931 -2.288301 1 1
237
- 229 72.9258 -3.445007 -3.259084 2 2
238
- 230 73.0435 -3.026006 -3.562451 1 1
239
- 231 73.1602 -2.355911 -2.825404 1 1
240
- 232 73.2759 -2.643193 -1.938690 2 2
241
- 233 73.3906 -1.430526 -2.020864 1 1
242
- 234 73.5043 -3.069871 -2.869358 2 2
243
- 235 73.6170 -3.004472 -2.532036 2 2
244
- 236 73.3051 -2.005425 -1.520579 2 1
245
- 237 73.4177 -1.778486 -1.955533 1 1
246
- 238 73.5294 -3.066458 -2.674766 2 2
247
- 239 73.2218 -2.565737 -2.818681 1 2
248
- 240 72.9167 -6.779342 -6.842921 1 2
249
- 241 72.6141 -2.458865 -3.139222 1 2
250
- 242 72.7273 -5.707737 -5.381508 2 2
251
- 243 72.4280 -4.254044 -4.076239 2 1
252
- 244 72.5410 -2.282548 -4.027785 1 1
253
- 245 72.6531 -3.783362 -4.718744 1 1
254
- 246 72.7642 -2.501517 -2.994351 1 1
255
- 247 72.8745 -3.155675 -3.022227 2 2
256
- 248 72.9839 -2.896198 -2.573284 2 2
257
- 249 73.0924 -3.081625 -2.204087 2 2
258
- 250 73.2000 -2.232381 -2.968300 1 1
259
- 251 73.3068 -1.314571 -1.773203 1 1
260
- 252 73.4127 -6.417959 -5.910969 2 2
261
- 253 73.1225 -5.482332 -4.965070 2 1
262
- 254 73.2283 -4.476695 -3.960488 2 2
263
- 255 73.3333 -3.268882 -4.624633 1 1
264
- 256 73.4375 -3.429986 -3.541162 1 1
265
- 257 73.1518 -2.620425 -2.030959 2 1
266
- 258 72.8682 -3.543163 -3.799613 1 2
267
- 259 72.9730 -5.049572 -5.425591 1 1
268
- 260 73.0769 -5.708809 -5.767672 1 1
269
- 261 72.7969 -3.553879 -3.628354 1 2
270
- 262 72.9008 -4.901840 -4.314059 2 2
271
- 263 73.0038 -3.659578 -4.079973 1 1
272
- 264 73.1061 -4.076969 -3.596601 2 2
273
- 265 73.2075 -1.262818 -2.546921 1 1
274
- 266 73.3083 -4.816278 -5.873499 1 1
275
- 267 73.0337 -3.429285 -2.844565 2 1
276
- 268 72.7612 -2.816242 -2.813941 2 1
277
- 269 72.8625 -2.817671 -2.614367 2 2
278
- 270 72.9630 -2.449739 -3.416535 1 1
279
- 271 73.0627 -2.725473 -2.630955 2 2
280
- 272 73.1618 -3.880650 -3.408276 2 2
281
- 273 73.2601 -3.571701 -3.451499 2 2
282
- 274 73.3577 -3.534782 -3.425641 2 2
283
- 275 73.4545 -2.204642 -2.461058 1 1
284
- 276 73.1884 -2.647093 -2.418870 2 1
285
- 277 73.2852 -3.201306 -3.631627 1 1
286
- 278 73.3813 -2.596450 -2.611785 1 1
287
- 279 73.4767 -5.345634 -3.613614 2 2
288
- 280 73.5714 -3.595841 -3.301513 2 2
289
- 281 73.6655 -2.461615 -1.537360 2 2
290
- 282 73.7589 -4.021332 -3.530984 2 2
291
- 283 73.4982 -3.375992 -3.461246 1 2
292
- 284 73.5915 -2.487349 -3.094053 1 1
293
- 285 73.6842 -2.879642 -4.709011 1 1
294
- 286 73.4266 -3.667463 -3.649336 2 1
295
- 287 73.5192 -2.957089 -4.850106 1 1
296
- 288 73.6111 -2.927827 -2.505708 2 2
297
- 289 73.7024 -1.834129 -2.421445 1 1
298
- 290 73.7931 -4.068408 -4.773040 1 1
299
- 291 73.8832 -1.782238 -2.683509 1 1
300
- 292 73.9726 -2.989087 -1.927127 2 2
301
- 293 74.0614 -3.368802 -2.970841 2 2
302
- 294 73.8095 -3.683075 -3.109463 2 1
303
- 295 73.8983 -1.558988 -1.650176 1 1
304
- 296 73.9865 -4.844454 -4.741416 2 2
305
- 297 73.7374 -1.517498 -1.864350 1 2
306
- 298 73.8255 -5.137451 -4.752472 2 2
307
- 299 73.9130 -2.049959 -1.946394 2 2
308
- 300 74.0000 -2.351267 -2.812598 1 1
309
- 301 74.0864 -4.255333 -5.289795 1 1
310
- 302 74.1722 -4.167719 -3.633873 2 2
311
- 303 73.9274 -8.719552 -8.319284 2 1
312
- 304 74.0132 -4.672309 -3.506915 2 2
313
- 305 74.0984 -3.388185 -2.958119 2 2
314
- 306 74.1830 -2.082194 -1.792724 2 2
315
- 307 74.2671 -2.698517 -3.005575 1 1
316
- 308 74.3506 -5.192213 -4.473172 2 2
317
- 309 74.4337 -3.239612 -3.996408 1 1
318
- 310 74.5161 -6.352751 -5.979263 2 2
319
- 311 74.5981 -1.029899 -4.861214 1 1
320
- 312 74.3590 -4.614117 -4.550209 2 1
321
- 313 74.1214 -4.211654 -4.314736 1 2
322
- 314 74.2038 -4.666368 -4.200562 2 2
323
- 315 74.2857 -2.097257 -3.551987 1 1
324
- 316 74.3671 -7.610831 -4.999755 2 2
325
- 317 74.1325 -4.694881 -4.657408 2 1
326
- 318 74.2138 -4.521602 -4.165960 2 2
327
- 319 74.2947 -1.195716 -1.628789 1 1
328
- 320 74.3750 -4.128767 -5.039813 1 1
329
- 321 74.1433 -2.768193 -2.816655 1 2
330
- 322 74.2236 -2.523704 -2.414067 2 2
331
- 323 74.3034 -3.641275 -4.173637 1 1
332
- 324 74.3827 -3.804944 -4.029255 1 1
333
- 325 74.4615 -5.370188 -5.181993 2 2
334
- 326 74.5399 -2.462981 -1.673864 2 2
335
- 327 74.6177 -2.105320 -2.697392 1 1
336
- 328 74.3902 -3.835672 -3.812532 2 1
337
- 329 74.4681 -2.872759 -7.243905 1 1
338
- 330 74.5455 -1.275175 -2.246134 1 1
339
- 331 74.3202 -4.245040 -3.348197 2 1
340
- 332 74.3976 -1.844552 -2.623899 1 1
341
- 333 74.4745 -4.283337 -1.842027 2 2
342
- 334 74.5509 -3.329037 -5.310305 1 1
343
- 335 74.3284 -5.027473 -4.598526 2 1
344
- 336 74.1071 -3.155446 -3.548695 1 2
345
- 337 74.1840 -2.186845 -1.570042 2 2
346
- 338 73.9645 -3.607443 -2.998877 2 1
347
- 339 73.7463 -5.094945 -5.511173 1 2
348
- 340 73.8235 -4.190701 -4.936995 1 1
349
- 341 73.9003 -3.034158 -4.320374 1 1
350
- 342 73.9766 -5.334294 -4.695610 2 2
351
- 343 74.0525 -2.977282 -3.660366 1 1
352
- 344 74.1279 -2.190057 -3.228602 1 1
353
- 345 74.2029 -0.626351 -2.455887 1 1
354
- 346 73.9884 -6.051841 -6.484429 1 2
355
- 347 74.0634 -3.088856 -2.793014 2 2
356
- 348 74.1379 -0.898877 -1.826244 1 1
357
- 349 73.9255 -3.151857 -4.064706 1 2
358
- 350 73.7143 -1.075493 -2.609091 1 2
359
- 351 73.7892 -2.874998 -3.523483 1 1
360
- 352 73.8636 -3.340033 -2.817244 2 2
361
- 353 73.9377 -2.181058 -2.282683 1 1
362
- 354 74.0113 -6.396269 -5.602628 2 2
363
- 355 74.0845 -2.670461 -2.347555 2 2
364
- 356 73.8764 -3.349471 -3.547118 1 2
365
- 357 73.9496 -4.686559 -4.227039 2 2
366
- 358 74.0223 -2.260488 -1.978855 2 2
367
- 359 74.0947 -5.525271 -3.296864 2 2
368
- 360 73.8889 -3.004126 -3.150442 1 2
369
- 361 73.9612 -2.688240 -1.288083 2 2
370
- 362 74.0331 -6.324646 -5.512069 2 2
371
- 363 73.8292 -2.855438 -2.418763 2 1
372
- 364 73.9011 -5.630062 -5.867810 1 1
373
- 365 73.9726 -5.364902 -5.583847 1 1
374
- 366 73.7705 -2.292462 -7.122671 1 2
375
- 367 73.8420 -4.842243 -5.295360 1 1
376
- 368 73.9130 -4.090939 -3.473523 2 2
377
- 369 73.7127 -3.011694 -3.563547 1 2
378
- 370 73.5135 -5.546354 -4.930087 2 1
379
- 371 73.5849 -2.820848 -3.890816 1 1
380
- 372 73.6559 -2.151518 -2.415026 1 1
381
- 373 73.4584 -4.515782 -4.082705 2 1
382
- 374 73.5294 -1.790385 -2.613996 1 1
383
- 375 73.6000 -3.642614 -2.900727 2 2
384
- 376 73.6702 -5.226352 -3.869139 2 2
385
- 377 73.7401 -4.396749 -3.426917 2 2
386
- 378 73.5450 -3.124795 -3.520097 1 2
387
- 379 73.3509 -0.942289 -2.370735 1 2
388
- 380 73.4211 -5.114569 -6.124789 1 1
389
- 381 73.4908 -10.410570 -9.275751 2 2
390
- 382 73.2984 -2.331265 -2.352142 1 2
391
- 383 73.3681 -2.426794 -1.341770 2 2
392
- 384 73.1771 -3.566568 -2.840603 2 1
393
- 385 73.2468 -0.896301 -0.963231 1 1
394
- 386 73.3161 -2.159667 -14.328373 1 1
395
- 387 73.3850 -1.781934 -2.524079 1 1
396
- 388 73.4536 -6.163622 -5.976901 2 2
397
- 389 73.5219 -2.567226 -2.450389 2 2
398
- 390 73.5897 -2.595279 -2.641294 1 1
399
- 391 73.4015 -4.144369 -4.592997 1 2
400
- 392 73.2143 -4.583393 -3.432135 2 1
401
- 393 73.2824 -5.949368 -5.890294 2 2
402
- 394 73.3503 -1.809447 -3.133298 1 1
403
- 395 73.4177 -3.204443 -3.017951 2 2
404
- 396 73.4848 -2.418753 -3.039348 1 1
405
- 397 73.5516 -5.961924 -6.966512 1 1
406
- 398 73.6181 -2.656947 -5.274647 1 1
407
- 399 73.6842 -2.687699 -3.423018 1 1
408
- 400 73.7500 -1.444777 -6.884112 1 1
409
- 401 73.8155 -3.507215 -3.836247 1 1
410
- 402 73.6318 -3.595879 -3.679206 1 2
411
- 403 73.4491 -6.287627 -6.589804 1 2
412
- 404 73.5149 -4.295917 -2.894621 2 2
413
- 405 73.5802 -3.212205 -4.122794 1 1
414
- 406 73.6453 -7.626432 -8.048147 1 1
415
- 407 73.4644 -4.679551 -4.845160 1 2
416
- 408 73.2843 -3.033138 -3.183559 1 2
417
- 409 73.3496 -3.500361 -4.390828 1 1
418
- 410 73.1707 -3.560316 -3.465704 2 1
419
- 411 73.2360 -3.428816 -4.794067 1 1
420
- 412 73.3010 -3.627094 -2.927358 2 2
421
- 413 73.3656 -3.219486 -2.362383 2 2
422
- 414 73.1884 -3.863708 -4.238345 1 2
423
- 415 73.2530 -3.277202 -2.834544 2 2
424
- 416 73.3173 -4.267930 -2.186495 2 2
425
- 417 73.3813 -4.340305 -5.165311 1 1
426
- 418 73.4450 -5.706326 -6.083143 1 1
427
- 419 73.5084 -4.024795 -3.720536 2 2
428
- 420 73.3333 -2.945022 -3.058397 1 2
429
- 421 73.3967 -4.127964 -3.792524 2 2
430
- 422 73.4597 -1.819027 -1.989531 1 1
431
- 423 73.5225 -3.503348 -3.625186 1 1
432
- 424 73.5849 -4.527975 -5.255636 1 1
433
- 425 73.6471 -5.290812 -3.024893 2 2
434
- 426 73.7089 -1.982967 -1.551545 2 2
435
- 427 73.7705 -3.325425 -4.293497 1 1
436
- 428 73.5981 -4.051291 -3.552773 2 1
437
- 429 73.6597 -3.159734 -3.016990 2 2
438
- 430 73.4884 -5.438791 -5.286695 2 1
439
- 431 73.5499 -3.327726 -5.113939 1 1
440
- 432 73.3796 -3.457570 -4.157622 1 2
441
- 433 73.4411 -2.374108 -1.742486 2 2
442
- 434 73.5023 -6.643731 -7.311871 1 1
443
- 435 73.5632 -3.347744 -3.484333 1 1
444
- 436 73.6239 -2.108257 -2.123971 1 1
445
- 437 73.6842 -3.918169 -3.741963 2 2
446
- 438 73.7443 -4.082474 -3.894559 2 2
447
- 439 73.8041 -2.572007 -3.388228 1 1
448
- 440 73.8636 -3.158524 -3.855799 1 1
449
- 441 73.9229 -3.212707 -4.863079 1 1
450
- 442 73.7557 -4.481774 -4.411451 2 1
451
- 443 73.8149 -1.655012 -3.373112 1 1
452
- 444 73.6486 -5.946774 -6.097239 1 2
453
- 445 73.7079 -1.529339 -2.431052 1 1
454
- 446 73.7668 -3.534453 -2.530081 2 2
455
- 447 73.6018 -2.588278 -2.181815 2 1
456
- 448 73.6607 -4.467649 -4.110218 2 2
457
- 449 73.7194 -2.545395 -2.164889 2 2
458
- 450 73.7778 -5.137840 -4.371132 2 2
459
- 451 73.8359 -4.293706 -3.939998 2 2
460
- 452 73.8938 -3.814142 -4.054947 1 1
461
- 453 73.9514 -4.040983 -3.223340 2 2
462
- 454 74.0088 -3.619658 -4.037779 1 1
463
- 455 73.8462 -1.116610 -10.638597 1 2
464
- 456 73.9035 -3.904641 -3.817601 2 2
465
- 457 73.7418 -8.128403 -7.613269 2 1
466
- 458 73.7991 -3.230211 -2.417378 2 2
467
- 459 73.8562 -4.217892 -4.055873 2 2
468
- 460 73.9130 -0.640282 -9.077652 1 1
469
- 461 73.9696 -2.053627 -3.086675 1 1
470
- 462 74.0260 -4.968442 -5.032511 1 1
471
- 463 74.0821 -1.243342 -1.598421 1 1
472
- 464 74.1379 -1.694854 -2.277694 1 1
473
- 465 74.1935 -2.939697 -2.004017 2 2
474
- 466 74.2489 -5.366941 -5.307671 2 2
475
- 467 74.0899 -5.061944 -4.874321 2 1
476
- 468 74.1453 -4.739499 -3.349350 2 2
477
- 469 73.9872 -1.797047 -1.484317 2 1
478
- 470 74.0426 -1.686642 -1.152340 2 2
479
- 471 74.0977 -2.079234 -2.524843 1 1
480
- 472 74.1525 -3.511921 -3.473855 2 2
481
- 473 74.2072 -3.644297 -3.168065 2 2
482
- 474 74.2616 -3.932699 -5.270673 1 1
483
- 475 74.3158 -1.828746 -1.775000 2 2
484
- 476 74.3697 -4.535498 -4.533732 2 2
485
- 477 74.2138 -4.319655 -4.913993 1 2
486
- 478 74.2678 -2.682616 -2.871461 1 1
487
- 479 74.1127 -3.237067 -3.171643 2 1
488
- 480 74.1667 -4.697837 -4.210918 2 2
489
- 481 74.2204 -2.005623 -2.144783 1 1
490
- 482 74.0664 -4.624760 -5.747929 1 2
491
- 483 74.1201 -3.659523 -4.439328 1 1
492
- 484 73.9669 -2.788472 -2.362981 2 1
493
- 485 74.0206 -1.299971 -2.022721 1 1
494
- 486 74.0741 -1.963803 -2.557101 1 1
495
- 487 74.1273 -3.258224 -2.929211 2 2
496
- 488 74.1803 -3.551328 -2.578625 2 2
497
- 489 74.2331 -2.048493 -2.664156 1 1
498
- 490 74.2857 -2.553596 -2.917374 1 1
499
- 491 74.3381 -5.172626 -4.181754 2 2
500
- 492 74.3902 -5.123173 -4.322001 2 2
501
- 493 74.4422 -1.879903 -2.697326 1 1
502
- 494 74.4939 -3.423608 -2.793411 2 2
503
- 495 74.5455 -3.469082 -3.629844 1 1
504
- 496 74.5968 -3.322784 -3.186725 2 2
505
- 497 74.6479 -4.465838 -4.899814 1 1
506
- 498 74.4980 -1.737673 -2.031303 1 2
507
- 499 74.5491 -2.559955 -3.222151 1 1
508
- 500 74.6000 -5.271616 -4.480466 2 2
509
- 501 74.6507 -2.058311 -2.786067 1 1
510
- 502 74.7012 -1.652876 -2.559481 1 1
511
- 503 74.7515 -3.202427 -2.794119 2 2
512
- 504 74.8016 -2.582986 -2.798527 1 1
513
- 505 74.8515 -7.226496 -6.852238 2 2
514
- 506 74.9012 -4.165890 -4.996271 1 1
515
- 507 74.9507 -3.025445 -3.356387 1 1
516
- 508 75.0000 -5.327171 -2.962811 2 2
517
- 509 75.0491 -6.418972 -9.008363 1 1
518
- 510 74.9020 -3.370440 -3.161754 2 1
519
- 511 74.9511 -5.198432 -3.884438 2 2
520
- 512 75.0000 -2.635322 -3.093017 1 1
521
- 513 75.0487 -2.957242 -2.770604 2 2
522
- 514 74.9027 -4.520969 -3.819526 2 1
523
- 515 74.9515 -5.818917 -9.124447 1 1
524
- 516 75.0000 -4.141595 -4.243093 1 1
525
- 517 75.0484 -1.526764 -1.912157 1 1
526
- 518 74.9035 -2.418163 -2.491269 1 2
527
- 519 74.9518 -6.956728 -6.775826 2 2
528
- 520 75.0000 -2.553126 -2.421533 2 2
529
- 521 75.0480 -1.562638 -1.682557 1 1
530
- 522 75.0958 -3.709049 -2.135051 2 2
531
- 523 75.1434 -3.798799 -3.567518 2 2
532
- 524 75.1908 -4.149500 -4.647893 1 1
533
- 525 75.2381 -3.912787 -2.457306 2 2
534
- 526 75.2852 -0.889567 -13.738211 1 1
535
- 527 75.3321 -5.372822 -3.669619 2 2
536
- 528 75.3788 -5.811016 -5.463561 2 2
537
- 529 75.4253 -4.822235 -5.620322 1 1
538
- 530 75.4717 -5.022682 -2.653132 2 2
539
- 531 75.5179 -3.418991 -3.614422 1 1
540
- 532 75.5639 -2.127144 -3.322840 1 1
541
- 533 75.6098 -3.020441 -1.928100 2 2
542
- 534 75.6554 -2.541463 -3.151340 1 1
543
- 535 75.7009 -3.548643 -3.513875 2 2
544
- 536 75.5597 -2.402041 -2.183172 2 1
545
- 537 75.6052 -4.679294 -5.895792 1 1
546
- 538 75.6506 -1.852656 -4.206015 1 1
547
- 539 75.6957 -4.086423 -1.722164 2 2
548
- 540 75.7407 -2.774204 -2.711746 2 2
549
- 541 75.7856 -2.394989 -2.875411 1 1
550
- 542 75.6458 -3.932674 -4.776586 1 2
551
- 543 75.6906 -5.417171 -3.757820 2 2
552
- 544 75.7353 -3.049807 -3.923652 1 1
553
- 545 75.5963 -2.403556 -3.621523 1 2
554
- 546 75.6410 -2.767727 -4.524349 1 1
555
- 547 75.6856 -5.888534 -5.043692 2 2
556
- 548 75.5474 -4.541018 -4.499819 2 1
557
- 549 75.5920 -4.808993 -5.093032 1 1
558
- 550 75.4545 -2.549449 -2.619564 1 2
559
- 551 75.3176 -0.840255 -13.521287 1 2
560
- 552 75.1812 -2.025428 -1.844774 2 1
561
- 553 75.2260 -5.110896 -5.067779 2 2
562
- 554 75.2708 -4.715612 -4.469124 2 2
563
- 555 75.3153 -1.716932 -1.290617 2 2
564
- 556 75.3597 -5.689326 -5.323028 2 2
565
- 557 75.4039 -4.099650 -4.745424 1 1
566
- 558 75.4480 -1.657624 -2.953225 1 1
567
- 559 75.4919 -2.365246 -2.247701 2 2
568
- 560 75.3571 -2.777634 -2.545958 2 1
569
- 561 75.4011 -2.088021 -2.586043 1 1
570
- 562 75.4448 -0.706679 -1.867057 1 1
571
- 563 75.3108 -1.149895 -2.274382 1 2
572
- 564 75.3546 -2.489038 -2.837980 1 1
573
- 565 75.3982 -4.633721 -4.459692 2 2
574
- 566 75.4417 -4.201175 -5.139040 1 1
575
- 567 75.4850 -2.142979 -1.616494 2 2
576
- 568 75.5282 -7.394202 -2.624197 2 2
577
- 569 75.3954 -4.958041 -3.762949 2 1
578
- 570 75.4386 -4.013446 -5.322971 1 1
579
- 571 75.4816 -3.892194 -2.786295 2 2
580
- 572 75.5245 -2.550028 -2.265697 2 2
581
- 573 75.3927 -3.571561 -3.669240 1 2
582
- 574 75.4355 -3.545841 -3.489399 2 2
583
- 575 75.4783 -5.136491 -5.029059 2 2
584
- 576 75.5208 -2.834326 -3.594345 1 1
585
- 577 75.5633 -4.379715 -3.817612 2 2
586
- 578 75.6055 -3.100468 -3.071972 2 2
587
- 579 75.4750 -4.389610 -4.827219 1 2
588
- 580 75.5172 -1.959630 -2.845995 1 1
589
- 581 75.5594 -6.279334 -6.742680 1 1
590
- 582 75.4296 -5.119176 -5.411338 1 2
591
- 583 75.4717 -4.401750 -4.005840 2 2
592
- 584 75.3425 -4.387334 -4.262518 2 1
593
- 585 75.3846 -0.384587 -1.628523 1 1
594
- 586 75.4266 -3.358140 -3.606642 1 1
595
- 587 75.4685 -4.668347 -4.932898 1 1
596
- 588 75.5102 -3.285444 -2.268255 2 2
597
- 589 75.5518 -3.744897 -3.623471 2 2
598
- 590 75.5932 -3.773042 -2.964484 2 2
599
- 591 75.6345 -3.634943 -3.024361 2 2
600
- 592 75.6757 -3.123950 -4.010086 1 1
601
- 593 75.7167 -4.010739 -3.360199 2 2
602
- 594 75.5892 -4.161704 -3.975892 2 1
603
- 595 75.4622 -1.612117 -1.704621 1 2
604
- 596 75.5034 -4.807516 -4.613880 2 2
605
- 597 75.5444 -1.527814 -1.750337 1 1
606
- 598 75.5853 -2.903980 -4.020386 1 1
607
- 599 75.6260 -2.687763 -2.948350 1 1
608
- 600 75.6667 -3.944320 -3.760309 2 2
609
- 601 75.7072 -3.358094 -3.289455 2 2
610
- 602 75.7475 -2.638398 -4.194172 1 1
611
- 603 75.7877 -5.977133 -5.362628 2 2
612
- 604 75.6623 -2.652020 -2.679337 1 2
613
- 605 75.5372 -2.685736 -2.069267 2 1
614
- 606 75.5776 -3.062736 -2.256572 2 2
615
- 607 75.4530 -2.298013 -1.663786 2 1
616
- 608 75.4934 -1.965669 -2.358217 1 1
617
- 609 75.5337 -5.740290 -6.406355 1 1
618
- 610 75.5738 -2.486228 -2.487078 1 1
619
- 611 75.6137 -4.563884 -4.693784 1 1
620
- 612 75.6536 -3.372368 -5.300733 1 1
621
- 613 75.6933 -2.501020 -2.157609 2 2
622
- 614 75.7329 -3.209221 -1.954489 2 2
623
- 615 75.7724 -2.137489 -1.893120 2 2
624
- 616 75.8117 -3.438048 -3.434723 2 2
625
- 617 75.8509 -10.464756 -3.108429 2 2
626
- 618 75.7282 -3.888889 -3.957535 1 2
627
- 619 75.7674 -3.479577 -2.991679 2 2
628
- 620 75.8065 -4.004972 -4.132057 1 1
629
- 621 75.8454 -3.410353 -2.688413 2 2
630
- 622 75.8842 -2.801104 -2.822226 1 1
631
- 623 75.7624 -2.816032 -2.539619 2 1
632
- 624 75.8013 -4.250970 -3.462081 2 2
633
- 625 75.8400 -2.821331 -5.146391 1 1
634
- 626 75.7188 -2.428139 -2.084649 2 1
635
- 627 75.7576 -3.815704 -3.984559 1 1
636
- 628 75.7962 -2.425143 -1.853194 2 2
637
- 629 75.8347 -1.724862 -2.156420 1 1
638
- 630 75.8730 -1.103720 -1.622644 1 1
639
- 631 75.9113 -1.359509 -2.505064 1 1
640
- 632 75.9494 -1.942289 -1.758737 2 2
641
- 633 75.9874 -3.015258 -2.434245 2 2
642
- 634 76.0252 -2.271550 -2.936107 1 1
643
- 635 75.9055 -3.969324 -4.838528 1 2
644
- 636 75.9434 -2.036525 -3.936955 1 1
645
- 637 75.9812 -3.174199 -5.580781 1 1
646
- 638 76.0188 -4.944930 -5.764577 1 1
647
- 639 76.0563 -1.376277 -1.697919 1 1
648
- 640 75.9375 -1.456854 -2.159033 1 2
649
- 641 75.9750 -4.852647 -5.428761 1 1
650
- 642 76.0125 -4.561034 -2.770575 2 2
651
- 643 76.0498 -1.363290 -0.632957 2 2
652
- 644 76.0870 -3.844088 -4.626991 1 1
653
- 645 76.1240 -1.242547 -1.366919 1 1
654
- 646 76.0062 -4.049517 -3.907229 2 1
655
- 647 75.8887 -3.791716 -3.559279 2 1
656
- 648 75.9259 -3.257583 -3.209868 2 2
657
- 649 75.9630 -2.645808 -2.916827 1 1
658
- 650 76.0000 -3.195621 -4.138227 1 1
659
- 651 75.8833 -5.676670 -5.777790 1 2
660
- 652 75.9202 -5.161368 -4.659410 2 2
661
- 653 75.9571 -6.732273 -5.663486 2 2
662
- 654 75.8410 -3.064004 -3.318025 1 2
663
- 655 75.8779 -2.765828 -2.697480 2 2
664
- 656 75.9146 -4.748520 -2.485467 2 2
665
- 657 75.7991 -5.241420 -5.702142 1 2
666
- 658 75.8359 -5.157384 -3.503994 2 2
667
- 659 75.8725 -2.096110 -1.712119 2 2
668
- 660 75.9091 -3.400744 -2.068552 2 2
669
- 661 75.9455 -2.624367 -2.949879 1 1
670
- 662 75.9819 -3.228938 -3.681572 1 1
671
- 663 76.0181 -5.899842 -4.233528 2 2
672
- 664 76.0542 -3.102243 -3.134512 1 1
673
- 665 76.0902 -3.684474 -3.397198 2 2
674
- 666 76.1261 -5.121403 -5.371277 1 1
675
- 667 76.0120 -5.960656 -8.999909 1 2
676
- 668 76.0479 -2.470937 -2.790975 1 1
677
- 669 76.0837 -2.613865 -3.193088 1 1
678
- 670 75.9701 -3.289725 -2.748748 2 1
679
- 671 76.0060 -3.201033 -4.718028 1 1
680
- 672 75.8929 -2.544482 -2.559305 1 2
681
- 673 75.7801 -1.978209 -2.002410 1 2
682
- 674 75.6677 -5.690968 -5.592777 2 1
683
- 675 75.5556 -4.398221 -4.097316 2 1
684
- 676 75.4438 -2.029287 -2.380334 1 2
685
- 677 75.4801 -3.699288 -4.073133 1 1
686
- 678 75.5162 -5.196963 -5.174656 2 2
687
- 679 75.5523 -5.119704 -4.947227 2 2
688
- 680 75.5882 -3.147749 -4.989898 1 1
689
- 681 75.4772 -3.266996 -2.157176 2 1
690
- 682 75.3666 -1.732355 -1.909596 1 2
691
- 683 75.4026 -1.562985 -2.478549 1 1
692
- 684 75.4386 -4.263360 -7.941386 1 1
693
- 685 75.4745 -4.043590 -1.915896 2 2
694
- 686 75.5102 -5.355157 -4.826468 2 2
695
- 687 75.5459 -2.846364 -1.790536 2 2
696
- 688 75.5814 -6.965519 -7.270120 1 1
697
- 689 75.6168 -2.487659 -1.798958 2 2
698
- 690 75.6522 -2.488676 -2.482813 2 2
699
- 691 75.5427 -3.859327 -3.345597 2 1
700
- 692 75.5780 -1.762353 -3.111535 1 1
701
- 693 75.6133 -4.029905 -4.454050 1 1
702
- 694 75.6484 -3.537996 -3.123510 2 2
703
- 695 75.6835 -4.514824 -5.137204 1 1
704
- 696 75.5747 -5.007354 -4.606453 2 1
705
- 697 75.6098 -2.587639 -2.635678 1 1
706
- 698 75.6447 -4.074047 -3.607794 2 2
707
- 699 75.6795 -4.040362 -5.814511 1 1
708
- 700 75.7143 -3.233794 -2.394702 2 2
709
- 701 75.6063 -1.969417 -1.631428 2 1
710
- 702 75.4986 -4.148823 -4.858560 1 2
711
- 703 75.3912 -4.637606 -3.717263 2 1
712
- 704 75.4261 -1.550582 -1.809230 1 1
713
- 705 75.4610 -2.438355 -1.893970 2 2
714
- 706 75.4958 -2.824809 -2.663210 2 2
715
- 707 75.3890 -4.759365 -4.706255 2 1
716
- 708 75.4237 -4.020359 -3.184421 2 2
717
- 709 75.4584 -3.492242 -2.902037 2 2
718
- 710 75.4930 -2.602914 -3.695196 1 1
719
- 711 75.5274 -2.396650 -2.082987 2 2
720
- 712 75.4213 -2.936412 -3.736639 1 2
721
- 713 75.3156 -3.075726 -3.210787 1 2
722
- 714 75.3501 -2.822523 -2.385545 2 2
723
- 715 75.3846 -5.924175 -4.708241 2 2
724
- 716 75.4190 -4.585432 -3.729401 2 2
725
- 717 75.4533 -2.186985 -0.626700 2 2
726
- 718 75.4875 -3.395630 -3.636514 1 1
727
- 719 75.3825 -4.982621 -4.567061 2 1
728
- 720 75.2778 -1.553466 -4.839521 1 2
729
- 721 75.1734 -4.372953 -4.940229 1 2
730
- 722 75.0693 -2.554374 -2.494893 2 1
731
- 723 75.1037 -2.974472 -2.982300 1 1
732
- 724 75.0000 -3.041473 -3.522211 1 2
733
- 725 74.8966 -2.755078 -2.887428 1 2
734
- 726 74.7934 -4.212942 -4.296736 1 2
735
- 727 74.8281 -2.953955 -2.199020 2 2
736
- 728 74.8626 -4.481767 -4.563629 1 1
737
- 729 74.8971 -1.845540 -2.693937 1 1
738
- 730 74.7945 -3.383373 -3.217380 2 1
739
- 731 74.8290 -3.312839 -3.966016 1 1
740
- 732 74.8634 -3.142950 -2.355440 2 2
741
- 733 74.7613 -3.263499 -3.656607 1 2
742
- 734 74.7956 -0.127934 -7.714783 1 1
743
- 735 74.6939 -2.842407 -2.573126 2 1
744
- 736 74.7283 -2.917937 -3.738621 1 1
745
- 737 74.6269 -4.187613 -3.899069 2 1
746
- 738 74.6612 -4.414064 -4.225746 2 2
747
- 739 74.6955 -0.311135 -8.679609 1 1
748
- 740 74.7297 -4.156615 -3.278522 2 2
749
- 741 74.6289 -4.029593 -3.781061 2 1
750
- 742 74.6631 -2.023935 -1.491524 2 2
751
- 743 74.6972 -3.399635 -3.239281 2 2
752
- 744 74.7312 -4.994440 -3.905042 2 2
753
- 745 74.7651 -3.551758 -4.438610 1 1
754
- 746 74.6649 -6.032127 -5.851011 2 1
755
- 747 74.6988 -4.775389 -5.147573 1 1
756
- 748 74.7326 -3.364788 -3.011202 2 2
757
- 749 74.7664 -3.140563 -3.592016 1 1
758
- 750 74.8000 -1.344720 -8.287935 1 1
759
 
760
- Final Winogrande score(750 tasks): 74.8000 +/- 1.5864
 
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 36 key-value pairs and 292 tensors from ./Watt-Tool-8B-F16.gguf (version GGUF V3 (latest))
4
 
5
+ Final Winogrande score(750 tasks): 74.0000 +/- 1.6027
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
 
7
+ llama_perf_context_print: load time = 621.33 ms
8
+ llama_perf_context_print: prompt eval time = 86368.81 ms / 22255 tokens ( 3.88 ms per token, 257.67 tokens per second)
9
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
10
+ llama_perf_context_print: total time = 87630.93 ms / 22256 tokens
11
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-Q4_K_M-naive.arc ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 42 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q4_K_M-naive.gguf (version GGUF V3 (latest))
4
+
5
+ Final result: 62.5668 +/- 1.7707
6
+ Random chance: 25.0251 +/- 1.5848
7
+
8
+
9
+ llama_perf_context_print: load time = 707.57 ms
10
+ llama_perf_context_print: prompt eval time = 164606.88 ms / 36539 tokens ( 4.50 ms per token, 221.98 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 166874.76 ms / 36540 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-Q4_K_M-naive.hsw ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 42 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q4_K_M-naive.gguf (version GGUF V3 (latest))
4
+
5
+ 750 77.73333333
6
+
7
+
8
+ llama_perf_context_print: load time = 306.76 ms
9
+ llama_perf_context_print: prompt eval time = 436291.37 ms / 122836 tokens ( 3.55 ms per token, 281.55 tokens per second)
10
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
11
+ llama_perf_context_print: total time = 441964.91 ms / 122837 tokens
12
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-Q4_K_M-naive.mmlu ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 42 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q4_K_M-naive.gguf (version GGUF V3 (latest))
4
+
5
+ Final result: 42.0000 +/- 1.8034
6
+ Random chance: 25.0000 +/- 1.5822
7
+
8
+
9
+ llama_perf_context_print: load time = 304.34 ms
10
+ llama_perf_context_print: prompt eval time = 262641.92 ms / 69673 tokens ( 3.77 ms per token, 265.28 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 265464.52 ms / 69674 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-Q4_K_M-naive.ppx ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ====== Perplexity statistics ======
2
+ Mean PPL(Q) : 7.409510 ± 0.046740
3
+ Mean PPL(base) : 7.237090 ± 0.045539
4
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.65%
5
+ Mean ln(PPL(Q)/PPL(base)) : 0.023545 ± 0.000530
6
+ Mean PPL(Q)/PPL(base) : 1.023825 ± 0.000543
7
+ Mean PPL(Q)-PPL(base) : 0.172420 ± 0.004061
8
+
9
+ ====== KL divergence statistics ======
10
+ Mean KLD: 0.017663 ± 0.000107
11
+ Maximum KLD: 5.749704
12
+ 99.9% KLD: 0.447724
13
+ 99.0% KLD: 0.139140
14
+ 99.0% KLD: 0.139140
15
+ Median KLD: 0.010320
16
+ 10.0% KLD: 0.000617
17
+ 5.0% KLD: 0.000201
18
+ 1.0% KLD: 0.000027
19
+ Minimum KLD: -0.000129
20
+
21
+ ====== Token probability statistics ======
22
+ Mean Δp: -0.531 ± 0.010 %
23
+ Maximum Δp: 55.716%
24
+ 99.9% Δp: 17.458%
25
+ 99.0% Δp: 8.256%
26
+ 95.0% Δp: 3.790%
27
+ 90.0% Δp: 2.138%
28
+ 75.0% Δp: 0.367%
29
+ Median Δp: -0.034%
30
+ 25.0% Δp: -1.129%
31
+ 10.0% Δp: -3.654%
32
+ 5.0% Δp: -5.855%
33
+ 1.0% Δp: -12.744%
34
+ 0.1% Δp: -31.910%
35
+ Minimum Δp: -99.362%
36
+ RMS Δp : 3.658 ± 0.032 %
37
+ Same top p: 93.743 ± 0.064 %
scores/Watt-Tool-8B-Q4_K_M-naive.tqa ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 42 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q4_K_M-naive.gguf (version GGUF V3 (latest))
4
+
5
+ Final result: 36.8098 +/- 2.6753
6
+ Random chance: 28.5214 +/- 2.5046
7
+
8
+
9
+ llama_perf_context_print: load time = 306.51 ms
10
+ llama_perf_context_print: prompt eval time = 78347.98 ms / 17655 tokens ( 4.44 ms per token, 225.34 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 79593.31 ms / 17656 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-Q4_K_M-naive.wng ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 42 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q4_K_M-naive.gguf (version GGUF V3 (latest))
4
+
5
+ Final Winogrande score(750 tasks): 73.6000 +/- 1.6106
6
+
7
+ llama_perf_context_print: load time = 295.82 ms
8
+ llama_perf_context_print: prompt eval time = 90900.17 ms / 22246 tokens ( 4.09 ms per token, 244.73 tokens per second)
9
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
10
+ llama_perf_context_print: total time = 92103.74 ms / 22247 tokens
11
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-iq3_m.arc CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 869 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 869 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 100.00000000
13
- 3 100.00000000
14
- 4 75.00000000
15
- 5 80.00000000
16
- 6 83.33333333
17
- 7 85.71428571
18
- 8 87.50000000
19
- 9 77.77777778
20
- 10 80.00000000
21
- 11 72.72727273
22
- 12 75.00000000
23
- 13 69.23076923
24
- 14 71.42857143
25
- 15 73.33333333
26
- 16 75.00000000
27
- 17 76.47058824
28
- 18 72.22222222
29
- 19 68.42105263
30
- 20 65.00000000
31
- 21 66.66666667
32
- 22 68.18181818
33
- 23 69.56521739
34
- 24 70.83333333
35
- 25 68.00000000
36
- 26 65.38461538
37
- 27 66.66666667
38
- 28 64.28571429
39
- 29 65.51724138
40
- 30 63.33333333
41
- 31 61.29032258
42
- 32 59.37500000
43
- 33 57.57575758
44
- 34 55.88235294
45
- 35 57.14285714
46
- 36 55.55555556
47
- 37 56.75675676
48
- 38 55.26315789
49
- 39 56.41025641
50
- 40 55.00000000
51
- 41 56.09756098
52
- 42 54.76190476
53
- 43 53.48837209
54
- 44 52.27272727
55
- 45 53.33333333
56
- 46 54.34782609
57
- 47 55.31914894
58
- 48 56.25000000
59
- 49 57.14285714
60
- 50 56.00000000
61
- 51 54.90196078
62
- 52 55.76923077
63
- 53 54.71698113
64
- 54 53.70370370
65
- 55 52.72727273
66
- 56 53.57142857
67
- 57 54.38596491
68
- 58 55.17241379
69
- 59 55.93220339
70
- 60 56.66666667
71
- 61 57.37704918
72
- 62 56.45161290
73
- 63 57.14285714
74
- 64 57.81250000
75
- 65 56.92307692
76
- 66 57.57575758
77
- 67 58.20895522
78
- 68 58.82352941
79
- 69 59.42028986
80
- 70 58.57142857
81
- 71 57.74647887
82
- 72 56.94444444
83
- 73 57.53424658
84
- 74 58.10810811
85
- 75 58.66666667
86
- 76 59.21052632
87
- 77 59.74025974
88
- 78 60.25641026
89
- 79 59.49367089
90
- 80 60.00000000
91
- 81 59.25925926
92
- 82 59.75609756
93
- 83 59.03614458
94
- 84 59.52380952
95
- 85 60.00000000
96
- 86 60.46511628
97
- 87 60.91954023
98
- 88 61.36363636
99
- 89 61.79775281
100
- 90 62.22222222
101
- 91 62.63736264
102
- 92 63.04347826
103
- 93 62.36559140
104
- 94 61.70212766
105
- 95 62.10526316
106
- 96 62.50000000
107
- 97 61.85567010
108
- 98 61.22448980
109
- 99 61.61616162
110
- 100 62.00000000
111
- 101 61.38613861
112
- 102 61.76470588
113
- 103 61.16504854
114
- 104 60.57692308
115
- 105 60.95238095
116
- 106 60.37735849
117
- 107 60.74766355
118
- 108 60.18518519
119
- 109 59.63302752
120
- 110 60.00000000
121
- 111 60.36036036
122
- 112 60.71428571
123
- 113 60.17699115
124
- 114 60.52631579
125
- 115 60.86956522
126
- 116 61.20689655
127
- 117 60.68376068
128
- 118 61.01694915
129
- 119 60.50420168
130
- 120 60.83333333
131
- 121 60.33057851
132
- 122 60.65573770
133
- 123 60.97560976
134
- 124 60.48387097
135
- 125 60.80000000
136
- 126 61.11111111
137
- 127 61.41732283
138
- 128 61.71875000
139
- 129 61.24031008
140
- 130 60.76923077
141
- 131 61.06870229
142
- 132 61.36363636
143
- 133 61.65413534
144
- 134 61.19402985
145
- 135 60.74074074
146
- 136 60.29411765
147
- 137 60.58394161
148
- 138 60.86956522
149
- 139 61.15107914
150
- 140 61.42857143
151
- 141 60.99290780
152
- 142 61.26760563
153
- 143 61.53846154
154
- 144 61.11111111
155
- 145 61.37931034
156
- 146 60.95890411
157
- 147 61.22448980
158
- 148 61.48648649
159
- 149 61.74496644
160
- 150 62.00000000
161
- 151 61.58940397
162
- 152 61.84210526
163
- 153 62.09150327
164
- 154 61.68831169
165
- 155 61.29032258
166
- 156 61.53846154
167
- 157 61.78343949
168
- 158 61.39240506
169
- 159 61.63522013
170
- 160 61.25000000
171
- 161 61.49068323
172
- 162 61.72839506
173
- 163 61.34969325
174
- 164 61.58536585
175
- 165 61.81818182
176
- 166 62.04819277
177
- 167 61.67664671
178
- 168 61.90476190
179
- 169 61.53846154
180
- 170 61.76470588
181
- 171 61.98830409
182
- 172 61.62790698
183
- 173 61.84971098
184
- 174 62.06896552
185
- 175 61.71428571
186
- 176 61.93181818
187
- 177 62.14689266
188
- 178 62.35955056
189
- 179 62.56983240
190
- 180 62.77777778
191
- 181 62.43093923
192
- 182 62.63736264
193
- 183 62.29508197
194
- 184 61.95652174
195
- 185 62.16216216
196
- 186 62.36559140
197
- 187 62.56684492
198
- 188 62.23404255
199
- 189 61.90476190
200
- 190 61.57894737
201
- 191 61.78010471
202
- 192 61.97916667
203
- 193 62.17616580
204
- 194 62.37113402
205
- 195 62.56410256
206
- 196 62.75510204
207
- 197 62.94416244
208
- 198 62.62626263
209
- 199 62.31155779
210
- 200 62.50000000
211
- 201 62.68656716
212
- 202 62.37623762
213
- 203 62.56157635
214
- 204 62.74509804
215
- 205 62.43902439
216
- 206 62.13592233
217
- 207 61.83574879
218
- 208 62.01923077
219
- 209 62.20095694
220
- 210 62.38095238
221
- 211 62.55924171
222
- 212 62.26415094
223
- 213 62.44131455
224
- 214 62.61682243
225
- 215 62.79069767
226
- 216 62.50000000
227
- 217 62.67281106
228
- 218 62.38532110
229
- 219 62.55707763
230
- 220 62.27272727
231
- 221 62.44343891
232
- 222 62.16216216
233
- 223 61.88340807
234
- 224 62.05357143
235
- 225 61.77777778
236
- 226 61.50442478
237
- 227 61.67400881
238
- 228 61.84210526
239
- 229 62.00873362
240
- 230 62.17391304
241
- 231 61.90476190
242
- 232 61.63793103
243
- 233 61.80257511
244
- 234 61.53846154
245
- 235 61.27659574
246
- 236 61.01694915
247
- 237 60.75949367
248
- 238 60.92436975
249
- 239 61.08786611
250
- 240 61.25000000
251
- 241 61.41078838
252
- 242 61.57024793
253
- 243 61.72839506
254
- 244 61.47540984
255
- 245 61.63265306
256
- 246 61.38211382
257
- 247 61.53846154
258
- 248 61.69354839
259
- 249 61.44578313
260
- 250 61.60000000
261
- 251 61.75298805
262
- 252 61.90476190
263
- 253 62.05533597
264
- 254 61.81102362
265
- 255 61.96078431
266
- 256 62.10937500
267
- 257 61.86770428
268
- 258 62.01550388
269
- 259 61.77606178
270
- 260 61.92307692
271
- 261 62.06896552
272
- 262 62.21374046
273
- 263 62.35741445
274
- 264 62.50000000
275
- 265 62.64150943
276
- 266 62.40601504
277
- 267 62.54681648
278
- 268 62.31343284
279
- 269 62.45353160
280
- 270 62.59259259
281
- 271 62.73062731
282
- 272 62.50000000
283
- 273 62.27106227
284
- 274 62.04379562
285
- 275 61.81818182
286
- 276 61.95652174
287
- 277 62.09386282
288
- 278 61.87050360
289
- 279 62.00716846
290
- 280 62.14285714
291
- 281 62.27758007
292
- 282 62.41134752
293
- 283 62.19081272
294
- 284 62.32394366
295
- 285 62.45614035
296
- 286 62.58741259
297
- 287 62.36933798
298
- 288 62.50000000
299
- 289 62.28373702
300
- 290 62.41379310
301
- 291 62.54295533
302
- 292 62.67123288
303
- 293 62.79863481
304
- 294 62.58503401
305
- 295 62.71186441
306
- 296 62.83783784
307
- 297 62.96296296
308
- 298 63.08724832
309
- 299 63.21070234
310
- 300 63.33333333
311
- 301 63.12292359
312
- 302 63.24503311
313
- 303 63.36633663
314
- 304 63.15789474
315
- 305 63.27868852
316
- 306 63.07189542
317
- 307 63.19218241
318
- 308 63.31168831
319
- 309 63.43042071
320
- 310 63.54838710
321
- 311 63.66559486
322
- 312 63.78205128
323
- 313 63.89776358
324
- 314 63.69426752
325
- 315 63.49206349
326
- 316 63.60759494
327
- 317 63.72239748
328
- 318 63.52201258
329
- 319 63.32288401
330
- 320 63.12500000
331
- 321 63.23987539
332
- 322 63.35403727
333
- 323 63.46749226
334
- 324 63.27160494
335
- 325 63.38461538
336
- 326 63.49693252
337
- 327 63.30275229
338
- 328 63.41463415
339
- 329 63.52583587
340
- 330 63.33333333
341
- 331 63.44410876
342
- 332 63.55421687
343
- 333 63.66366366
344
- 334 63.77245509
345
- 335 63.58208955
346
- 336 63.69047619
347
- 337 63.79821958
348
- 338 63.60946746
349
- 339 63.71681416
350
- 340 63.82352941
351
- 341 63.92961877
352
- 342 63.74269006
353
- 343 63.84839650
354
- 344 63.95348837
355
- 345 63.76811594
356
- 346 63.87283237
357
- 347 63.68876081
358
- 348 63.50574713
359
- 349 63.61031519
360
- 350 63.42857143
361
- 351 63.53276353
362
- 352 63.63636364
363
- 353 63.73937677
364
- 354 63.55932203
365
- 355 63.38028169
366
- 356 63.48314607
367
- 357 63.58543417
368
- 358 63.68715084
369
- 359 63.50974930
370
- 360 63.61111111
371
- 361 63.71191136
372
- 362 63.81215470
373
- 363 63.63636364
374
- 364 63.46153846
375
- 365 63.56164384
376
- 366 63.38797814
377
- 367 63.21525886
378
- 368 63.31521739
379
- 369 63.41463415
380
- 370 63.51351351
381
- 371 63.34231806
382
- 372 63.17204301
383
- 373 63.27077748
384
- 374 63.36898396
385
- 375 63.46666667
386
- 376 63.29787234
387
- 377 63.12997347
388
- 378 63.22751323
389
- 379 63.06068602
390
- 380 63.15789474
391
- 381 62.99212598
392
- 382 62.82722513
393
- 383 62.66318538
394
- 384 62.76041667
395
- 385 62.85714286
396
- 386 62.95336788
397
- 387 62.79069767
398
- 388 62.62886598
399
- 389 62.46786632
400
- 390 62.30769231
401
- 391 62.40409207
402
- 392 62.50000000
403
- 393 62.59541985
404
- 394 62.69035533
405
- 395 62.78481013
406
- 396 62.62626263
407
- 397 62.72040302
408
- 398 62.56281407
409
- 399 62.65664160
410
- 400 62.75000000
411
- 401 62.59351621
412
- 402 62.43781095
413
- 403 62.53101737
414
- 404 62.62376238
415
- 405 62.71604938
416
- 406 62.80788177
417
- 407 62.89926290
418
- 408 62.99019608
419
- 409 62.83618582
420
- 410 62.92682927
421
- 411 63.01703163
422
- 412 63.10679612
423
- 413 62.95399516
424
- 414 63.04347826
425
- 415 62.89156627
426
- 416 62.98076923
427
- 417 62.82973621
428
- 418 62.91866029
429
- 419 62.76849642
430
- 420 62.85714286
431
- 421 62.70783848
432
- 422 62.79620853
433
- 423 62.88416076
434
- 424 62.73584906
435
- 425 62.82352941
436
- 426 62.67605634
437
- 427 62.76346604
438
- 428 62.61682243
439
- 429 62.70396270
440
- 430 62.55813953
441
- 431 62.64501160
442
- 432 62.73148148
443
- 433 62.58660508
444
- 434 62.67281106
445
- 435 62.75862069
446
- 436 62.84403670
447
- 437 62.92906178
448
- 438 62.78538813
449
- 439 62.87015945
450
- 440 62.72727273
451
- 441 62.58503401
452
- 442 62.44343891
453
- 443 62.30248307
454
- 444 62.16216216
455
- 445 62.24719101
456
- 446 62.33183857
457
- 447 62.41610738
458
- 448 62.50000000
459
- 449 62.58351893
460
- 450 62.44444444
461
- 451 62.30598670
462
- 452 62.38938053
463
- 453 62.25165563
464
- 454 62.33480176
465
- 455 62.19780220
466
- 456 62.28070175
467
- 457 62.36323851
468
- 458 62.44541485
469
- 459 62.52723312
470
- 460 62.60869565
471
- 461 62.68980477
472
- 462 62.77056277
473
- 463 62.63498920
474
- 464 62.50000000
475
- 465 62.58064516
476
- 466 62.66094421
477
- 467 62.74089936
478
- 468 62.60683761
479
- 469 62.47334755
480
- 470 62.55319149
481
- 471 62.63269639
482
- 472 62.50000000
483
- 473 62.36786469
484
- 474 62.44725738
485
- 475 62.52631579
486
- 476 62.60504202
487
- 477 62.68343816
488
- 478 62.76150628
489
- 479 62.83924843
490
- 480 62.91666667
491
- 481 62.78586279
492
- 482 62.86307054
493
- 483 62.93995859
494
- 484 63.01652893
495
- 485 63.09278351
496
- 486 63.16872428
497
- 487 63.03901437
498
- 488 63.11475410
499
- 489 62.98568507
500
- 490 62.85714286
501
- 491 62.72912424
502
- 492 62.80487805
503
- 493 62.67748479
504
- 494 62.75303644
505
- 495 62.62626263
506
- 496 62.70161290
507
- 497 62.57545272
508
- 498 62.65060241
509
- 499 62.72545090
510
- 500 62.80000000
511
- 501 62.67465070
512
- 502 62.74900398
513
- 503 62.82306163
514
- 504 62.89682540
515
- 505 62.97029703
516
- 506 62.84584980
517
- 507 62.72189349
518
- 508 62.59842520
519
- 509 62.47544204
520
- 510 62.35294118
521
- 511 62.23091977
522
- 512 62.30468750
523
- 513 62.18323587
524
- 514 62.25680934
525
- 515 62.13592233
526
- 516 62.20930233
527
- 517 62.28239845
528
- 518 62.35521236
529
- 519 62.42774566
530
- 520 62.30769231
531
- 521 62.38003839
532
- 522 62.45210728
533
- 523 62.33269598
534
- 524 62.21374046
535
- 525 62.09523810
536
- 526 61.97718631
537
- 527 61.85958254
538
- 528 61.74242424
539
- 529 61.62570888
540
- 530 61.69811321
541
- 531 61.58192090
542
- 532 61.65413534
543
- 533 61.72607880
544
- 534 61.61048689
545
- 535 61.49532710
546
- 536 61.56716418
547
- 537 61.63873371
548
- 538 61.71003717
549
- 539 61.78107607
550
- 540 61.66666667
551
- 541 61.73752311
552
- 542 61.80811808
553
- 543 61.69429098
554
- 544 61.76470588
555
- 545 61.83486239
556
- 546 61.72161172
557
- 547 61.60877514
558
- 548 61.67883212
559
- 549 61.74863388
560
- 550 61.81818182
561
- 551 61.70598911
562
- 552 61.59420290
563
- 553 61.66365280
564
- 554 61.73285199
565
- 555 61.80180180
566
- 556 61.69064748
567
- 557 61.75942549
568
- 558 61.64874552
569
- 559 61.71735242
570
- 560 61.78571429
571
- 561 61.85383244
572
- 562 61.92170819
573
- 563 61.98934281
574
- 564 62.05673759
575
- 565 61.94690265
576
- 566 62.01413428
577
- 567 61.90476190
578
- 568 61.97183099
579
- 569 61.86291740
580
- 570 61.75438596
581
- 571 61.82136602
582
- 572 61.71328671
583
- 573 61.78010471
584
- 574 61.84668990
585
- 575 61.73913043
586
- 576 61.80555556
587
- 577 61.87175043
588
- 578 61.93771626
589
- 579 62.00345423
590
- 580 61.89655172
591
- 581 61.79001721
592
- 582 61.68384880
593
- 583 61.74957118
594
- 584 61.81506849
595
- 585 61.88034188
596
- 586 61.94539249
597
- 587 62.01022147
598
- 588 62.07482993
599
- 589 62.13921902
600
- 590 62.20338983
601
- 591 62.26734349
602
- 592 62.16216216
603
- 593 62.05733558
604
- 594 62.12121212
605
- 595 62.01680672
606
- 596 61.91275168
607
- 597 61.80904523
608
- 598 61.70568562
609
- 599 61.76961603
610
- 600 61.66666667
611
- 601 61.73044925
612
- 602 61.62790698
613
- 603 61.52570481
614
- 604 61.42384106
615
- 605 61.48760331
616
- 606 61.55115512
617
- 607 61.61449753
618
- 608 61.67763158
619
- 609 61.57635468
620
- 610 61.63934426
621
- 611 61.53846154
622
- 612 61.60130719
623
- 613 61.50081566
624
- 614 61.56351792
625
- 615 61.46341463
626
- 616 61.36363636
627
- 617 61.42625608
628
- 618 61.48867314
629
- 619 61.38933764
630
- 620 61.45161290
631
- 621 61.51368760
632
- 622 61.41479100
633
- 623 61.31621188
634
- 624 61.21794872
635
- 625 61.12000000
636
- 626 61.02236422
637
- 627 61.08452951
638
- 628 61.14649682
639
- 629 61.04928458
640
- 630 61.11111111
641
- 631 61.17274168
642
- 632 61.07594937
643
- 633 60.97946288
644
- 634 61.04100946
645
- 635 61.10236220
646
- 636 61.16352201
647
- 637 61.06750392
648
- 638 61.12852665
649
- 639 61.18935837
650
- 640 61.09375000
651
- 641 61.15444618
652
- 642 61.05919003
653
- 643 60.96423017
654
- 644 61.02484472
655
- 645 60.93023256
656
- 646 60.99071207
657
- 647 61.05100464
658
- 648 61.11111111
659
- 649 61.01694915
660
- 650 61.07692308
661
- 651 61.13671275
662
- 652 61.19631902
663
- 653 61.10260337
664
- 654 61.16207951
665
- 655 61.06870229
666
- 656 60.97560976
667
- 657 60.88280061
668
- 658 60.79027356
669
- 659 60.69802731
670
- 660 60.75757576
671
- 661 60.66565809
672
- 662 60.72507553
673
- 663 60.78431373
674
- 664 60.84337349
675
- 665 60.90225564
676
- 666 60.81081081
677
- 667 60.86956522
678
- 668 60.77844311
679
- 669 60.68759342
680
- 670 60.74626866
681
- 671 60.65573770
682
- 672 60.71428571
683
- 673 60.62407132
684
- 674 60.53412463
685
- 675 60.44444444
686
- 676 60.35502959
687
- 677 60.41358936
688
- 678 60.47197640
689
- 679 60.53019146
690
- 680 60.58823529
691
- 681 60.49926579
692
- 682 60.41055718
693
- 683 60.32210835
694
- 684 60.38011696
695
- 685 60.43795620
696
- 686 60.34985423
697
- 687 60.26200873
698
- 688 60.31976744
699
- 689 60.23222061
700
- 690 60.28985507
701
- 691 60.34732272
702
- 692 60.40462428
703
- 693 60.46176046
704
- 694 60.51873199
705
- 695 60.43165468
706
- 696 60.34482759
707
- 697 60.25824964
708
- 698 60.17191977
709
- 699 60.22889843
710
- 700 60.28571429
711
- 701 60.34236805
712
- 702 60.25641026
713
- 703 60.31294452
714
- 704 60.36931818
715
- 705 60.42553191
716
- 706 60.48158640
717
- 707 60.53748232
718
- 708 60.59322034
719
- 709 60.64880113
720
- 710 60.70422535
721
- 711 60.75949367
722
- 712 60.81460674
723
- 713 60.86956522
724
- 714 60.78431373
725
- 715 60.69930070
726
- 716 60.75418994
727
- 717 60.66945607
728
- 718 60.58495822
729
- 719 60.50069541
730
- 720 60.55555556
731
- 721 60.61026352
732
- 722 60.66481994
733
- 723 60.71922545
734
- 724 60.63535912
735
- 725 60.68965517
736
- 726 60.60606061
737
- 727 60.52269601
738
- 728 60.57692308
739
- 729 60.63100137
740
- 730 60.68493151
741
- 731 60.73871409
742
- 732 60.79234973
743
- 733 60.84583902
744
- 734 60.89918256
745
- 735 60.95238095
746
- 736 61.00543478
747
- 737 60.92265943
748
- 738 60.97560976
749
- 739 60.89309878
750
- 740 60.81081081
751
- 741 60.86369771
752
- 742 60.78167116
753
- 743 60.83445491
754
- 744 60.75268817
755
- 745 60.80536913
756
- 746 60.85790885
757
- 747 60.77643909
758
- 748 60.69518717
759
- 749 60.61415220
760
- 750 60.66666667
761
 
762
- Final result: 60.6667 +/- 1.7849
763
- Random chance: 25.0083 +/- 1.5824
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-IQ3_M.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 57.6203 +/- 1.8080
6
+ Random chance: 25.0251 +/- 1.5848
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 1615.23 ms
10
+ llama_perf_context_print: prompt eval time = 160568.89 ms / 36381 tokens ( 4.41 ms per token, 226.58 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 162816.72 ms / 36382 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-iq3_m.hsw CHANGED
@@ -1,760 +1,12 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- hellaswag_score : loaded 10042 tasks from prompt.
6
- ================================= is_spm = 0
7
- hellaswag_score : selecting 750 randomized tasks.
8
- hellaswag_score : calculating hellaswag score over selected tasks.
9
 
10
- task acc_norm
11
- 1 0.00000000
12
- 2 50.00000000
13
- 3 66.66666667
14
- 4 75.00000000
15
- 5 80.00000000
16
- 6 83.33333333
17
- 7 71.42857143
18
- 8 75.00000000
19
- 9 77.77777778
20
- 10 80.00000000
21
- 11 81.81818182
22
- 12 75.00000000
23
- 13 76.92307692
24
- 14 78.57142857
25
- 15 80.00000000
26
- 16 81.25000000
27
- 17 76.47058824
28
- 18 77.77777778
29
- 19 73.68421053
30
- 20 75.00000000
31
- 21 76.19047619
32
- 22 77.27272727
33
- 23 78.26086957
34
- 24 75.00000000
35
- 25 72.00000000
36
- 26 69.23076923
37
- 27 66.66666667
38
- 28 67.85714286
39
- 29 65.51724138
40
- 30 63.33333333
41
- 31 64.51612903
42
- 32 62.50000000
43
- 33 63.63636364
44
- 34 64.70588235
45
- 35 65.71428571
46
- 36 66.66666667
47
- 37 64.86486486
48
- 38 65.78947368
49
- 39 66.66666667
50
- 40 65.00000000
51
- 41 65.85365854
52
- 42 64.28571429
53
- 43 65.11627907
54
- 44 65.90909091
55
- 45 64.44444444
56
- 46 65.21739130
57
- 47 65.95744681
58
- 48 66.66666667
59
- 49 67.34693878
60
- 50 66.00000000
61
- 51 66.66666667
62
- 52 65.38461538
63
- 53 64.15094340
64
- 54 62.96296296
65
- 55 61.81818182
66
- 56 60.71428571
67
- 57 61.40350877
68
- 58 62.06896552
69
- 59 62.71186441
70
- 60 63.33333333
71
- 61 63.93442623
72
- 62 64.51612903
73
- 63 65.07936508
74
- 64 65.62500000
75
- 65 66.15384615
76
- 66 65.15151515
77
- 67 65.67164179
78
- 68 66.17647059
79
- 69 65.21739130
80
- 70 65.71428571
81
- 71 64.78873239
82
- 72 65.27777778
83
- 73 65.75342466
84
- 74 66.21621622
85
- 75 65.33333333
86
- 76 65.78947368
87
- 77 66.23376623
88
- 78 66.66666667
89
- 79 67.08860759
90
- 80 67.50000000
91
- 81 67.90123457
92
- 82 68.29268293
93
- 83 68.67469880
94
- 84 69.04761905
95
- 85 69.41176471
96
- 86 69.76744186
97
- 87 70.11494253
98
- 88 70.45454545
99
- 89 69.66292135
100
- 90 70.00000000
101
- 91 70.32967033
102
- 92 69.56521739
103
- 93 69.89247312
104
- 94 70.21276596
105
- 95 70.52631579
106
- 96 70.83333333
107
- 97 71.13402062
108
- 98 71.42857143
109
- 99 71.71717172
110
- 100 71.00000000
111
- 101 71.28712871
112
- 102 70.58823529
113
- 103 69.90291262
114
- 104 70.19230769
115
- 105 70.47619048
116
- 106 70.75471698
117
- 107 71.02803738
118
- 108 71.29629630
119
- 109 70.64220183
120
- 110 70.90909091
121
- 111 71.17117117
122
- 112 71.42857143
123
- 113 71.68141593
124
- 114 71.92982456
125
- 115 72.17391304
126
- 116 71.55172414
127
- 117 70.94017094
128
- 118 71.18644068
129
- 119 71.42857143
130
- 120 70.83333333
131
- 121 71.07438017
132
- 122 71.31147541
133
- 123 71.54471545
134
- 124 71.77419355
135
- 125 72.00000000
136
- 126 72.22222222
137
- 127 72.44094488
138
- 128 71.87500000
139
- 129 72.09302326
140
- 130 71.53846154
141
- 131 71.75572519
142
- 132 71.96969697
143
- 133 72.18045113
144
- 134 72.38805970
145
- 135 72.59259259
146
- 136 72.79411765
147
- 137 72.99270073
148
- 138 73.18840580
149
- 139 72.66187050
150
- 140 72.85714286
151
- 141 73.04964539
152
- 142 73.23943662
153
- 143 73.42657343
154
- 144 73.61111111
155
- 145 73.79310345
156
- 146 73.97260274
157
- 147 74.14965986
158
- 148 73.64864865
159
- 149 73.82550336
160
- 150 74.00000000
161
- 151 73.50993377
162
- 152 73.02631579
163
- 153 73.20261438
164
- 154 73.37662338
165
- 155 73.54838710
166
- 156 73.07692308
167
- 157 72.61146497
168
- 158 72.78481013
169
- 159 72.32704403
170
- 160 72.50000000
171
- 161 72.67080745
172
- 162 72.83950617
173
- 163 73.00613497
174
- 164 73.17073171
175
- 165 73.33333333
176
- 166 73.49397590
177
- 167 73.65269461
178
- 168 73.80952381
179
- 169 73.37278107
180
- 170 73.52941176
181
- 171 73.68421053
182
- 172 73.83720930
183
- 173 73.98843931
184
- 174 74.13793103
185
- 175 73.71428571
186
- 176 73.86363636
187
- 177 74.01129944
188
- 178 74.15730337
189
- 179 73.74301676
190
- 180 73.88888889
191
- 181 73.48066298
192
- 182 73.62637363
193
- 183 73.77049180
194
- 184 73.36956522
195
- 185 73.51351351
196
- 186 73.11827957
197
- 187 72.72727273
198
- 188 72.87234043
199
- 189 72.48677249
200
- 190 72.63157895
201
- 191 72.77486911
202
- 192 72.91666667
203
- 193 72.53886010
204
- 194 72.68041237
205
- 195 72.30769231
206
- 196 72.44897959
207
- 197 72.08121827
208
- 198 71.71717172
209
- 199 71.85929648
210
- 200 72.00000000
211
- 201 72.13930348
212
- 202 72.27722772
213
- 203 72.41379310
214
- 204 72.54901961
215
- 205 72.68292683
216
- 206 72.81553398
217
- 207 72.94685990
218
- 208 72.59615385
219
- 209 72.72727273
220
- 210 72.85714286
221
- 211 72.98578199
222
- 212 73.11320755
223
- 213 73.23943662
224
- 214 73.36448598
225
- 215 73.02325581
226
- 216 73.14814815
227
- 217 72.81105991
228
- 218 72.93577982
229
- 219 73.05936073
230
- 220 73.18181818
231
- 221 72.85067873
232
- 222 72.97297297
233
- 223 73.09417040
234
- 224 73.21428571
235
- 225 73.33333333
236
- 226 73.45132743
237
- 227 73.12775330
238
- 228 73.24561404
239
- 229 73.36244541
240
- 230 73.47826087
241
- 231 73.16017316
242
- 232 72.84482759
243
- 233 72.96137339
244
- 234 73.07692308
245
- 235 72.76595745
246
- 236 72.45762712
247
- 237 72.57383966
248
- 238 72.68907563
249
- 239 72.80334728
250
- 240 72.91666667
251
- 241 73.02904564
252
- 242 73.14049587
253
- 243 73.25102881
254
- 244 72.95081967
255
- 245 72.65306122
256
- 246 72.35772358
257
- 247 72.46963563
258
- 248 72.58064516
259
- 249 72.69076305
260
- 250 72.80000000
261
- 251 72.90836653
262
- 252 73.01587302
263
- 253 72.72727273
264
- 254 72.83464567
265
- 255 72.94117647
266
- 256 73.04687500
267
- 257 72.76264591
268
- 258 72.86821705
269
- 259 72.97297297
270
- 260 73.07692308
271
- 261 73.18007663
272
- 262 72.90076336
273
- 263 73.00380228
274
- 264 73.10606061
275
- 265 73.20754717
276
- 266 72.93233083
277
- 267 73.03370787
278
- 268 73.13432836
279
- 269 73.23420074
280
- 270 73.33333333
281
- 271 73.06273063
282
- 272 73.16176471
283
- 273 72.89377289
284
- 274 72.99270073
285
- 275 72.72727273
286
- 276 72.82608696
287
- 277 72.92418773
288
- 278 73.02158273
289
- 279 73.11827957
290
- 280 73.21428571
291
- 281 73.30960854
292
- 282 73.40425532
293
- 283 73.49823322
294
- 284 73.59154930
295
- 285 73.33333333
296
- 286 73.42657343
297
- 287 73.51916376
298
- 288 73.61111111
299
- 289 73.70242215
300
- 290 73.79310345
301
- 291 73.88316151
302
- 292 73.63013699
303
- 293 73.72013652
304
- 294 73.80952381
305
- 295 73.89830508
306
- 296 73.98648649
307
- 297 74.07407407
308
- 298 74.16107383
309
- 299 73.91304348
310
- 300 74.00000000
311
- 301 74.08637874
312
- 302 74.17218543
313
- 303 73.92739274
314
- 304 73.68421053
315
- 305 73.77049180
316
- 306 73.85620915
317
- 307 73.61563518
318
- 308 73.70129870
319
- 309 73.78640777
320
- 310 73.87096774
321
- 311 73.95498392
322
- 312 74.03846154
323
- 313 74.12140575
324
- 314 74.20382166
325
- 315 74.28571429
326
- 316 74.36708861
327
- 317 74.13249211
328
- 318 73.89937107
329
- 319 73.66771160
330
- 320 73.75000000
331
- 321 73.52024922
332
- 322 73.29192547
333
- 323 73.37461300
334
- 324 73.45679012
335
- 325 73.53846154
336
- 326 73.61963190
337
- 327 73.39449541
338
- 328 73.47560976
339
- 329 73.55623100
340
- 330 73.63636364
341
- 331 73.71601208
342
- 332 73.79518072
343
- 333 73.87387387
344
- 334 73.65269461
345
- 335 73.73134328
346
- 336 73.80952381
347
- 337 73.59050445
348
- 338 73.37278107
349
- 339 73.45132743
350
- 340 73.52941176
351
- 341 73.60703812
352
- 342 73.68421053
353
- 343 73.76093294
354
- 344 73.83720930
355
- 345 73.91304348
356
- 346 73.69942197
357
- 347 73.77521614
358
- 348 73.56321839
359
- 349 73.35243553
360
- 350 73.42857143
361
- 351 73.50427350
362
- 352 73.57954545
363
- 353 73.65439093
364
- 354 73.72881356
365
- 355 73.80281690
366
- 356 73.87640449
367
- 357 73.94957983
368
- 358 74.02234637
369
- 359 74.09470752
370
- 360 74.16666667
371
- 361 74.23822715
372
- 362 74.30939227
373
- 363 74.38016529
374
- 364 74.45054945
375
- 365 74.52054795
376
- 366 74.59016393
377
- 367 74.65940054
378
- 368 74.72826087
379
- 369 74.79674797
380
- 370 74.86486486
381
- 371 74.93261456
382
- 372 75.00000000
383
- 373 74.79892761
384
- 374 74.86631016
385
- 375 74.93333333
386
- 376 75.00000000
387
- 377 74.80106101
388
- 378 74.86772487
389
- 379 74.93403694
390
- 380 75.00000000
391
- 381 75.06561680
392
- 382 75.13089005
393
- 383 75.19582245
394
- 384 75.26041667
395
- 385 75.32467532
396
- 386 75.38860104
397
- 387 75.45219638
398
- 388 75.25773196
399
- 389 75.32133676
400
- 390 75.38461538
401
- 391 75.19181586
402
- 392 75.25510204
403
- 393 75.31806616
404
- 394 75.38071066
405
- 395 75.44303797
406
- 396 75.50505051
407
- 397 75.56675063
408
- 398 75.62814070
409
- 399 75.68922306
410
- 400 75.75000000
411
- 401 75.81047382
412
- 402 75.87064677
413
- 403 75.68238213
414
- 404 75.74257426
415
- 405 75.55555556
416
- 406 75.61576355
417
- 407 75.67567568
418
- 408 75.73529412
419
- 409 75.79462103
420
- 410 75.60975610
421
- 411 75.42579075
422
- 412 75.48543689
423
- 413 75.54479419
424
- 414 75.60386473
425
- 415 75.42168675
426
- 416 75.48076923
427
- 417 75.53956835
428
- 418 75.59808612
429
- 419 75.65632458
430
- 420 75.71428571
431
- 421 75.77197150
432
- 422 75.82938389
433
- 423 75.88652482
434
- 424 75.94339623
435
- 425 76.00000000
436
- 426 76.05633803
437
- 427 75.87822014
438
- 428 75.93457944
439
- 429 75.75757576
440
- 430 75.81395349
441
- 431 75.63805104
442
- 432 75.46296296
443
- 433 75.28868360
444
- 434 75.34562212
445
- 435 75.17241379
446
- 436 75.22935780
447
- 437 75.05720824
448
- 438 75.11415525
449
- 439 75.17084282
450
- 440 75.00000000
451
- 441 75.05668934
452
- 442 75.11312217
453
- 443 75.16930023
454
- 444 75.22522523
455
- 445 75.28089888
456
- 446 75.33632287
457
- 447 75.39149888
458
- 448 75.44642857
459
- 449 75.50111359
460
- 450 75.55555556
461
- 451 75.38802661
462
- 452 75.22123894
463
- 453 75.27593819
464
- 454 75.33039648
465
- 455 75.38461538
466
- 456 75.43859649
467
- 457 75.49234136
468
- 458 75.54585153
469
- 459 75.59912854
470
- 460 75.65217391
471
- 461 75.70498915
472
- 462 75.75757576
473
- 463 75.80993521
474
- 464 75.86206897
475
- 465 75.69892473
476
- 466 75.75107296
477
- 467 75.80299786
478
- 468 75.64102564
479
- 469 75.47974414
480
- 470 75.31914894
481
- 471 75.37154989
482
- 472 75.42372881
483
- 473 75.47568710
484
- 474 75.52742616
485
- 475 75.57894737
486
- 476 75.63025210
487
- 477 75.47169811
488
- 478 75.52301255
489
- 479 75.36534447
490
- 480 75.41666667
491
- 481 75.25987526
492
- 482 75.31120332
493
- 483 75.36231884
494
- 484 75.20661157
495
- 485 75.05154639
496
- 486 75.10288066
497
- 487 75.15400411
498
- 488 75.00000000
499
- 489 75.05112474
500
- 490 75.10204082
501
- 491 75.15274949
502
- 492 75.20325203
503
- 493 75.25354970
504
- 494 75.30364372
505
- 495 75.35353535
506
- 496 75.40322581
507
- 497 75.45271630
508
- 498 75.50200803
509
- 499 75.35070140
510
- 500 75.40000000
511
- 501 75.24950100
512
- 502 75.29880478
513
- 503 75.34791252
514
- 504 75.39682540
515
- 505 75.44554455
516
- 506 75.49407115
517
- 507 75.54240631
518
- 508 75.59055118
519
- 509 75.63850688
520
- 510 75.68627451
521
- 511 75.73385519
522
- 512 75.58593750
523
- 513 75.43859649
524
- 514 75.48638132
525
- 515 75.53398058
526
- 516 75.38759690
527
- 517 75.24177950
528
- 518 75.28957529
529
- 519 75.33718690
530
- 520 75.38461538
531
- 521 75.23992322
532
- 522 75.28735632
533
- 523 75.33460803
534
- 524 75.38167939
535
- 525 75.42857143
536
- 526 75.28517110
537
- 527 75.33206831
538
- 528 75.37878788
539
- 529 75.42533081
540
- 530 75.47169811
541
- 531 75.51789077
542
- 532 75.56390977
543
- 533 75.60975610
544
- 534 75.65543071
545
- 535 75.70093458
546
- 536 75.74626866
547
- 537 75.79143389
548
- 538 75.83643123
549
- 539 75.69573284
550
- 540 75.74074074
551
- 541 75.78558226
552
- 542 75.83025830
553
- 543 75.69060773
554
- 544 75.73529412
555
- 545 75.77981651
556
- 546 75.82417582
557
- 547 75.86837294
558
- 548 75.91240876
559
- 549 75.95628415
560
- 550 75.81818182
561
- 551 75.86206897
562
- 552 75.72463768
563
- 553 75.76853526
564
- 554 75.81227437
565
- 555 75.67567568
566
- 556 75.71942446
567
- 557 75.58348294
568
- 558 75.62724014
569
- 559 75.67084079
570
- 560 75.71428571
571
- 561 75.75757576
572
- 562 75.80071174
573
- 563 75.84369449
574
- 564 75.88652482
575
- 565 75.75221239
576
- 566 75.61837456
577
- 567 75.66137566
578
- 568 75.70422535
579
- 569 75.57117750
580
- 570 75.61403509
581
- 571 75.65674256
582
- 572 75.69930070
583
- 573 75.74171030
584
- 574 75.78397213
585
- 575 75.82608696
586
- 576 75.86805556
587
- 577 75.90987868
588
- 578 75.95155709
589
- 579 75.99309154
590
- 580 76.03448276
591
- 581 75.90361446
592
- 582 75.94501718
593
- 583 75.98627787
594
- 584 76.02739726
595
- 585 76.06837607
596
- 586 76.10921502
597
- 587 76.14991482
598
- 588 76.02040816
599
- 589 75.89134126
600
- 590 75.93220339
601
- 591 75.97292724
602
- 592 76.01351351
603
- 593 75.88532884
604
- 594 75.75757576
605
- 595 75.79831933
606
- 596 75.83892617
607
- 597 75.87939698
608
- 598 75.91973244
609
- 599 75.79298831
610
- 600 75.83333333
611
- 601 75.87354409
612
- 602 75.91362126
613
- 603 75.95356551
614
- 604 75.99337748
615
- 605 76.03305785
616
- 606 75.90759076
617
- 607 75.94728171
618
- 608 75.82236842
619
- 609 75.86206897
620
- 610 75.90163934
621
- 611 75.94108020
622
- 612 75.98039216
623
- 613 75.85644372
624
- 614 75.89576547
625
- 615 75.93495935
626
- 616 75.81168831
627
- 617 75.68881686
628
- 618 75.72815534
629
- 619 75.76736672
630
- 620 75.80645161
631
- 621 75.84541063
632
- 622 75.88424437
633
- 623 75.92295345
634
- 624 75.96153846
635
- 625 76.00000000
636
- 626 76.03833866
637
- 627 76.07655502
638
- 628 76.11464968
639
- 629 75.99364070
640
- 630 75.87301587
641
- 631 75.91125198
642
- 632 75.79113924
643
- 633 75.67140600
644
- 634 75.70977918
645
- 635 75.74803150
646
- 636 75.78616352
647
- 637 75.82417582
648
- 638 75.86206897
649
- 639 75.89984351
650
- 640 75.78125000
651
- 641 75.81903276
652
- 642 75.85669782
653
- 643 75.89424572
654
- 644 75.93167702
655
- 645 75.81395349
656
- 646 75.85139319
657
- 647 75.88871716
658
- 648 75.92592593
659
- 649 75.80893683
660
- 650 75.84615385
661
- 651 75.88325653
662
- 652 75.92024540
663
- 653 75.95712098
664
- 654 75.99388379
665
- 655 76.03053435
666
- 656 76.06707317
667
- 657 76.10350076
668
- 658 76.13981763
669
- 659 76.02427921
670
- 660 75.90909091
671
- 661 75.94553707
672
- 662 75.98187311
673
- 663 76.01809955
674
- 664 76.05421687
675
- 665 76.09022556
676
- 666 76.12612613
677
- 667 76.16191904
678
- 668 76.19760479
679
- 669 76.23318386
680
- 670 76.26865672
681
- 671 76.30402385
682
- 672 76.33928571
683
- 673 76.37444279
684
- 674 76.40949555
685
- 675 76.44444444
686
- 676 76.47928994
687
- 677 76.51403250
688
- 678 76.54867257
689
- 679 76.58321060
690
- 680 76.61764706
691
- 681 76.65198238
692
- 682 76.68621701
693
- 683 76.72035139
694
- 684 76.75438596
695
- 685 76.78832117
696
- 686 76.82215743
697
- 687 76.85589520
698
- 688 76.88953488
699
- 689 76.92307692
700
- 690 76.95652174
701
- 691 76.98986975
702
- 692 77.02312139
703
- 693 76.91197691
704
- 694 76.94524496
705
- 695 76.97841727
706
- 696 77.01149425
707
- 697 77.04447633
708
- 698 77.07736390
709
- 699 77.11015737
710
- 700 77.14285714
711
- 701 77.17546362
712
- 702 77.20797721
713
- 703 77.24039829
714
- 704 77.27272727
715
- 705 77.30496454
716
- 706 77.33711048
717
- 707 77.36916549
718
- 708 77.40112994
719
- 709 77.43300423
720
- 710 77.46478873
721
- 711 77.35583685
722
- 712 77.38764045
723
- 713 77.41935484
724
- 714 77.45098039
725
- 715 77.34265734
726
- 716 77.37430168
727
- 717 77.40585774
728
- 718 77.43732591
729
- 719 77.46870654
730
- 720 77.50000000
731
- 721 77.53120666
732
- 722 77.56232687
733
- 723 77.59336100
734
- 724 77.62430939
735
- 725 77.65517241
736
- 726 77.68595041
737
- 727 77.71664374
738
- 728 77.74725275
739
- 729 77.77777778
740
- 730 77.80821918
741
- 731 77.83857729
742
- 732 77.73224044
743
- 733 77.62619372
744
- 734 77.65667575
745
- 735 77.55102041
746
- 736 77.58152174
747
- 737 77.47625509
748
- 738 77.37127371
749
- 739 77.26657645
750
- 740 77.29729730
751
- 741 77.32793522
752
- 742 77.35849057
753
- 743 77.38896366
754
- 744 77.41935484
755
- 745 77.44966443
756
- 746 77.47989276
757
- 747 77.37617135
758
- 748 77.40641711
759
- 749 77.43658211
760
- 750 77.46666667
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-IQ3_M.gguf (version GGUF V3 (latest))
4
 
5
+ 750 78.80000000
 
 
 
 
6
 
7
+
8
+ llama_perf_context_print: load time = 279.18 ms
9
+ llama_perf_context_print: prompt eval time = 433163.50 ms / 124534 tokens ( 3.48 ms per token, 287.50 tokens per second)
10
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
11
+ llama_perf_context_print: total time = 438833.07 ms / 124535 tokens
12
+ ggml_metal_free: deallocating
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scores/Watt-Tool-8B-iq3_m.mmlu CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 1548 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 1548 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 50.00000000
13
- 3 33.33333333
14
- 4 50.00000000
15
- 5 40.00000000
16
- 6 33.33333333
17
- 7 42.85714286
18
- 8 50.00000000
19
- 9 44.44444444
20
- 10 50.00000000
21
- 11 45.45454545
22
- 12 41.66666667
23
- 13 38.46153846
24
- 14 35.71428571
25
- 15 40.00000000
26
- 16 43.75000000
27
- 17 47.05882353
28
- 18 44.44444444
29
- 19 47.36842105
30
- 20 50.00000000
31
- 21 52.38095238
32
- 22 50.00000000
33
- 23 47.82608696
34
- 24 50.00000000
35
- 25 52.00000000
36
- 26 50.00000000
37
- 27 51.85185185
38
- 28 53.57142857
39
- 29 55.17241379
40
- 30 56.66666667
41
- 31 58.06451613
42
- 32 59.37500000
43
- 33 57.57575758
44
- 34 58.82352941
45
- 35 60.00000000
46
- 36 58.33333333
47
- 37 56.75675676
48
- 38 55.26315789
49
- 39 53.84615385
50
- 40 52.50000000
51
- 41 51.21951220
52
- 42 50.00000000
53
- 43 51.16279070
54
- 44 50.00000000
55
- 45 51.11111111
56
- 46 50.00000000
57
- 47 48.93617021
58
- 48 50.00000000
59
- 49 48.97959184
60
- 50 48.00000000
61
- 51 47.05882353
62
- 52 46.15384615
63
- 53 45.28301887
64
- 54 46.29629630
65
- 55 47.27272727
66
- 56 48.21428571
67
- 57 49.12280702
68
- 58 50.00000000
69
- 59 49.15254237
70
- 60 48.33333333
71
- 61 49.18032787
72
- 62 48.38709677
73
- 63 47.61904762
74
- 64 46.87500000
75
- 65 46.15384615
76
- 66 45.45454545
77
- 67 44.77611940
78
- 68 44.11764706
79
- 69 43.47826087
80
- 70 44.28571429
81
- 71 45.07042254
82
- 72 44.44444444
83
- 73 43.83561644
84
- 74 44.59459459
85
- 75 45.33333333
86
- 76 46.05263158
87
- 77 46.75324675
88
- 78 47.43589744
89
- 79 46.83544304
90
- 80 47.50000000
91
- 81 48.14814815
92
- 82 47.56097561
93
- 83 48.19277108
94
- 84 48.80952381
95
- 85 48.23529412
96
- 86 47.67441860
97
- 87 48.27586207
98
- 88 48.86363636
99
- 89 48.31460674
100
- 90 47.77777778
101
- 91 47.25274725
102
- 92 46.73913043
103
- 93 46.23655914
104
- 94 45.74468085
105
- 95 45.26315789
106
- 96 45.83333333
107
- 97 45.36082474
108
- 98 44.89795918
109
- 99 44.44444444
110
- 100 44.00000000
111
- 101 44.55445545
112
- 102 44.11764706
113
- 103 43.68932039
114
- 104 44.23076923
115
- 105 44.76190476
116
- 106 45.28301887
117
- 107 44.85981308
118
- 108 44.44444444
119
- 109 44.03669725
120
- 110 43.63636364
121
- 111 44.14414414
122
- 112 44.64285714
123
- 113 44.24778761
124
- 114 44.73684211
125
- 115 45.21739130
126
- 116 44.82758621
127
- 117 44.44444444
128
- 118 44.06779661
129
- 119 44.53781513
130
- 120 44.16666667
131
- 121 44.62809917
132
- 122 44.26229508
133
- 123 43.90243902
134
- 124 43.54838710
135
- 125 44.00000000
136
- 126 43.65079365
137
- 127 43.30708661
138
- 128 42.96875000
139
- 129 42.63565891
140
- 130 42.30769231
141
- 131 42.74809160
142
- 132 43.18181818
143
- 133 42.85714286
144
- 134 42.53731343
145
- 135 42.22222222
146
- 136 42.64705882
147
- 137 42.33576642
148
- 138 42.02898551
149
- 139 42.44604317
150
- 140 42.85714286
151
- 141 43.26241135
152
- 142 43.66197183
153
- 143 44.05594406
154
- 144 43.75000000
155
- 145 44.13793103
156
- 146 44.52054795
157
- 147 44.21768707
158
- 148 44.59459459
159
- 149 44.29530201
160
- 150 44.00000000
161
- 151 43.70860927
162
- 152 44.07894737
163
- 153 44.44444444
164
- 154 44.15584416
165
- 155 43.87096774
166
- 156 43.58974359
167
- 157 43.94904459
168
- 158 44.30379747
169
- 159 44.02515723
170
- 160 44.37500000
171
- 161 44.09937888
172
- 162 43.82716049
173
- 163 43.55828221
174
- 164 43.29268293
175
- 165 43.63636364
176
- 166 43.37349398
177
- 167 43.71257485
178
- 168 43.45238095
179
- 169 43.19526627
180
- 170 42.94117647
181
- 171 42.69005848
182
- 172 42.44186047
183
- 173 42.19653179
184
- 174 41.95402299
185
- 175 41.71428571
186
- 176 42.04545455
187
- 177 41.80790960
188
- 178 41.57303371
189
- 179 41.89944134
190
- 180 41.66666667
191
- 181 41.43646409
192
- 182 41.75824176
193
- 183 42.07650273
194
- 184 41.84782609
195
- 185 42.16216216
196
- 186 42.47311828
197
- 187 42.24598930
198
- 188 42.02127660
199
- 189 42.32804233
200
- 190 42.10526316
201
- 191 42.40837696
202
- 192 42.70833333
203
- 193 42.48704663
204
- 194 42.26804124
205
- 195 42.56410256
206
- 196 42.85714286
207
- 197 42.63959391
208
- 198 42.92929293
209
- 199 42.71356784
210
- 200 42.50000000
211
- 201 42.28855721
212
- 202 42.07920792
213
- 203 41.87192118
214
- 204 41.66666667
215
- 205 41.95121951
216
- 206 41.74757282
217
- 207 41.54589372
218
- 208 41.34615385
219
- 209 41.14832536
220
- 210 40.95238095
221
- 211 41.23222749
222
- 212 41.03773585
223
- 213 40.84507042
224
- 214 40.65420561
225
- 215 40.93023256
226
- 216 41.20370370
227
- 217 41.01382488
228
- 218 41.28440367
229
- 219 41.55251142
230
- 220 41.81818182
231
- 221 41.62895928
232
- 222 41.44144144
233
- 223 41.70403587
234
- 224 41.51785714
235
- 225 41.77777778
236
- 226 41.59292035
237
- 227 41.40969163
238
- 228 41.66666667
239
- 229 41.48471616
240
- 230 41.30434783
241
- 231 41.12554113
242
- 232 41.37931034
243
- 233 41.20171674
244
- 234 41.45299145
245
- 235 41.27659574
246
- 236 41.10169492
247
- 237 40.92827004
248
- 238 40.75630252
249
- 239 40.58577406
250
- 240 40.83333333
251
- 241 40.66390041
252
- 242 40.90909091
253
- 243 41.15226337
254
- 244 40.98360656
255
- 245 41.22448980
256
- 246 41.05691057
257
- 247 41.29554656
258
- 248 41.12903226
259
- 249 40.96385542
260
- 250 40.80000000
261
- 251 41.03585657
262
- 252 40.87301587
263
- 253 40.71146245
264
- 254 40.55118110
265
- 255 40.39215686
266
- 256 40.62500000
267
- 257 40.85603113
268
- 258 40.69767442
269
- 259 40.54054054
270
- 260 40.76923077
271
- 261 40.61302682
272
- 262 40.45801527
273
- 263 40.30418251
274
- 264 40.15151515
275
- 265 40.37735849
276
- 266 40.60150376
277
- 267 40.44943820
278
- 268 40.29850746
279
- 269 40.52044610
280
- 270 40.74074074
281
- 271 40.59040590
282
- 272 40.44117647
283
- 273 40.29304029
284
- 274 40.14598540
285
- 275 40.00000000
286
- 276 40.21739130
287
- 277 40.43321300
288
- 278 40.28776978
289
- 279 40.14336918
290
- 280 40.00000000
291
- 281 40.21352313
292
- 282 40.07092199
293
- 283 39.92932862
294
- 284 39.78873239
295
- 285 40.00000000
296
- 286 39.86013986
297
- 287 40.06968641
298
- 288 39.93055556
299
- 289 39.79238754
300
- 290 40.00000000
301
- 291 40.20618557
302
- 292 40.41095890
303
- 293 40.61433447
304
- 294 40.47619048
305
- 295 40.33898305
306
- 296 40.20270270
307
- 297 40.06734007
308
- 298 40.26845638
309
- 299 40.46822742
310
- 300 40.66666667
311
- 301 40.53156146
312
- 302 40.72847682
313
- 303 40.59405941
314
- 304 40.46052632
315
- 305 40.65573770
316
- 306 40.52287582
317
- 307 40.71661238
318
- 308 40.58441558
319
- 309 40.45307443
320
- 310 40.32258065
321
- 311 40.51446945
322
- 312 40.70512821
323
- 313 40.89456869
324
- 314 40.76433121
325
- 315 40.63492063
326
- 316 40.50632911
327
- 317 40.69400631
328
- 318 40.56603774
329
- 319 40.75235110
330
- 320 40.62500000
331
- 321 40.49844237
332
- 322 40.68322981
333
- 323 40.55727554
334
- 324 40.43209877
335
- 325 40.61538462
336
- 326 40.49079755
337
- 327 40.67278287
338
- 328 40.54878049
339
- 329 40.42553191
340
- 330 40.30303030
341
- 331 40.18126888
342
- 332 40.06024096
343
- 333 40.24024024
344
- 334 40.11976048
345
- 335 40.29850746
346
- 336 40.17857143
347
- 337 40.05934718
348
- 338 39.94082840
349
- 339 39.82300885
350
- 340 40.00000000
351
- 341 40.17595308
352
- 342 40.35087719
353
- 343 40.23323615
354
- 344 40.11627907
355
- 345 40.00000000
356
- 346 40.17341040
357
- 347 40.34582133
358
- 348 40.22988506
359
- 349 40.11461318
360
- 350 40.28571429
361
- 351 40.17094017
362
- 352 40.34090909
363
- 353 40.22662890
364
- 354 40.11299435
365
- 355 40.00000000
366
- 356 40.16853933
367
- 357 40.33613445
368
- 358 40.22346369
369
- 359 40.11142061
370
- 360 40.27777778
371
- 361 40.44321330
372
- 362 40.60773481
373
- 363 40.49586777
374
- 364 40.38461538
375
- 365 40.27397260
376
- 366 40.43715847
377
- 367 40.32697548
378
- 368 40.21739130
379
- 369 40.10840108
380
- 370 40.27027027
381
- 371 40.16172507
382
- 372 40.05376344
383
- 373 39.94638070
384
- 374 39.83957219
385
- 375 40.00000000
386
- 376 39.89361702
387
- 377 40.05305040
388
- 378 40.21164021
389
- 379 40.10554090
390
- 380 40.00000000
391
- 381 40.15748031
392
- 382 40.31413613
393
- 383 40.46997389
394
- 384 40.36458333
395
- 385 40.51948052
396
- 386 40.41450777
397
- 387 40.31007752
398
- 388 40.20618557
399
- 389 40.35989717
400
- 390 40.51282051
401
- 391 40.66496164
402
- 392 40.81632653
403
- 393 40.71246819
404
- 394 40.60913706
405
- 395 40.75949367
406
- 396 40.90909091
407
- 397 41.05793451
408
- 398 40.95477387
409
- 399 41.10275689
410
- 400 41.00000000
411
- 401 41.14713217
412
- 402 41.04477612
413
- 403 40.94292804
414
- 404 40.84158416
415
- 405 40.98765432
416
- 406 40.88669951
417
- 407 40.78624079
418
- 408 40.68627451
419
- 409 40.83129584
420
- 410 40.73170732
421
- 411 40.63260341
422
- 412 40.53398058
423
- 413 40.43583535
424
- 414 40.57971014
425
- 415 40.48192771
426
- 416 40.62500000
427
- 417 40.76738609
428
- 418 40.66985646
429
- 419 40.81145585
430
- 420 40.71428571
431
- 421 40.85510689
432
- 422 40.75829384
433
- 423 40.66193853
434
- 424 40.80188679
435
- 425 40.94117647
436
- 426 41.07981221
437
- 427 40.98360656
438
- 428 40.88785047
439
- 429 41.02564103
440
- 430 40.93023256
441
- 431 41.06728538
442
- 432 41.20370370
443
- 433 41.10854503
444
- 434 41.24423963
445
- 435 41.14942529
446
- 436 41.05504587
447
- 437 40.96109840
448
- 438 41.09589041
449
- 439 41.23006834
450
- 440 41.13636364
451
- 441 41.04308390
452
- 442 40.95022624
453
- 443 40.85778781
454
- 444 40.76576577
455
- 445 40.67415730
456
- 446 40.80717489
457
- 447 40.71588367
458
- 448 40.62500000
459
- 449 40.53452116
460
- 450 40.66666667
461
- 451 40.79822616
462
- 452 40.70796460
463
- 453 40.83885210
464
- 454 40.74889868
465
- 455 40.65934066
466
- 456 40.78947368
467
- 457 40.91903720
468
- 458 40.82969432
469
- 459 40.74074074
470
- 460 40.86956522
471
- 461 40.78091106
472
- 462 40.69264069
473
- 463 40.82073434
474
- 464 40.94827586
475
- 465 40.86021505
476
- 466 40.77253219
477
- 467 40.89935760
478
- 468 40.81196581
479
- 469 40.93816631
480
- 470 40.85106383
481
- 471 40.76433121
482
- 472 40.67796610
483
- 473 40.80338266
484
- 474 40.71729958
485
- 475 40.63157895
486
- 476 40.75630252
487
- 477 40.67085954
488
- 478 40.79497908
489
- 479 40.91858038
490
- 480 41.04166667
491
- 481 40.95634096
492
- 482 40.87136929
493
- 483 40.78674948
494
- 484 40.70247934
495
- 485 40.61855670
496
- 486 40.53497942
497
- 487 40.65708419
498
- 488 40.57377049
499
- 489 40.49079755
500
- 490 40.61224490
501
- 491 40.52953157
502
- 492 40.44715447
503
- 493 40.36511156
504
- 494 40.48582996
505
- 495 40.60606061
506
- 496 40.72580645
507
- 497 40.64386318
508
- 498 40.56224900
509
- 499 40.68136273
510
- 500 40.80000000
511
- 501 40.71856287
512
- 502 40.63745020
513
- 503 40.55666004
514
- 504 40.47619048
515
- 505 40.39603960
516
- 506 40.51383399
517
- 507 40.43392505
518
- 508 40.55118110
519
- 509 40.47151277
520
- 510 40.58823529
521
- 511 40.50880626
522
- 512 40.42968750
523
- 513 40.35087719
524
- 514 40.27237354
525
- 515 40.38834951
526
- 516 40.50387597
527
- 517 40.61895551
528
- 518 40.73359073
529
- 519 40.65510597
530
- 520 40.76923077
531
- 521 40.69097889
532
- 522 40.80459770
533
- 523 40.72657744
534
- 524 40.64885496
535
- 525 40.57142857
536
- 526 40.49429658
537
- 527 40.41745731
538
- 528 40.53030303
539
- 529 40.64272212
540
- 530 40.75471698
541
- 531 40.67796610
542
- 532 40.60150376
543
- 533 40.52532833
544
- 534 40.63670412
545
- 535 40.74766355
546
- 536 40.67164179
547
- 537 40.78212291
548
- 538 40.89219331
549
- 539 40.81632653
550
- 540 40.74074074
551
- 541 40.66543438
552
- 542 40.77490775
553
- 543 40.69981584
554
- 544 40.80882353
555
- 545 40.73394495
556
- 546 40.65934066
557
- 547 40.58500914
558
- 548 40.69343066
559
- 549 40.61930783
560
- 550 40.54545455
561
- 551 40.65335753
562
- 552 40.57971014
563
- 553 40.50632911
564
- 554 40.43321300
565
- 555 40.36036036
566
- 556 40.46762590
567
- 557 40.57450628
568
- 558 40.68100358
569
- 559 40.60822898
570
- 560 40.53571429
571
- 561 40.46345811
572
- 562 40.56939502
573
- 563 40.67495560
574
- 564 40.60283688
575
- 565 40.70796460
576
- 566 40.63604240
577
- 567 40.56437390
578
- 568 40.66901408
579
- 569 40.59753954
580
- 570 40.52631579
581
- 571 40.45534151
582
- 572 40.55944056
583
- 573 40.48865620
584
- 574 40.41811847
585
- 575 40.34782609
586
- 576 40.45138889
587
- 577 40.55459272
588
- 578 40.48442907
589
- 579 40.41450777
590
- 580 40.51724138
591
- 581 40.44750430
592
- 582 40.37800687
593
- 583 40.48027444
594
- 584 40.58219178
595
- 585 40.68376068
596
- 586 40.78498294
597
- 587 40.71550256
598
- 588 40.64625850
599
- 589 40.57724958
600
- 590 40.50847458
601
- 591 40.43993232
602
- 592 40.37162162
603
- 593 40.47217538
604
- 594 40.40404040
605
- 595 40.33613445
606
- 596 40.43624161
607
- 597 40.36850921
608
- 598 40.30100334
609
- 599 40.40066778
610
- 600 40.33333333
611
- 601 40.26622296
612
- 602 40.19933555
613
- 603 40.29850746
614
- 604 40.23178808
615
- 605 40.16528926
616
- 606 40.09900990
617
- 607 40.19769357
618
- 608 40.13157895
619
- 609 40.06568144
620
- 610 40.00000000
621
- 611 40.09819967
622
- 612 40.03267974
623
- 613 40.13050571
624
- 614 40.06514658
625
- 615 40.00000000
626
- 616 40.09740260
627
- 617 40.19448947
628
- 618 40.12944984
629
- 619 40.06462036
630
- 620 40.00000000
631
- 621 39.93558776
632
- 622 39.87138264
633
- 623 39.80738363
634
- 624 39.74358974
635
- 625 39.68000000
636
- 626 39.77635783
637
- 627 39.71291866
638
- 628 39.64968153
639
- 629 39.74562798
640
- 630 39.84126984
641
- 631 39.93660856
642
- 632 40.03164557
643
- 633 39.96840442
644
- 634 39.90536278
645
- 635 39.84251969
646
- 636 39.77987421
647
- 637 39.71742543
648
- 638 39.65517241
649
- 639 39.74960876
650
- 640 39.84375000
651
- 641 39.78159126
652
- 642 39.71962617
653
- 643 39.65785381
654
- 644 39.75155280
655
- 645 39.68992248
656
- 646 39.62848297
657
- 647 39.56723338
658
- 648 39.50617284
659
- 649 39.44530046
660
- 650 39.38461538
661
- 651 39.32411674
662
- 652 39.26380368
663
- 653 39.20367534
664
- 654 39.29663609
665
- 655 39.23664122
666
- 656 39.17682927
667
- 657 39.11719939
668
- 658 39.20972644
669
- 659 39.15022762
670
- 660 39.09090909
671
- 661 39.03177005
672
- 662 38.97280967
673
- 663 38.91402715
674
- 664 38.85542169
675
- 665 38.79699248
676
- 666 38.73873874
677
- 667 38.83058471
678
- 668 38.92215569
679
- 669 39.01345291
680
- 670 39.10447761
681
- 671 39.04619970
682
- 672 38.98809524
683
- 673 38.93016345
684
- 674 38.87240356
685
- 675 38.81481481
686
- 676 38.75739645
687
- 677 38.70014771
688
- 678 38.79056047
689
- 679 38.73343152
690
- 680 38.82352941
691
- 681 38.91336270
692
- 682 38.85630499
693
- 683 38.79941435
694
- 684 38.88888889
695
- 685 38.83211679
696
- 686 38.92128280
697
- 687 38.86462882
698
- 688 38.80813953
699
- 689 38.75181422
700
- 690 38.69565217
701
- 691 38.78437048
702
- 692 38.72832370
703
- 693 38.67243867
704
- 694 38.76080692
705
- 695 38.70503597
706
- 696 38.79310345
707
- 697 38.73744620
708
- 698 38.82521490
709
- 699 38.91273247
710
- 700 38.85714286
711
- 701 38.94436519
712
- 702 39.03133903
713
- 703 39.11806543
714
- 704 39.20454545
715
- 705 39.14893617
716
- 706 39.09348442
717
- 707 39.03818953
718
- 708 38.98305085
719
- 709 38.92806770
720
- 710 38.87323944
721
- 711 38.95921238
722
- 712 39.04494382
723
- 713 39.13043478
724
- 714 39.07563025
725
- 715 39.02097902
726
- 716 38.96648045
727
- 717 38.91213389
728
- 718 38.85793872
729
- 719 38.94297636
730
- 720 38.88888889
731
- 721 38.83495146
732
- 722 38.78116343
733
- 723 38.86583679
734
- 724 38.81215470
735
- 725 38.89655172
736
- 726 38.98071625
737
- 727 38.92709766
738
- 728 38.87362637
739
- 729 38.82030178
740
- 730 38.76712329
741
- 731 38.71409029
742
- 732 38.66120219
743
- 733 38.74488404
744
- 734 38.82833787
745
- 735 38.77551020
746
- 736 38.72282609
747
- 737 38.80597015
748
- 738 38.88888889
749
- 739 38.97158322
750
- 740 38.91891892
751
- 741 38.86639676
752
- 742 38.94878706
753
- 743 39.03095559
754
- 744 39.11290323
755
- 745 39.06040268
756
- 746 39.00804290
757
- 747 38.95582329
758
- 748 38.90374332
759
- 749 38.85180240
760
- 750 38.93333333
761
 
762
- Final result: 38.9333 +/- 1.7816
763
- Random chance: 25.0000 +/- 1.5822
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-IQ3_M.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 36.2667 +/- 1.7567
6
+ Random chance: 25.0000 +/- 1.5822
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 280.68 ms
10
+ llama_perf_context_print: prompt eval time = 260791.25 ms / 70687 tokens ( 3.69 ms per token, 271.05 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 263549.69 ms / 70688 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-iq3_m.ppx CHANGED
@@ -1,37 +1,37 @@
1
  ====== Perplexity statistics ======
2
- Mean PPL(Q) : 9.087639 ± 0.058224
3
- Mean PPL(base) : 7.534124 ± 0.048206
4
- Cor(ln(PPL(Q)), ln(PPL(base))): 96.96%
5
- Mean ln(PPL(Q)/PPL(base)) : 0.187473 ± 0.001579
6
- Mean PPL(Q)/PPL(base) : 1.206197 ± 0.001905
7
- Mean PPL(Q)-PPL(base) : 1.553515 ± 0.016466
8
 
9
  ====== KL divergence statistics ======
10
- Mean KLD: 0.173555 ± 0.000494
11
- Maximum KLD: 5.439730
12
- 99.9% KLD: 2.047228
13
- 99.0% KLD: 0.821818
14
- 99.0% KLD: 0.821818
15
- Median KLD: 0.143569
16
- 10.0% KLD: 0.011327
17
- 5.0% KLD: 0.003726
18
- 1.0% KLD: 0.000528
19
  Minimum KLD: 0.000000
20
 
21
  ====== Token probability statistics ======
22
- Mean Δp: -4.759 ± 0.031 %
23
- Maximum Δp: 81.197%
24
- 99.9% Δp: 38.726%
25
- 99.0% Δp: 21.998%
26
- 95.0% Δp: 9.357%
27
- 90.0% Δp: 4.162%
28
- 75.0% Δp: 0.081%
29
- Median Δp: -1.171%
30
- 25.0% Δp: -8.442%
31
- 10.0% Δp: -19.664%
32
- 5.0% Δp: -27.455%
33
- 1.0% Δp: -44.384%
34
- 0.1% Δp: -74.316%
35
- Minimum Δp: -97.360%
36
- RMS Δp : 12.642 ± 0.047 %
37
- Same top p: 77.787 ± 0.110 %
 
1
  ====== Perplexity statistics ======
2
+ Mean PPL(Q) : 8.963688 ± 0.058386
3
+ Mean PPL(base) : 7.237090 ± 0.045539
4
+ Cor(ln(PPL(Q)), ln(PPL(base))): 95.93%
5
+ Mean ln(PPL(Q)/PPL(base)) : 0.213963 ± 0.001840
6
+ Mean PPL(Q)/PPL(base) : 1.238576 ± 0.002279
7
+ Mean PPL(Q)-PPL(base) : 1.726598 ± 0.019534
8
 
9
  ====== KL divergence statistics ======
10
+ Mean KLD: 0.209768 ± 0.000734
11
+ Maximum KLD: 11.045219
12
+ 99.9% KLD: 3.037534
13
+ 99.0% KLD: 1.327696
14
+ 99.0% KLD: 1.327696
15
+ Median KLD: 0.145406
16
+ 10.0% KLD: 0.013515
17
+ 5.0% KLD: 0.004453
18
+ 1.0% KLD: 0.000589
19
  Minimum KLD: 0.000000
20
 
21
  ====== Token probability statistics ======
22
+ Mean Δp: -4.187 ± 0.035 %
23
+ Maximum Δp: 87.898%
24
+ 99.9% Δp: 53.785%
25
+ 99.0% Δp: 29.326%
26
+ 95.0% Δp: 12.210%
27
+ 90.0% Δp: 5.784%
28
+ 75.0% Δp: 0.302%
29
+ Median Δp: -0.896%
30
+ 25.0% Δp: -7.479%
31
+ 10.0% Δp: -19.150%
32
+ 5.0% Δp: -28.492%
33
+ 1.0% Δp: -52.697%
34
+ 0.1% Δp: -83.945%
35
+ Minimum Δp: -97.765%
36
+ RMS Δp : 13.969 ± 0.056 %
37
+ Same top p: 77.664 ± 0.110 %
scores/Watt-Tool-8B-iq3_m.tqa CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 817 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 817 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 50.00000000
13
- 3 33.33333333
14
- 4 25.00000000
15
- 5 20.00000000
16
- 6 16.66666667
17
- 7 14.28571429
18
- 8 12.50000000
19
- 9 11.11111111
20
- 10 10.00000000
21
- 11 9.09090909
22
- 12 16.66666667
23
- 13 23.07692308
24
- 14 21.42857143
25
- 15 26.66666667
26
- 16 31.25000000
27
- 17 29.41176471
28
- 18 27.77777778
29
- 19 26.31578947
30
- 20 25.00000000
31
- 21 23.80952381
32
- 22 22.72727273
33
- 23 21.73913043
34
- 24 25.00000000
35
- 25 24.00000000
36
- 26 23.07692308
37
- 27 22.22222222
38
- 28 21.42857143
39
- 29 24.13793103
40
- 30 23.33333333
41
- 31 22.58064516
42
- 32 21.87500000
43
- 33 24.24242424
44
- 34 23.52941176
45
- 35 22.85714286
46
- 36 22.22222222
47
- 37 24.32432432
48
- 38 23.68421053
49
- 39 25.64102564
50
- 40 25.00000000
51
- 41 24.39024390
52
- 42 23.80952381
53
- 43 25.58139535
54
- 44 25.00000000
55
- 45 26.66666667
56
- 46 26.08695652
57
- 47 27.65957447
58
- 48 29.16666667
59
- 49 28.57142857
60
- 50 30.00000000
61
- 51 29.41176471
62
- 52 28.84615385
63
- 53 28.30188679
64
- 54 27.77777778
65
- 55 29.09090909
66
- 56 28.57142857
67
- 57 28.07017544
68
- 58 27.58620690
69
- 59 28.81355932
70
- 60 30.00000000
71
- 61 29.50819672
72
- 62 30.64516129
73
- 63 30.15873016
74
- 64 31.25000000
75
- 65 30.76923077
76
- 66 30.30303030
77
- 67 29.85074627
78
- 68 29.41176471
79
- 69 30.43478261
80
- 70 30.00000000
81
- 71 29.57746479
82
- 72 30.55555556
83
- 73 30.13698630
84
- 74 29.72972973
85
- 75 29.33333333
86
- 76 28.94736842
87
- 77 28.57142857
88
- 78 28.20512821
89
- 79 27.84810127
90
- 80 28.75000000
91
- 81 28.39506173
92
- 82 29.26829268
93
- 83 28.91566265
94
- 84 28.57142857
95
- 85 28.23529412
96
- 86 27.90697674
97
- 87 27.58620690
98
- 88 27.27272727
99
- 89 28.08988764
100
- 90 27.77777778
101
- 91 27.47252747
102
- 92 27.17391304
103
- 93 26.88172043
104
- 94 26.59574468
105
- 95 26.31578947
106
- 96 27.08333333
107
- 97 27.83505155
108
- 98 27.55102041
109
- 99 28.28282828
110
- 100 29.00000000
111
- 101 29.70297030
112
- 102 29.41176471
113
- 103 29.12621359
114
- 104 28.84615385
115
- 105 28.57142857
116
- 106 28.30188679
117
- 107 28.97196262
118
- 108 29.62962963
119
- 109 29.35779817
120
- 110 30.00000000
121
- 111 29.72972973
122
- 112 30.35714286
123
- 113 30.97345133
124
- 114 31.57894737
125
- 115 31.30434783
126
- 116 31.03448276
127
- 117 30.76923077
128
- 118 30.50847458
129
- 119 30.25210084
130
- 120 30.83333333
131
- 121 31.40495868
132
- 122 31.14754098
133
- 123 30.89430894
134
- 124 31.45161290
135
- 125 31.20000000
136
- 126 30.95238095
137
- 127 30.70866142
138
- 128 30.46875000
139
- 129 31.00775194
140
- 130 31.53846154
141
- 131 31.29770992
142
- 132 31.06060606
143
- 133 30.82706767
144
- 134 30.59701493
145
- 135 31.11111111
146
- 136 30.88235294
147
- 137 31.38686131
148
- 138 31.88405797
149
- 139 32.37410072
150
- 140 32.14285714
151
- 141 31.91489362
152
- 142 31.69014085
153
- 143 31.46853147
154
- 144 31.94444444
155
- 145 31.72413793
156
- 146 31.50684932
157
- 147 31.97278912
158
- 148 31.75675676
159
- 149 31.54362416
160
- 150 31.33333333
161
- 151 31.78807947
162
- 152 31.57894737
163
- 153 31.37254902
164
- 154 31.16883117
165
- 155 31.61290323
166
- 156 32.05128205
167
- 157 31.84713376
168
- 158 32.27848101
169
- 159 32.07547170
170
- 160 31.87500000
171
- 161 31.67701863
172
- 162 31.48148148
173
- 163 31.28834356
174
- 164 31.09756098
175
- 165 31.51515152
176
- 166 31.92771084
177
- 167 31.73652695
178
- 168 32.14285714
179
- 169 31.95266272
180
- 170 31.76470588
181
- 171 31.57894737
182
- 172 31.39534884
183
- 173 31.21387283
184
- 174 31.60919540
185
- 175 31.42857143
186
- 176 31.81818182
187
- 177 31.63841808
188
- 178 31.46067416
189
- 179 31.28491620
190
- 180 31.66666667
191
- 181 31.49171271
192
- 182 31.31868132
193
- 183 31.14754098
194
- 184 30.97826087
195
- 185 30.81081081
196
- 186 31.18279570
197
- 187 31.01604278
198
- 188 30.85106383
199
- 189 30.68783069
200
- 190 30.52631579
201
- 191 30.36649215
202
- 192 30.20833333
203
- 193 30.05181347
204
- 194 30.41237113
205
- 195 30.25641026
206
- 196 30.10204082
207
- 197 30.45685279
208
- 198 30.80808081
209
- 199 30.65326633
210
- 200 31.00000000
211
- 201 30.84577114
212
- 202 30.69306931
213
- 203 30.54187192
214
- 204 30.39215686
215
- 205 30.73170732
216
- 206 30.58252427
217
- 207 30.91787440
218
- 208 30.76923077
219
- 209 30.62200957
220
- 210 30.47619048
221
- 211 30.33175355
222
- 212 30.18867925
223
- 213 30.04694836
224
- 214 29.90654206
225
- 215 29.76744186
226
- 216 29.62962963
227
- 217 29.95391705
228
- 218 30.27522936
229
- 219 30.13698630
230
- 220 30.00000000
231
- 221 29.86425339
232
- 222 29.72972973
233
- 223 30.04484305
234
- 224 30.35714286
235
- 225 30.22222222
236
- 226 30.08849558
237
- 227 30.39647577
238
- 228 30.26315789
239
- 229 30.13100437
240
- 230 30.00000000
241
- 231 29.87012987
242
- 232 29.74137931
243
- 233 30.04291845
244
- 234 30.34188034
245
- 235 30.21276596
246
- 236 30.50847458
247
- 237 30.80168776
248
- 238 31.09243697
249
- 239 30.96234310
250
- 240 30.83333333
251
- 241 30.70539419
252
- 242 30.57851240
253
- 243 30.45267490
254
- 244 30.32786885
255
- 245 30.20408163
256
- 246 30.08130081
257
- 247 30.36437247
258
- 248 30.24193548
259
- 249 30.52208835
260
- 250 30.40000000
261
- 251 30.67729084
262
- 252 30.55555556
263
- 253 30.43478261
264
- 254 30.31496063
265
- 255 30.19607843
266
- 256 30.07812500
267
- 257 29.96108949
268
- 258 29.84496124
269
- 259 29.72972973
270
- 260 29.61538462
271
- 261 29.88505747
272
- 262 29.77099237
273
- 263 29.65779468
274
- 264 29.92424242
275
- 265 30.18867925
276
- 266 30.07518797
277
- 267 29.96254682
278
- 268 29.85074627
279
- 269 29.73977695
280
- 270 29.62962963
281
- 271 29.52029520
282
- 272 29.41176471
283
- 273 29.67032967
284
- 274 29.56204380
285
- 275 29.45454545
286
- 276 29.34782609
287
- 277 29.24187726
288
- 278 29.13669065
289
- 279 29.03225806
290
- 280 29.28571429
291
- 281 29.53736655
292
- 282 29.43262411
293
- 283 29.68197880
294
- 284 29.57746479
295
- 285 29.47368421
296
- 286 29.72027972
297
- 287 29.96515679
298
- 288 29.86111111
299
- 289 29.75778547
300
- 290 29.65517241
301
- 291 29.89690722
302
- 292 29.79452055
303
- 293 30.03412969
304
- 294 29.93197279
305
- 295 30.16949153
306
- 296 30.40540541
307
- 297 30.63973064
308
- 298 30.53691275
309
- 299 30.43478261
310
- 300 30.33333333
311
- 301 30.23255814
312
- 302 30.13245033
313
- 303 30.03300330
314
- 304 29.93421053
315
- 305 29.83606557
316
- 306 29.73856209
317
- 307 29.64169381
318
- 308 29.87012987
319
- 309 29.77346278
320
- 310 29.67741935
321
- 311 29.58199357
322
- 312 29.80769231
323
- 313 29.71246006
324
- 314 29.61783439
325
- 315 29.52380952
326
- 316 29.43037975
327
- 317 29.33753943
328
- 318 29.24528302
329
- 319 29.15360502
330
- 320 29.06250000
331
- 321 28.97196262
332
- 322 29.19254658
333
- 323 29.10216718
334
- 324 29.01234568
335
- 325 28.92307692
336
- 326 28.83435583
337
- 327 29.05198777
338
- 328 29.26829268
339
- 329 29.17933131
340
- 330 29.39393939
341
- 331 29.30513595
342
- 332 29.51807229
343
- 333 29.72972973
344
- 334 29.64071856
345
- 335 29.85074627
346
- 336 30.05952381
347
- 337 29.97032641
348
- 338 29.88165680
349
- 339 30.08849558
350
- 340 30.00000000
351
- 341 30.20527859
352
- 342 30.40935673
353
- 343 30.32069971
354
- 344 30.52325581
355
- 345 30.72463768
356
- 346 30.63583815
357
- 347 30.83573487
358
- 348 30.74712644
359
- 349 30.65902579
360
- 350 30.57142857
361
- 351 30.48433048
362
- 352 30.68181818
363
- 353 30.59490085
364
- 354 30.50847458
365
- 355 30.70422535
366
- 356 30.61797753
367
- 357 30.53221289
368
- 358 30.44692737
369
- 359 30.36211699
370
- 360 30.27777778
371
- 361 30.19390582
372
- 362 30.11049724
373
- 363 30.30303030
374
- 364 30.21978022
375
- 365 30.13698630
376
- 366 30.05464481
377
- 367 30.24523161
378
- 368 30.16304348
379
- 369 30.08130081
380
- 370 30.00000000
381
- 371 29.91913747
382
- 372 30.10752688
383
- 373 30.02680965
384
- 374 30.21390374
385
- 375 30.13333333
386
- 376 30.05319149
387
- 377 30.23872679
388
- 378 30.15873016
389
- 379 30.07915567
390
- 380 30.26315789
391
- 381 30.18372703
392
- 382 30.10471204
393
- 383 30.02610966
394
- 384 29.94791667
395
- 385 29.87012987
396
- 386 29.79274611
397
- 387 29.71576227
398
- 388 29.89690722
399
- 389 29.82005141
400
- 390 29.74358974
401
- 391 29.92327366
402
- 392 29.84693878
403
- 393 30.02544529
404
- 394 30.20304569
405
- 395 30.12658228
406
- 396 30.30303030
407
- 397 30.47858942
408
- 398 30.40201005
409
- 399 30.32581454
410
- 400 30.25000000
411
- 401 30.42394015
412
- 402 30.59701493
413
- 403 30.76923077
414
- 404 30.69306931
415
- 405 30.61728395
416
- 406 30.54187192
417
- 407 30.46683047
418
- 408 30.39215686
419
- 409 30.31784841
420
- 410 30.24390244
421
- 411 30.17031630
422
- 412 30.33980583
423
- 413 30.26634383
424
- 414 30.43478261
425
- 415 30.60240964
426
- 416 30.52884615
427
- 417 30.69544365
428
- 418 30.62200957
429
- 419 30.54892601
430
- 420 30.71428571
431
- 421 30.87885986
432
- 422 30.80568720
433
- 423 30.96926714
434
- 424 30.89622642
435
- 425 30.82352941
436
- 426 30.75117371
437
- 427 30.67915691
438
- 428 30.60747664
439
- 429 30.53613054
440
- 430 30.46511628
441
- 431 30.39443155
442
- 432 30.55555556
443
- 433 30.71593533
444
- 434 30.87557604
445
- 435 30.80459770
446
- 436 30.73394495
447
- 437 30.66361556
448
- 438 30.82191781
449
- 439 30.97949886
450
- 440 30.90909091
451
- 441 30.83900227
452
- 442 30.76923077
453
- 443 30.69977427
454
- 444 30.85585586
455
- 445 30.78651685
456
- 446 30.71748879
457
- 447 30.64876957
458
- 448 30.58035714
459
- 449 30.73496659
460
- 450 30.66666667
461
- 451 30.59866962
462
- 452 30.53097345
463
- 453 30.68432671
464
- 454 30.61674009
465
- 455 30.54945055
466
- 456 30.48245614
467
- 457 30.41575492
468
- 458 30.34934498
469
- 459 30.28322440
470
- 460 30.21739130
471
- 461 30.15184382
472
- 462 30.30303030
473
- 463 30.45356371
474
- 464 30.60344828
475
- 465 30.75268817
476
- 466 30.68669528
477
- 467 30.62098501
478
- 468 30.76923077
479
- 469 30.70362473
480
- 470 30.63829787
481
- 471 30.78556263
482
- 472 30.93220339
483
- 473 30.86680761
484
- 474 31.01265823
485
- 475 31.15789474
486
- 476 31.09243697
487
- 477 31.02725367
488
- 478 30.96234310
489
- 479 30.89770355
490
- 480 30.83333333
491
- 481 30.76923077
492
- 482 30.91286307
493
- 483 30.84886128
494
- 484 30.78512397
495
- 485 30.92783505
496
- 486 30.86419753
497
- 487 31.00616016
498
- 488 30.94262295
499
- 489 31.08384458
500
- 490 31.22448980
501
- 491 31.36456212
502
- 492 31.50406504
503
- 493 31.44016227
504
- 494 31.57894737
505
- 495 31.51515152
506
- 496 31.45161290
507
- 497 31.38832998
508
- 498 31.32530120
509
- 499 31.26252505
510
- 500 31.20000000
511
- 501 31.33732535
512
- 502 31.27490040
513
- 503 31.21272366
514
- 504 31.34920635
515
- 505 31.28712871
516
- 506 31.22529644
517
- 507 31.36094675
518
- 508 31.29921260
519
- 509 31.43418468
520
- 510 31.56862745
521
- 511 31.50684932
522
- 512 31.44531250
523
- 513 31.38401559
524
- 514 31.32295720
525
- 515 31.45631068
526
- 516 31.39534884
527
- 517 31.52804642
528
- 518 31.46718147
529
- 519 31.40655106
530
- 520 31.34615385
531
- 521 31.28598848
532
- 522 31.41762452
533
- 523 31.35755258
534
- 524 31.29770992
535
- 525 31.23809524
536
- 526 31.17870722
537
- 527 31.30929791
538
- 528 31.25000000
539
- 529 31.37996219
540
- 530 31.32075472
541
- 531 31.26177024
542
- 532 31.20300752
543
- 533 31.33208255
544
- 534 31.27340824
545
- 535 31.21495327
546
- 536 31.15671642
547
- 537 31.28491620
548
- 538 31.22676580
549
- 539 31.16883117
550
- 540 31.29629630
551
- 541 31.23844732
552
- 542 31.18081181
553
- 543 31.12338858
554
- 544 31.25000000
555
- 545 31.19266055
556
- 546 31.13553114
557
- 547 31.26142596
558
- 548 31.38686131
559
- 549 31.32969035
560
- 550 31.27272727
561
- 551 31.21597096
562
- 552 31.34057971
563
- 553 31.46473779
564
- 554 31.40794224
565
- 555 31.35135135
566
- 556 31.29496403
567
- 557 31.23877917
568
- 558 31.18279570
569
- 559 31.12701252
570
- 560 31.25000000
571
- 561 31.19429590
572
- 562 31.13879004
573
- 563 31.08348135
574
- 564 31.20567376
575
- 565 31.32743363
576
- 566 31.44876325
577
- 567 31.39329806
578
- 568 31.51408451
579
- 569 31.63444640
580
- 570 31.57894737
581
- 571 31.69877408
582
- 572 31.64335664
583
- 573 31.58813264
584
- 574 31.70731707
585
- 575 31.82608696
586
- 576 31.94444444
587
- 577 32.06239168
588
- 578 32.17993080
589
- 579 32.29706390
590
- 580 32.41379310
591
- 581 32.35800344
592
- 582 32.47422680
593
- 583 32.41852487
594
- 584 32.53424658
595
- 585 32.47863248
596
- 586 32.59385666
597
- 587 32.53833049
598
- 588 32.48299320
599
- 589 32.42784380
600
- 590 32.37288136
601
- 591 32.31810491
602
- 592 32.26351351
603
- 593 32.20910624
604
- 594 32.32323232
605
- 595 32.26890756
606
- 596 32.21476510
607
- 597 32.16080402
608
- 598 32.10702341
609
- 599 32.05342237
610
- 600 32.00000000
611
- 601 32.11314476
612
- 602 32.22591362
613
- 603 32.33830846
614
- 604 32.28476821
615
- 605 32.23140496
616
- 606 32.34323432
617
- 607 32.28995058
618
- 608 32.23684211
619
- 609 32.18390805
620
- 610 32.13114754
621
- 611 32.07855974
622
- 612 32.18954248
623
- 613 32.13703100
624
- 614 32.08469055
625
- 615 32.03252033
626
- 616 32.14285714
627
- 617 32.25283630
628
- 618 32.20064725
629
- 619 32.31017771
630
- 620 32.25806452
631
- 621 32.20611916
632
- 622 32.31511254
633
- 623 32.26324238
634
- 624 32.21153846
635
- 625 32.16000000
636
- 626 32.10862620
637
- 627 32.05741627
638
- 628 32.16560510
639
- 629 32.11446741
640
- 630 32.06349206
641
- 631 32.17115689
642
- 632 32.12025316
643
- 633 32.22748815
644
- 634 32.33438486
645
- 635 32.28346457
646
- 636 32.23270440
647
- 637 32.18210361
648
- 638 32.13166144
649
- 639 32.08137715
650
- 640 32.18750000
651
- 641 32.13728549
652
- 642 32.24299065
653
- 643 32.19284603
654
- 644 32.14285714
655
- 645 32.24806202
656
- 646 32.19814241
657
- 647 32.14837713
658
- 648 32.25308642
659
- 649 32.20338983
660
- 650 32.30769231
661
- 651 32.41167435
662
- 652 32.51533742
663
- 653 32.61868300
664
- 654 32.56880734
665
- 655 32.51908397
666
- 656 32.62195122
667
- 657 32.72450533
668
- 658 32.67477204
669
- 659 32.77693475
670
- 660 32.72727273
671
- 661 32.82904690
672
- 662 32.77945619
673
- 663 32.73001508
674
- 664 32.68072289
675
- 665 32.63157895
676
- 666 32.73273273
677
- 667 32.83358321
678
- 668 32.78443114
679
- 669 32.73542601
680
- 670 32.83582090
681
- 671 32.78688525
682
- 672 32.73809524
683
- 673 32.68945022
684
- 674 32.64094955
685
- 675 32.74074074
686
- 676 32.84023669
687
- 677 32.79172821
688
- 678 32.74336283
689
- 679 32.69513991
690
- 680 32.64705882
691
- 681 32.59911894
692
- 682 32.69794721
693
- 683 32.65007321
694
- 684 32.74853801
695
- 685 32.70072993
696
- 686 32.65306122
697
- 687 32.60553130
698
- 688 32.55813953
699
- 689 32.65602322
700
- 690 32.60869565
701
- 691 32.70622287
702
- 692 32.80346821
703
- 693 32.90043290
704
- 694 32.85302594
705
- 695 32.80575540
706
- 696 32.90229885
707
- 697 32.85509326
708
- 698 32.80802292
709
- 699 32.76108727
710
- 700 32.85714286
711
- 701 32.81027104
712
- 702 32.90598291
713
- 703 32.85917496
714
- 704 32.81250000
715
- 705 32.76595745
716
- 706 32.86118980
717
- 707 32.95615276
718
- 708 32.90960452
719
- 709 32.86318759
720
- 710 32.95774648
721
- 711 32.91139241
722
- 712 33.00561798
723
- 713 32.95932679
724
- 714 32.91316527
725
- 715 33.00699301
726
- 716 32.96089385
727
- 717 33.05439331
728
- 718 33.14763231
729
- 719 33.24061196
730
- 720 33.19444444
731
- 721 33.14840499
732
- 722 33.10249307
733
- 723 33.05670816
734
- 724 33.14917127
735
- 725 33.10344828
736
- 726 33.19559229
737
- 727 33.14993122
738
- 728 33.10439560
739
- 729 33.19615912
740
- 730 33.15068493
741
- 731 33.10533516
742
- 732 33.19672131
743
- 733 33.15143247
744
- 734 33.10626703
745
- 735 33.06122449
746
- 736 33.01630435
747
- 737 33.10719132
748
- 738 33.06233062
749
- 739 33.01759134
750
- 740 32.97297297
751
- 741 33.06342780
752
- 742 33.01886792
753
- 743 32.97442799
754
- 744 33.06451613
755
- 745 33.02013423
756
- 746 33.10991957
757
- 747 33.19946452
758
- 748 33.15508021
759
- 749 33.24432577
760
- 750 33.33333333
761
 
762
- Final result: 33.3333 +/- 1.7225
763
- Random chance: 19.8992 +/- 1.4588
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-IQ3_M.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 33.2308 +/- 2.6169
6
+ Random chance: 28.5589 +/- 2.5094
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 280.73 ms
10
+ llama_perf_context_print: prompt eval time = 76784.24 ms / 17625 tokens ( 4.36 ms per token, 229.54 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 78027.49 ms / 17626 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-iq3_m.wng CHANGED
@@ -1,760 +1,11 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- winogrande_score : loaded 1267 tasks from prompt.
6
- winogrande_score : selecting 750 random tasks
7
- winogrande_score : tokenizing selected tasks
8
- winogrande_score : calculating winogrande score over selected tasks.
9
- 1 100.0000 -2.996793 -3.789603 1 1
10
- 2 100.0000 -1.766274 -1.964174 1 1
11
- 3 100.0000 -3.105679 -2.742753 2 2
12
- 4 100.0000 -4.772802 -4.813527 1 1
13
- 5 80.0000 -2.714985 -3.275630 1 2
14
- 6 83.3333 -4.758648 -4.835182 1 1
15
- 7 85.7143 -4.347487 -2.950920 2 2
16
- 8 87.5000 -3.265830 -2.586989 2 2
17
- 9 88.8889 -4.935044 -6.023360 1 1
18
- 10 90.0000 -1.899926 -1.770180 2 2
19
- 11 90.9091 -8.747679 -5.112798 2 2
20
- 12 91.6667 -3.286166 -2.305194 2 2
21
- 13 92.3077 -3.066601 -5.001747 1 1
22
- 14 92.8571 -3.273115 -3.760223 1 1
23
- 15 86.6667 -4.251678 -6.021888 1 2
24
- 16 87.5000 -4.159921 -3.389218 2 2
25
- 17 82.3529 -6.431203 -6.855700 1 2
26
- 18 83.3333 -1.750394 -3.813005 1 1
27
- 19 84.2105 -3.460366 -5.244384 1 1
28
- 20 80.0000 -2.844701 -2.785684 2 1
29
- 21 80.9524 -1.709715 -3.168737 1 1
30
- 22 77.2727 -2.556822 -2.685428 1 2
31
- 23 78.2609 -4.829033 -5.060349 1 1
32
- 24 75.0000 -2.669718 -2.347223 2 1
33
- 25 72.0000 -2.061445 -2.168485 1 2
34
- 26 73.0769 -3.779017 -4.367446 1 1
35
- 27 70.3704 -5.095303 -5.206311 1 2
36
- 28 71.4286 -5.027713 -5.151516 1 1
37
- 29 68.9655 -3.932449 -3.804086 2 1
38
- 30 70.0000 -2.096231 -1.928476 2 2
39
- 31 70.9677 -5.237000 -5.475961 1 1
40
- 32 71.8750 -4.361012 -6.069465 1 1
41
- 33 72.7273 -4.592360 -5.914653 1 1
42
- 34 73.5294 -2.782722 -3.035621 1 1
43
- 35 74.2857 -4.796330 -4.741952 2 2
44
- 36 75.0000 -3.935987 -3.863780 2 2
45
- 37 75.6757 -3.144315 -3.483248 1 1
46
- 38 76.3158 -3.669079 -3.727727 1 1
47
- 39 76.9231 -5.874491 -4.560673 2 2
48
- 40 75.0000 -5.506020 -6.174532 1 2
49
- 41 75.6098 -2.217236 -2.676707 1 1
50
- 42 76.1905 -2.207700 -3.184650 1 1
51
- 43 74.4186 -3.231467 -3.248201 1 2
52
- 44 75.0000 -2.607497 -1.884352 2 2
53
- 45 75.5556 -0.198711 -10.986690 1 1
54
- 46 76.0870 -2.612989 -2.819827 1 1
55
- 47 74.4681 -3.942782 -4.074140 1 2
56
- 48 75.0000 -3.894149 -3.774293 2 2
57
- 49 73.4694 -2.990882 -1.844874 2 1
58
- 50 74.0000 -2.628322 -2.618844 2 2
59
- 51 74.5098 -2.210348 -2.712790 1 1
60
- 52 75.0000 -3.411656 -2.275015 2 2
61
- 53 75.4717 -4.197607 -4.479537 1 1
62
- 54 75.9259 -3.089318 -2.761200 2 2
63
- 55 76.3636 -3.068943 -2.900190 2 2
64
- 56 75.0000 -3.554597 -4.546863 1 2
65
- 57 75.4386 -3.490202 -3.053506 2 2
66
- 58 75.8621 -3.565822 -4.318027 1 1
67
- 59 76.2712 -3.433472 -3.387890 2 2
68
- 60 76.6667 -4.896966 -4.524152 2 2
69
- 61 77.0492 -3.328510 -3.593264 1 1
70
- 62 77.4194 -5.742146 -6.108333 1 1
71
- 63 76.1905 -4.577035 -4.907355 1 2
72
- 64 76.5625 -3.665051 -3.752741 1 1
73
- 65 76.9231 -5.356737 -4.527686 2 2
74
- 66 77.2727 -3.602356 -3.952232 1 1
75
- 67 77.6119 -3.296406 -2.593937 2 2
76
- 68 77.9412 -5.390588 -5.291931 2 2
77
- 69 78.2609 -4.565038 -7.042857 1 1
78
- 70 78.5714 -3.558702 -3.083185 2 2
79
- 71 77.4648 -3.301721 -2.582038 2 1
80
- 72 77.7778 -3.997717 -3.608180 2 2
81
- 73 76.7123 -2.608677 -2.440044 2 1
82
- 74 77.0270 -2.356861 -2.828512 1 1
83
- 75 77.3333 -2.003587 -2.507473 1 1
84
- 76 77.6316 -2.362700 -4.847168 1 1
85
- 77 77.9221 -2.722636 -2.179331 2 2
86
- 78 78.2051 -0.895315 -1.672489 1 1
87
- 79 77.2152 -2.680960 -3.403619 1 2
88
- 80 77.5000 -2.107623 -3.633865 1 1
89
- 81 77.7778 -6.005514 -4.637192 2 2
90
- 82 76.8293 -3.322503 -3.850119 1 2
91
- 83 77.1084 -6.255712 -4.952051 2 2
92
- 84 77.3810 -2.452492 -2.326011 2 2
93
- 85 77.6471 -4.447654 -5.041602 1 1
94
- 86 76.7442 -5.796220 -6.062174 1 2
95
- 87 75.8621 -2.028294 -1.872511 2 1
96
- 88 75.0000 -4.828620 -4.488709 2 1
97
- 89 75.2809 -3.657195 -3.996683 1 1
98
- 90 75.5556 -4.331297 -3.665882 2 2
99
- 91 74.7253 -5.074696 -4.861190 2 1
100
- 92 75.0000 -5.459994 -4.412688 2 2
101
- 93 74.1935 -3.086576 -3.109992 1 2
102
- 94 74.4681 -5.266660 -3.760736 2 2
103
- 95 74.7368 -3.957025 -4.258834 1 1
104
- 96 75.0000 -1.870796 -2.810139 1 1
105
- 97 75.2577 -5.282516 -5.763870 1 1
106
- 98 75.5102 -3.970552 -3.717415 2 2
107
- 99 74.7475 -2.843543 -2.676669 2 1
108
- 100 75.0000 -4.812362 -4.172173 2 2
109
- 101 74.2574 -4.788298 -4.814284 1 2
110
- 102 74.5098 -3.537376 -3.746772 1 1
111
- 103 74.7573 -2.830078 -2.692385 2 2
112
- 104 75.0000 -7.381618 -7.088392 2 2
113
- 105 75.2381 -1.467835 -2.401019 1 1
114
- 106 75.4717 -2.968574 -2.655803 2 2
115
- 107 75.7009 -2.568068 -2.229604 2 2
116
- 108 75.9259 -4.217192 -3.148280 2 2
117
- 109 75.2294 -4.413562 -5.167275 1 2
118
- 110 75.4545 -4.730681 -3.712240 2 2
119
- 111 75.6757 -3.117462 -2.716470 2 2
120
- 112 75.0000 -4.030105 -3.645368 2 1
121
- 113 74.3363 -5.165231 -6.199688 1 2
122
- 114 74.5614 -4.381731 -4.977725 1 1
123
- 115 74.7826 -1.258990 -2.169746 1 1
124
- 116 75.0000 -3.343508 -2.776088 2 2
125
- 117 74.3590 -4.528759 -4.789988 1 2
126
- 118 74.5763 -1.551563 -2.043292 1 1
127
- 119 74.7899 -1.911865 -2.713878 1 1
128
- 120 74.1667 -2.837347 -2.503358 2 1
129
- 121 74.3802 -1.050675 -0.746704 2 2
130
- 122 74.5902 -6.167936 -5.364534 2 2
131
- 123 74.7967 -3.450576 -4.141653 1 1
132
- 124 75.0000 -2.931594 -2.639284 2 2
133
- 125 74.4000 -2.427118 -2.439908 1 2
134
- 126 74.6032 -6.790971 -6.543124 2 2
135
- 127 74.8031 -2.259541 -1.581128 2 2
136
- 128 75.0000 -1.640095 -1.875864 1 1
137
- 129 75.1938 -2.552879 -1.639615 2 2
138
- 130 75.3846 -6.180727 -5.473829 2 2
139
- 131 74.8092 -3.420510 -3.463171 1 2
140
- 132 75.0000 -2.212027 -1.744500 2 2
141
- 133 75.1880 -2.325607 -4.127159 1 1
142
- 134 75.3731 -5.048766 -4.561748 2 2
143
- 135 75.5556 -4.569615 -5.482002 1 1
144
- 136 75.7353 -3.659473 -2.812425 2 2
145
- 137 75.9124 -5.173557 -5.478113 1 1
146
- 138 75.3623 -3.515645 -4.120496 1 2
147
- 139 75.5396 -4.548652 -4.824677 1 1
148
- 140 75.7143 -1.403230 -1.210400 2 2
149
- 141 75.8865 -5.302089 -4.204167 2 2
150
- 142 76.0563 -3.625842 -2.849770 2 2
151
- 143 76.2238 -4.139919 -7.474318 1 1
152
- 144 75.6944 -2.417112 -2.327529 2 1
153
- 145 75.1724 -1.230964 -13.817912 1 2
154
- 146 74.6575 -3.954980 -4.205729 1 2
155
- 147 74.8299 -4.979667 -4.354348 2 2
156
- 148 75.0000 -1.917653 -2.286666 1 1
157
- 149 75.1678 -4.450720 -4.272093 2 2
158
- 150 75.3333 -4.520557 -3.774753 2 2
159
- 151 74.8344 -1.958070 -1.883344 2 1
160
- 152 74.3421 -4.273956 -4.644961 1 2
161
- 153 74.5098 -2.784101 -3.365832 1 1
162
- 154 74.6753 -3.508737 -3.749246 1 1
163
- 155 74.1935 -2.698134 -2.955358 1 2
164
- 156 74.3590 -2.026529 -2.269351 1 1
165
- 157 74.5223 -2.769491 -4.068587 1 1
166
- 158 74.6835 -4.638029 -3.860526 2 2
167
- 159 74.8428 -3.770250 -4.860247 1 1
168
- 160 75.0000 -3.891957 -3.604249 2 2
169
- 161 75.1553 -1.696185 -2.267404 1 1
170
- 162 74.6914 -0.053286 -12.346473 1 2
171
- 163 74.8466 -2.537913 -2.788797 1 1
172
- 164 75.0000 -5.371875 -4.409680 2 2
173
- 165 75.1515 -3.958990 -3.907169 2 2
174
- 166 75.3012 -4.751889 -4.145329 2 2
175
- 167 74.8503 -4.569721 -3.970534 2 1
176
- 168 75.0000 -5.023438 -3.762268 2 2
177
- 169 74.5562 -4.389508 -5.216572 1 2
178
- 170 74.7059 -2.450009 -3.433662 1 1
179
- 171 74.8538 -3.617034 -4.431140 1 1
180
- 172 75.0000 -4.203106 -5.285625 1 1
181
- 173 75.1445 -3.042644 -2.674614 2 2
182
- 174 75.2874 -5.664001 -5.542048 2 2
183
- 175 75.4286 -4.339079 -4.574714 1 1
184
- 176 75.5682 -2.328093 -4.153268 1 1
185
- 177 75.7062 -6.247530 -6.193766 2 2
186
- 178 75.2809 -2.593415 -2.305955 2 1
187
- 179 75.4190 -2.358292 -2.751229 1 1
188
- 180 75.5556 -6.140658 -5.593593 2 2
189
- 181 75.6906 -3.834333 -2.786162 2 2
190
- 182 75.8242 -4.417058 -3.326639 2 2
191
- 183 75.9563 -5.225430 -5.024895 2 2
192
- 184 76.0870 -1.519560 -2.457959 1 1
193
- 185 75.6757 -4.537733 -4.674183 1 2
194
- 186 75.8065 -3.017460 -4.009912 1 1
195
- 187 75.9358 -4.754910 -4.402692 2 2
196
- 188 75.5319 -4.346316 -3.276922 2 1
197
- 189 75.1323 -4.113223 -4.601685 1 2
198
- 190 75.2632 -4.851724 -4.750816 2 2
199
- 191 75.3927 -3.528849 -4.138034 1 1
200
- 192 75.5208 -4.858772 -3.759407 2 2
201
- 193 75.6477 -4.136450 -4.127154 2 2
202
- 194 75.7732 -1.983138 -1.572278 2 2
203
- 195 75.3846 -2.994064 -2.816575 2 1
204
- 196 75.5102 -3.580158 -3.320386 2 2
205
- 197 75.6345 -1.357802 -4.517101 1 1
206
- 198 75.7576 -4.187136 -7.671094 1 1
207
- 199 75.8794 -5.214767 -4.111004 2 2
208
- 200 76.0000 -3.051458 -2.835288 2 2
209
- 201 76.1194 -2.873885 -3.233990 1 1
210
- 202 76.2376 -8.201182 -7.690842 2 2
211
- 203 76.3547 -3.157725 -2.558254 2 2
212
- 204 76.4706 -2.249370 -3.049535 1 1
213
- 205 76.5854 -3.612201 -2.395843 2 2
214
- 206 76.2136 -4.293222 -4.671841 1 2
215
- 207 76.3285 -3.205755 -3.777888 1 1
216
- 208 76.4423 -3.261423 -2.653475 2 2
217
- 209 76.5550 -4.270988 -3.191008 2 2
218
- 210 76.6667 -6.562067 -6.446175 2 2
219
- 211 76.3033 -4.949271 -4.552193 2 1
220
- 212 76.4151 -4.925678 -4.160634 2 2
221
- 213 76.0563 -3.692701 -4.689421 1 2
222
- 214 76.1682 -3.214695 -3.254238 1 1
223
- 215 76.2791 -4.352945 -4.068777 2 2
224
- 216 75.9259 -3.218786 -3.056115 2 1
225
- 217 76.0369 -1.408949 -2.042867 1 1
226
- 218 76.1468 -3.044974 -2.784997 2 2
227
- 219 76.2557 -3.284827 -3.360150 1 1
228
- 220 75.9091 -4.153646 -4.512938 1 2
229
- 221 76.0181 -4.000010 -8.021618 1 1
230
- 222 75.6757 -0.465218 -9.142362 1 2
231
- 223 75.7848 -3.469725 -3.466762 2 2
232
- 224 75.8929 -5.471722 -4.648843 2 2
233
- 225 75.5556 -2.933905 -3.091155 1 2
234
- 226 75.6637 -2.789979 -3.624089 1 1
235
- 227 75.7709 -3.067627 -3.182287 1 1
236
- 228 75.8772 -1.565137 -2.161165 1 1
237
- 229 75.9825 -3.463956 -3.271909 2 2
238
- 230 76.0870 -3.249723 -3.608573 1 1
239
- 231 76.1905 -2.552293 -3.209325 1 1
240
- 232 76.2931 -2.246167 -1.817492 2 2
241
- 233 76.3948 -1.329862 -2.030679 1 1
242
- 234 76.4957 -3.622311 -3.187947 2 2
243
- 235 76.5957 -3.055628 -2.623724 2 2
244
- 236 76.2712 -1.682006 -1.477149 2 1
245
- 237 76.3713 -3.080188 -3.085534 1 1
246
- 238 76.4706 -2.962541 -2.547988 2 2
247
- 239 76.1506 -2.666400 -2.731245 1 2
248
- 240 76.2500 -7.214667 -7.125328 2 2
249
- 241 75.9336 -2.501472 -3.371696 1 2
250
- 242 76.0331 -5.908028 -5.580513 2 2
251
- 243 75.7202 -3.860632 -3.391875 2 1
252
- 244 75.8197 -1.883807 -3.804210 1 1
253
- 245 75.9184 -3.894141 -4.493126 1 1
254
- 246 76.0163 -2.700037 -3.103764 1 1
255
- 247 76.1134 -3.221263 -3.184342 2 2
256
- 248 76.2097 -3.610471 -3.176808 2 2
257
- 249 76.3052 -3.232613 -2.076124 2 2
258
- 250 76.4000 -2.338475 -2.787102 1 1
259
- 251 76.4940 -1.800264 -1.949910 1 1
260
- 252 76.5873 -6.783570 -6.302971 2 2
261
- 253 76.2846 -5.349864 -5.160089 2 1
262
- 254 75.9843 -4.982727 -5.127346 1 2
263
- 255 76.0784 -3.409741 -4.375062 1 1
264
- 256 76.1719 -2.901046 -2.914320 1 1
265
- 257 75.8755 -2.821574 -2.226196 2 1
266
- 258 75.5814 -3.911757 -3.969365 1 2
267
- 259 75.6757 -5.005537 -5.585522 1 1
268
- 260 75.7692 -5.583511 -5.621291 1 1
269
- 261 75.8621 -3.696566 -3.679752 2 2
270
- 262 75.9542 -4.859819 -4.496526 2 2
271
- 263 76.0456 -4.100757 -4.261125 1 1
272
- 264 76.1364 -4.210273 -3.670712 2 2
273
- 265 76.2264 -1.164613 -2.127860 1 1
274
- 266 76.3158 -5.261276 -6.727902 1 1
275
- 267 76.0300 -3.211686 -3.148001 2 1
276
- 268 76.1194 -3.078450 -3.133437 1 1
277
- 269 76.2082 -3.016114 -2.811125 2 2
278
- 270 76.2963 -2.404410 -3.252763 1 1
279
- 271 76.3838 -3.168137 -2.954263 2 2
280
- 272 76.4706 -3.898564 -3.624198 2 2
281
- 273 76.5568 -3.564529 -3.468776 2 2
282
- 274 76.6423 -3.500096 -3.414810 2 2
283
- 275 76.3636 -3.092688 -3.038071 2 1
284
- 276 76.0870 -2.961086 -2.551814 2 1
285
- 277 76.1733 -2.772878 -3.509658 1 1
286
- 278 76.2590 -2.631871 -2.741297 1 1
287
- 279 76.3441 -5.655190 -3.426222 2 2
288
- 280 76.4286 -4.057175 -3.772405 2 2
289
- 281 76.5125 -2.716693 -1.778921 2 2
290
- 282 76.5957 -4.453942 -4.024188 2 2
291
- 283 76.3251 -3.362161 -4.012121 1 2
292
- 284 76.4085 -2.583750 -3.167080 1 1
293
- 285 76.4912 -3.285878 -4.888389 1 1
294
- 286 76.2238 -3.536055 -3.492226 2 1
295
- 287 76.3066 -3.184714 -4.404004 1 1
296
- 288 76.3889 -3.195195 -3.085974 2 2
297
- 289 76.4706 -1.995651 -2.383334 1 1
298
- 290 76.5517 -3.913084 -4.630651 1 1
299
- 291 76.6323 -2.149934 -3.276830 1 1
300
- 292 76.7123 -3.101957 -2.365869 2 2
301
- 293 76.7918 -3.904570 -3.340560 2 2
302
- 294 76.8707 -4.022257 -4.241902 1 1
303
- 295 76.6102 -2.019350 -1.997115 2 1
304
- 296 76.6892 -5.266312 -5.012389 2 2
305
- 297 76.4310 -1.632764 -1.944261 1 2
306
- 298 76.5101 -5.416333 -5.283889 2 2
307
- 299 76.5886 -2.471399 -2.394294 2 2
308
- 300 76.6667 -2.074692 -2.299829 1 1
309
- 301 76.7442 -3.973708 -4.406394 1 1
310
- 302 76.8212 -4.210261 -4.101169 2 2
311
- 303 76.5677 -9.014982 -8.838298 2 1
312
- 304 76.6447 -4.439731 -3.524297 2 2
313
- 305 76.7213 -3.545498 -3.214848 2 2
314
- 306 76.7974 -2.365666 -2.023725 2 2
315
- 307 76.8730 -2.900132 -3.202796 1 1
316
- 308 76.9481 -5.696802 -4.302923 2 2
317
- 309 77.0227 -3.824618 -4.715470 1 1
318
- 310 77.0968 -5.947269 -5.676313 2 2
319
- 311 77.1704 -0.659399 -4.594034 1 1
320
- 312 77.2436 -4.166177 -4.307451 1 1
321
- 313 76.9968 -4.610192 -4.868622 1 2
322
- 314 77.0701 -4.415612 -4.112204 2 2
323
- 315 77.1429 -1.945122 -3.039606 1 1
324
- 316 77.2152 -7.485093 -5.679261 2 2
325
- 317 76.9716 -4.961111 -4.909097 2 1
326
- 318 77.0440 -4.766862 -4.457537 2 2
327
- 319 77.1160 -1.304338 -1.530370 1 1
328
- 320 77.1875 -4.290006 -5.225870 1 1
329
- 321 77.2586 -2.957040 -2.943673 2 2
330
- 322 77.0186 -2.319967 -2.664924 1 2
331
- 323 77.0898 -3.790599 -3.952383 1 1
332
- 324 76.8519 -4.312448 -4.146027 2 1
333
- 325 76.9231 -5.522614 -4.736701 2 2
334
- 326 76.9939 -2.623704 -1.859680 2 2
335
- 327 77.0642 -2.517128 -3.061715 1 1
336
- 328 76.8293 -4.391160 -4.275026 2 1
337
- 329 76.8997 -3.476714 -7.797326 1 1
338
- 330 76.9697 -1.805292 -2.103216 1 1
339
- 331 76.7372 -4.154588 -3.304633 2 1
340
- 332 76.8072 -2.090883 -3.315475 1 1
341
- 333 76.8769 -3.925348 -1.827942 2 2
342
- 334 76.9461 -4.185332 -6.036137 1 1
343
- 335 76.7164 -5.530205 -4.941094 2 1
344
- 336 76.4881 -3.648689 -3.937146 1 2
345
- 337 76.5579 -2.139735 -1.498239 2 2
346
- 338 76.3314 -3.337484 -2.962694 2 1
347
- 339 76.1062 -5.535268 -5.903919 1 2
348
- 340 76.1765 -4.778726 -5.582592 1 1
349
- 341 76.2463 -3.966188 -5.239852 1 1
350
- 342 76.3158 -5.209695 -4.615998 2 2
351
- 343 76.3848 -3.048664 -3.682488 1 1
352
- 344 76.4535 -2.106262 -3.274798 1 1
353
- 345 76.5217 -0.940180 -3.425870 1 1
354
- 346 76.5896 -6.192531 -6.044642 2 2
355
- 347 76.6571 -3.811952 -3.369901 2 2
356
- 348 76.7241 -0.795600 -1.796034 1 1
357
- 349 76.5043 -3.396677 -4.228620 1 2
358
- 350 76.2857 -1.919666 -2.554481 1 2
359
- 351 76.3533 -2.809038 -3.594467 1 1
360
- 352 76.4205 -3.529017 -2.908872 2 2
361
- 353 76.4873 -2.454048 -2.696650 1 1
362
- 354 76.5537 -5.588192 -4.703644 2 2
363
- 355 76.6197 -2.782488 -2.303570 2 2
364
- 356 76.4045 -3.373203 -3.469512 1 2
365
- 357 76.4706 -4.713520 -4.386007 2 2
366
- 358 76.5363 -2.275677 -2.035708 2 2
367
- 359 76.6017 -5.150197 -3.056240 2 2
368
- 360 76.6667 -3.747510 -3.682742 2 2
369
- 361 76.7313 -3.393471 -1.681831 2 2
370
- 362 76.7956 -5.993749 -5.357779 2 2
371
- 363 76.5840 -2.983167 -2.691014 2 1
372
- 364 76.6484 -5.846803 -6.076901 1 1
373
- 365 76.4384 -5.484076 -5.479404 2 1
374
- 366 76.2295 -2.720112 -7.299872 1 2
375
- 367 76.2943 -4.892259 -5.032633 1 1
376
- 368 76.3587 -3.951429 -3.759658 2 2
377
- 369 76.1518 -4.018383 -4.216836 1 2
378
- 370 75.9459 -5.310339 -4.707185 2 1
379
- 371 76.0108 -2.159032 -3.058471 1 1
380
- 372 76.0753 -2.442824 -2.618950 1 1
381
- 373 75.8713 -4.031174 -3.076410 2 1
382
- 374 75.9358 -2.369320 -2.751298 1 1
383
- 375 76.0000 -3.594907 -2.836102 2 2
384
- 376 76.0638 -4.571621 -3.541450 2 2
385
- 377 76.1273 -3.831064 -3.301183 2 2
386
- 378 75.9259 -2.991852 -3.272280 1 2
387
- 379 75.7256 -0.702349 -2.296379 1 2
388
- 380 75.7895 -5.665767 -6.325861 1 1
389
- 381 75.8530 -9.863632 -9.142766 2 2
390
- 382 75.9162 -2.995848 -2.949385 2 2
391
- 383 75.9791 -2.325811 -2.044151 2 2
392
- 384 75.7812 -3.473151 -2.845114 2 1
393
- 385 75.5844 -1.162040 -1.102872 2 1
394
- 386 75.6477 -2.213192 -13.649847 1 1
395
- 387 75.7106 -1.973629 -2.527582 1 1
396
- 388 75.7732 -5.800248 -5.634243 2 2
397
- 389 75.5784 -2.974243 -3.031593 1 2
398
- 390 75.6410 -3.118165 -3.335625 1 1
399
- 391 75.4476 -3.976912 -4.403868 1 2
400
- 392 75.2551 -4.602829 -3.832384 2 1
401
- 393 75.3181 -6.177592 -5.717938 2 2
402
- 394 75.3807 -2.235994 -3.726058 1 1
403
- 395 75.4430 -3.843143 -3.290020 2 2
404
- 396 75.5051 -2.878797 -3.621457 1 1
405
- 397 75.5668 -6.635840 -7.360071 1 1
406
- 398 75.6281 -2.721007 -5.007327 1 1
407
- 399 75.6892 -2.520669 -3.681499 1 1
408
- 400 75.7500 -1.200240 -6.572565 1 1
409
- 401 75.8105 -3.539259 -3.988800 1 1
410
- 402 75.8706 -3.603897 -3.508890 2 2
411
- 403 75.6824 -5.707330 -6.213705 1 2
412
- 404 75.7426 -4.300483 -3.430648 2 2
413
- 405 75.8025 -3.475313 -4.329596 1 1
414
- 406 75.8621 -7.457608 -8.319613 1 1
415
- 407 75.6757 -4.689802 -4.811605 1 2
416
- 408 75.4902 -3.277456 -3.479868 1 2
417
- 409 75.5501 -3.754601 -4.585982 1 1
418
- 410 75.6098 -3.695467 -3.950722 1 1
419
- 411 75.6691 -3.898967 -5.293450 1 1
420
- 412 75.7282 -3.905590 -3.658594 2 2
421
- 413 75.7869 -3.425626 -2.788047 2 2
422
- 414 75.6039 -3.973189 -4.317998 1 2
423
- 415 75.6627 -3.496355 -3.317596 2 2
424
- 416 75.7212 -4.173635 -2.273693 2 2
425
- 417 75.7794 -4.193304 -4.908659 1 1
426
- 418 75.8373 -4.718334 -4.924798 1 1
427
- 419 75.8950 -3.959394 -3.887866 2 2
428
- 420 75.7143 -3.126073 -3.346396 1 2
429
- 421 75.7720 -4.096146 -3.733175 2 2
430
- 422 75.8294 -2.156788 -2.328066 1 1
431
- 423 75.6501 -3.473773 -3.468104 2 1
432
- 424 75.7075 -4.858806 -5.681521 1 1
433
- 425 75.7647 -5.306388 -3.418925 2 2
434
- 426 75.8216 -2.082694 -1.939879 2 2
435
- 427 75.8782 -3.696792 -4.349473 1 1
436
- 428 75.7009 -4.180840 -4.103384 2 1
437
- 429 75.7576 -3.062952 -3.019620 2 2
438
- 430 75.8140 -5.243644 -5.822688 1 1
439
- 431 75.8701 -3.667053 -5.365144 1 1
440
- 432 75.6944 -3.280022 -3.918091 1 2
441
- 433 75.7506 -2.550816 -1.803087 2 2
442
- 434 75.8065 -7.125130 -7.601790 1 1
443
- 435 75.8621 -3.648045 -3.736653 1 1
444
- 436 75.6881 -2.259445 -2.201411 2 1
445
- 437 75.7437 -4.345041 -3.887327 2 2
446
- 438 75.7991 -3.876300 -3.675940 2 2
447
- 439 75.8542 -2.824836 -3.388191 1 1
448
- 440 75.9091 -3.598480 -4.050237 1 1
449
- 441 75.9637 -3.754327 -4.776167 1 1
450
- 442 76.0181 -4.236638 -4.331945 1 1
451
- 443 76.0722 -1.875090 -3.270932 1 1
452
- 444 76.1261 -5.768084 -5.651508 2 2
453
- 445 76.1798 -1.610197 -2.314270 1 1
454
- 446 76.2332 -4.334198 -2.657498 2 2
455
- 447 76.0626 -3.317813 -2.893815 2 1
456
- 448 76.1161 -4.792416 -4.607845 2 2
457
- 449 76.1693 -2.574572 -2.061392 2 2
458
- 450 76.2222 -6.145314 -5.028388 2 2
459
- 451 76.2749 -4.022354 -3.651776 2 2
460
- 452 76.1062 -4.450176 -3.968100 2 1
461
- 453 76.1589 -4.337204 -3.566882 2 2
462
- 454 76.2115 -3.539694 -3.986739 1 1
463
- 455 76.0440 -2.112048 -10.486910 1 2
464
- 456 75.8772 -4.289437 -4.371389 1 2
465
- 457 75.7112 -7.888005 -7.670916 2 1
466
- 458 75.7642 -3.674484 -2.847732 2 2
467
- 459 75.8170 -4.073552 -3.935769 2 2
468
- 460 75.8696 -0.229504 -9.430177 1 1
469
- 461 75.9219 -2.675769 -3.646678 1 1
470
- 462 75.7576 -5.505221 -5.145451 2 1
471
- 463 75.8099 -1.430744 -1.668586 1 1
472
- 464 75.8621 -2.093298 -2.501263 1 1
473
- 465 75.9140 -3.619438 -2.794858 2 2
474
- 466 75.7511 -5.496015 -5.570718 1 2
475
- 467 75.5889 -5.134415 -4.929653 2 1
476
- 468 75.6410 -4.331724 -3.100625 2 2
477
- 469 75.4797 -1.843925 -1.679950 2 1
478
- 470 75.5319 -2.060586 -1.859825 2 2
479
- 471 75.5839 -2.743659 -3.033195 1 1
480
- 472 75.6356 -3.424139 -3.398317 2 2
481
- 473 75.6871 -4.171948 -3.296490 2 2
482
- 474 75.7384 -4.088644 -5.450495 1 1
483
- 475 75.7895 -2.043334 -1.802483 2 2
484
- 476 75.6303 -3.913585 -4.519627 1 2
485
- 477 75.4717 -4.260455 -4.747236 1 2
486
- 478 75.5230 -2.553922 -2.624067 1 1
487
- 479 75.5741 -2.906059 -2.927147 1 1
488
- 480 75.6250 -5.833463 -5.659633 2 2
489
- 481 75.6757 -2.194368 -2.491631 1 1
490
- 482 75.5187 -4.390950 -5.063113 1 2
491
- 483 75.5694 -4.560465 -5.222022 1 1
492
- 484 75.4132 -2.852795 -2.385432 2 1
493
- 485 75.4639 -1.398950 -2.042329 1 1
494
- 486 75.5144 -2.772468 -3.312011 1 1
495
- 487 75.5647 -3.738425 -3.346433 2 2
496
- 488 75.6148 -4.503445 -3.404963 2 2
497
- 489 75.6646 -2.320996 -2.780909 1 1
498
- 490 75.7143 -2.823640 -3.274848 1 1
499
- 491 75.7637 -5.460179 -4.463346 2 2
500
- 492 75.8130 -5.137753 -4.260631 2 2
501
- 493 75.8621 -1.480595 -2.825174 1 1
502
- 494 75.9109 -3.450761 -2.732931 2 2
503
- 495 75.7576 -3.937860 -3.920830 2 1
504
- 496 75.8065 -3.350376 -3.099708 2 2
505
- 497 75.8551 -4.938697 -5.084080 1 1
506
- 498 75.7028 -1.913092 -2.107364 1 2
507
- 499 75.7515 -3.002001 -3.484062 1 1
508
- 500 75.8000 -5.472087 -4.893366 2 2
509
- 501 75.8483 -2.490570 -3.037509 1 1
510
- 502 75.8964 -1.509306 -2.734908 1 1
511
- 503 75.9443 -3.742564 -3.293942 2 2
512
- 504 75.9921 -2.647938 -2.874998 1 1
513
- 505 75.8416 -6.915524 -7.230917 1 2
514
- 506 75.8893 -3.903782 -4.877000 1 1
515
- 507 75.9369 -3.278415 -3.513410 1 1
516
- 508 75.9843 -4.579016 -3.749412 2 2
517
- 509 76.0314 -6.581466 -9.198807 1 1
518
- 510 76.0784 -3.886309 -4.672490 1 1
519
- 511 76.1252 -5.805210 -4.520743 2 2
520
- 512 76.1719 -3.205832 -3.612730 1 1
521
- 513 76.2183 -3.021719 -2.979192 2 2
522
- 514 76.0700 -4.781627 -3.723665 2 1
523
- 515 76.1165 -6.097193 -7.971423 1 1
524
- 516 76.1628 -4.280977 -4.584671 1 1
525
- 517 76.2089 -1.661950 -2.108455 1 1
526
- 518 76.2548 -2.957500 -2.831029 2 2
527
- 519 76.1079 -5.964907 -6.160328 1 2
528
- 520 76.1538 -2.703760 -2.596437 2 2
529
- 521 76.1996 -2.096182 -2.235035 1 1
530
- 522 76.2452 -4.241663 -2.633535 2 2
531
- 523 76.2906 -3.836968 -3.141414 2 2
532
- 524 76.3359 -3.645159 -4.397972 1 1
533
- 525 76.3810 -4.171119 -2.010952 2 2
534
- 526 76.4259 -0.739266 -14.519868 1 1
535
- 527 76.4706 -6.189020 -4.092399 2 2
536
- 528 76.5152 -6.129576 -5.848916 2 2
537
- 529 76.5595 -5.048522 -5.651372 1 1
538
- 530 76.6038 -4.603324 -2.910119 2 2
539
- 531 76.6478 -3.601290 -3.917755 1 1
540
- 532 76.6917 -2.025319 -3.059090 1 1
541
- 533 76.7355 -3.966353 -2.524776 2 2
542
- 534 76.7790 -2.301569 -2.917866 1 1
543
- 535 76.6355 -3.615034 -3.626290 1 2
544
- 536 76.4925 -2.510150 -2.260227 2 1
545
- 537 76.5363 -5.083160 -6.039402 1 1
546
- 538 76.5799 -2.066234 -4.982938 1 1
547
- 539 76.6234 -4.074865 -1.924515 2 2
548
- 540 76.6667 -2.946506 -2.701921 2 2
549
- 541 76.7098 -2.699617 -3.366931 1 1
550
- 542 76.5683 -3.363741 -4.583934 1 2
551
- 543 76.6114 -5.594193 -3.939120 2 2
552
- 544 76.6544 -3.164377 -3.561853 1 1
553
- 545 76.5138 -2.163738 -4.023462 1 2
554
- 546 76.5568 -2.800000 -4.169002 1 1
555
- 547 76.5996 -5.750423 -5.273779 2 2
556
- 548 76.4599 -4.717620 -4.714372 2 1
557
- 549 76.5027 -5.160970 -5.544577 1 1
558
- 550 76.5455 -2.712624 -2.641489 2 2
559
- 551 76.4065 -0.985675 -12.342457 1 2
560
- 552 76.4493 -2.349810 -2.620615 1 1
561
- 553 76.4919 -5.077174 -4.913177 2 2
562
- 554 76.5343 -4.848772 -4.835861 2 2
563
- 555 76.5766 -2.068702 -1.635468 2 2
564
- 556 76.6187 -5.291370 -5.205962 2 2
565
- 557 76.4811 -4.905457 -4.207930 2 1
566
- 558 76.5233 -1.863149 -3.059641 1 1
567
- 559 76.5653 -2.601057 -2.542996 2 2
568
- 560 76.4286 -2.681178 -2.617763 2 1
569
- 561 76.4706 -2.031772 -2.681414 1 1
570
- 562 76.5125 -0.685210 -1.906248 1 1
571
- 563 76.3766 -1.467885 -2.449925 1 2
572
- 564 76.4184 -2.894483 -3.365100 1 1
573
- 565 76.4602 -4.752205 -4.328282 2 2
574
- 566 76.3251 -4.632388 -4.526495 2 1
575
- 567 76.3668 -2.094764 -1.729849 2 2
576
- 568 76.4085 -7.530218 -2.956098 2 2
577
- 569 76.2742 -5.567005 -4.539991 2 1
578
- 570 76.3158 -3.984439 -5.152901 1 1
579
- 571 76.3573 -4.189381 -2.991746 2 2
580
- 572 76.3986 -2.505805 -2.225173 2 2
581
- 573 76.2653 -3.922523 -4.138556 1 2
582
- 574 76.3066 -3.849652 -3.444874 2 2
583
- 575 76.3478 -5.385030 -5.319433 2 2
584
- 576 76.3889 -3.700586 -4.270683 1 1
585
- 577 76.4298 -4.124111 -3.817549 2 2
586
- 578 76.4706 -3.072803 -2.911284 2 2
587
- 579 76.3385 -4.093185 -4.666516 1 2
588
- 580 76.3793 -2.344485 -2.667194 1 1
589
- 581 76.2478 -6.259533 -6.186646 2 1
590
- 582 76.1168 -5.483279 -5.496644 1 2
591
- 583 76.1578 -4.946896 -4.561930 2 2
592
- 584 76.0274 -4.396478 -4.114932 2 1
593
- 585 76.0684 -0.449784 -1.420253 1 1
594
- 586 76.1092 -3.621907 -3.782053 1 1
595
- 587 75.9796 -5.175301 -4.944868 2 1
596
- 588 76.0204 -4.163341 -2.946942 2 2
597
- 589 75.8913 -3.829644 -4.061519 1 2
598
- 590 75.9322 -4.230101 -3.486883 2 2
599
- 591 75.9729 -3.862708 -3.420812 2 2
600
- 592 76.0135 -3.017377 -3.945351 1 1
601
- 593 76.0540 -3.888592 -3.302703 2 2
602
- 594 75.9259 -3.941840 -3.732743 2 1
603
- 595 75.7983 -1.609578 -1.634693 1 2
604
- 596 75.8389 -5.335799 -5.081780 2 2
605
- 597 75.8794 -1.886669 -2.120328 1 1
606
- 598 75.9197 -3.026354 -4.053049 1 1
607
- 599 75.9599 -3.076608 -3.143739 1 1
608
- 600 76.0000 -4.220076 -4.000574 2 2
609
- 601 76.0399 -3.181759 -3.035837 2 2
610
- 602 76.0797 -2.286826 -3.953130 1 1
611
- 603 76.1194 -5.993765 -5.921052 2 2
612
- 604 75.9934 -3.129565 -3.233773 1 2
613
- 605 75.8678 -2.696777 -2.528663 2 1
614
- 606 75.9076 -3.275169 -2.437806 2 2
615
- 607 75.7825 -2.374406 -1.936198 2 1
616
- 608 75.8224 -2.348693 -2.751401 1 1
617
- 609 75.8621 -5.482690 -6.396708 1 1
618
- 610 75.7377 -2.763980 -2.706255 2 1
619
- 611 75.7774 -4.952557 -5.021160 1 1
620
- 612 75.8170 -3.521968 -5.432558 1 1
621
- 613 75.8564 -2.481925 -2.323402 2 2
622
- 614 75.8958 -2.921986 -1.865138 2 2
623
- 615 75.9350 -2.660064 -2.647804 2 2
624
- 616 75.9740 -3.321735 -3.278207 2 2
625
- 617 76.0130 -12.086909 -3.504364 2 2
626
- 618 75.8900 -3.992441 -4.203560 1 2
627
- 619 75.9289 -3.324369 -2.781811 2 2
628
- 620 75.9677 -4.357634 -4.735207 1 1
629
- 621 76.0064 -3.759009 -2.746660 2 2
630
- 622 76.0450 -2.473429 -2.580270 1 1
631
- 623 75.9230 -2.891268 -2.622377 2 1
632
- 624 75.9615 -4.352462 -3.801316 2 2
633
- 625 76.0000 -3.359604 -5.059487 1 1
634
- 626 76.0383 -2.587899 -2.856139 1 1
635
- 627 76.0766 -3.733299 -3.846494 1 1
636
- 628 76.1146 -2.379298 -2.247467 2 2
637
- 629 76.1526 -1.713775 -2.364034 1 1
638
- 630 76.1905 -1.324038 -1.992201 1 1
639
- 631 76.2282 -1.429799 -2.771372 1 1
640
- 632 76.2658 -1.982614 -1.312669 2 2
641
- 633 76.3033 -2.925722 -2.403818 2 2
642
- 634 76.3407 -2.213782 -2.928970 1 1
643
- 635 76.2205 -4.120617 -4.574356 1 2
644
- 636 76.2579 -2.446742 -4.167469 1 1
645
- 637 76.2951 -3.106521 -5.363451 1 1
646
- 638 76.3323 -4.761424 -5.254287 1 1
647
- 639 76.3693 -1.400741 -1.620155 1 1
648
- 640 76.2500 -1.237688 -2.172265 1 2
649
- 641 76.2871 -3.613657 -4.739127 1 1
650
- 642 76.3240 -4.348909 -2.473633 2 2
651
- 643 76.3608 -1.012929 -0.602224 2 2
652
- 644 76.3975 -4.462524 -4.921176 1 1
653
- 645 76.4341 -1.366936 -2.253034 1 1
654
- 646 76.3158 -4.346196 -4.142342 2 1
655
- 647 76.3524 -4.300319 -4.406430 1 1
656
- 648 76.3889 -3.331406 -3.232542 2 2
657
- 649 76.4253 -2.450348 -3.018650 1 1
658
- 650 76.4615 -3.246452 -3.798904 1 1
659
- 651 76.4977 -5.652684 -5.397393 2 2
660
- 652 76.5337 -5.446811 -5.128776 2 2
661
- 653 76.5697 -6.337050 -5.623506 2 2
662
- 654 76.4526 -3.266230 -3.377176 1 2
663
- 655 76.4885 -2.748854 -2.601816 2 2
664
- 656 76.5244 -3.639142 -1.744498 2 2
665
- 657 76.4079 -5.291119 -5.692897 1 2
666
- 658 76.4438 -5.454621 -4.076755 2 2
667
- 659 76.4795 -2.200758 -1.880771 2 2
668
- 660 76.5152 -3.504233 -2.396311 2 2
669
- 661 76.5507 -2.587589 -2.952455 1 1
670
- 662 76.5861 -3.546146 -3.637146 1 1
671
- 663 76.6214 -6.186169 -4.789810 2 2
672
- 664 76.6566 -2.874935 -2.886268 1 1
673
- 665 76.6917 -4.089831 -3.709962 2 2
674
- 666 76.7267 -4.916738 -5.198173 1 1
675
- 667 76.6117 -5.202742 -9.450559 1 2
676
- 668 76.6467 -2.541075 -3.483309 1 1
677
- 669 76.6816 -3.243270 -3.675570 1 1
678
- 670 76.5672 -3.619210 -3.141994 2 1
679
- 671 76.6021 -3.627103 -4.672335 1 1
680
- 672 76.6369 -2.785439 -2.640066 2 2
681
- 673 76.5230 -1.807896 -1.911643 1 2
682
- 674 76.5579 -5.993984 -6.105067 1 1
683
- 675 76.5926 -4.469804 -4.506861 1 1
684
- 676 76.4793 -1.975446 -2.362810 1 2
685
- 677 76.5140 -3.733120 -4.296697 1 1
686
- 678 76.5487 -5.051122 -5.015167 2 2
687
- 679 76.5832 -5.295827 -4.669789 2 2
688
- 680 76.6176 -3.254405 -4.924428 1 1
689
- 681 76.5051 -3.708387 -2.970325 2 1
690
- 682 76.3930 -2.230338 -2.300605 1 2
691
- 683 76.4275 -1.655850 -2.764404 1 1
692
- 684 76.4620 -4.634916 -8.031740 1 1
693
- 685 76.4964 -3.994935 -1.616292 2 2
694
- 686 76.5306 -5.492788 -5.196081 2 2
695
- 687 76.5648 -2.672094 -1.858173 2 2
696
- 688 76.5988 -6.933484 -7.737554 1 1
697
- 689 76.6328 -2.828150 -2.448026 2 2
698
- 690 76.5217 -2.452675 -2.537375 1 2
699
- 691 76.4110 -3.598053 -3.177943 2 1
700
- 692 76.4451 -1.985059 -2.996325 1 1
701
- 693 76.4791 -4.641545 -4.988807 1 1
702
- 694 76.5130 -3.398436 -3.156051 2 2
703
- 695 76.5468 -4.852722 -5.714799 1 1
704
- 696 76.4368 -4.757531 -4.635437 2 1
705
- 697 76.4706 -2.740584 -2.785802 1 1
706
- 698 76.5043 -4.339527 -3.999101 2 2
707
- 699 76.5379 -3.757401 -5.703282 1 1
708
- 700 76.5714 -3.398460 -2.569705 2 2
709
- 701 76.4622 -1.957600 -1.574631 2 1
710
- 702 76.3533 -4.359742 -4.856097 1 2
711
- 703 76.2447 -4.540786 -3.357212 2 1
712
- 704 76.1364 -2.147459 -2.098414 2 1
713
- 705 76.1702 -2.780626 -2.302537 2 2
714
- 706 76.2040 -2.903928 -2.688923 2 2
715
- 707 76.0962 -4.957472 -4.824405 2 1
716
- 708 76.1299 -3.888412 -3.424496 2 2
717
- 709 76.1636 -3.682078 -2.815866 2 2
718
- 710 76.1972 -2.684817 -3.901605 1 1
719
- 711 76.2307 -2.797076 -2.214971 2 2
720
- 712 76.1236 -3.081785 -3.672879 1 2
721
- 713 76.0168 -3.571403 -3.732425 1 2
722
- 714 76.0504 -3.171300 -2.817441 2 2
723
- 715 76.0839 -5.771348 -4.259699 2 2
724
- 716 76.1173 -4.695037 -3.664240 2 2
725
- 717 76.1506 -2.282890 -0.924188 2 2
726
- 718 76.1838 -3.839834 -4.214931 1 1
727
- 719 76.0779 -5.091830 -4.872005 2 1
728
- 720 75.9722 -0.845932 -4.722952 1 2
729
- 721 75.8669 -4.993071 -5.200837 1 2
730
- 722 75.7618 -2.446730 -2.446667 2 1
731
- 723 75.7953 -2.698735 -2.912032 1 1
732
- 724 75.6906 -3.304324 -3.598771 1 2
733
- 725 75.5862 -3.383218 -3.568972 1 2
734
- 726 75.4821 -4.292620 -4.308592 1 2
735
- 727 75.5158 -3.237249 -2.715107 2 2
736
- 728 75.4121 -4.439497 -4.405553 2 1
737
- 729 75.4458 -2.142155 -2.849986 1 1
738
- 730 75.3425 -3.420123 -3.236832 2 1
739
- 731 75.3762 -3.242151 -4.220038 1 1
740
- 732 75.4098 -3.187196 -2.355076 2 2
741
- 733 75.3070 -3.771988 -3.922719 1 2
742
- 734 75.3406 -0.123064 -7.488885 1 1
743
- 735 75.2381 -2.784319 -2.701951 2 1
744
- 736 75.2717 -3.047448 -3.515901 1 1
745
- 737 75.1696 -4.287724 -3.843623 2 1
746
- 738 75.2033 -4.517854 -4.432768 2 2
747
- 739 75.2368 -0.343941 -9.102585 1 1
748
- 740 75.2703 -4.598773 -3.618499 2 2
749
- 741 75.1687 -4.154870 -3.960557 2 1
750
- 742 75.2022 -2.380736 -1.746523 2 2
751
- 743 75.2355 -3.715529 -3.575387 2 2
752
- 744 75.2688 -5.188425 -4.136256 2 2
753
- 745 75.3020 -4.164076 -4.948450 1 1
754
- 746 75.3351 -4.750057 -5.216600 1 1
755
- 747 75.2343 -5.430820 -4.808452 2 1
756
- 748 75.2674 -3.486638 -3.425215 2 2
757
- 749 75.3004 -3.318253 -3.730561 1 1
758
- 750 75.3333 -1.297676 -9.253905 1 1
759
 
760
- Final Winogrande score(750 tasks): 75.3333 +/- 1.5751
 
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-IQ3_M.gguf (version GGUF V3 (latest))
4
 
5
+ Final Winogrande score(750 tasks): 70.9333 +/- 1.6591
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
 
7
+ llama_perf_context_print: load time = 284.52 ms
8
+ llama_perf_context_print: prompt eval time = 89372.55 ms / 22269 tokens ( 4.01 ms per token, 249.17 tokens per second)
9
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
10
+ llama_perf_context_print: total time = 90553.60 ms / 22270 tokens
11
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-iq3_s.arc CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 869 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 869 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 100.00000000
13
- 3 100.00000000
14
- 4 75.00000000
15
- 5 80.00000000
16
- 6 83.33333333
17
- 7 85.71428571
18
- 8 87.50000000
19
- 9 77.77777778
20
- 10 80.00000000
21
- 11 72.72727273
22
- 12 75.00000000
23
- 13 69.23076923
24
- 14 71.42857143
25
- 15 73.33333333
26
- 16 75.00000000
27
- 17 76.47058824
28
- 18 72.22222222
29
- 19 73.68421053
30
- 20 70.00000000
31
- 21 71.42857143
32
- 22 72.72727273
33
- 23 69.56521739
34
- 24 70.83333333
35
- 25 68.00000000
36
- 26 69.23076923
37
- 27 70.37037037
38
- 28 67.85714286
39
- 29 68.96551724
40
- 30 66.66666667
41
- 31 64.51612903
42
- 32 65.62500000
43
- 33 63.63636364
44
- 34 61.76470588
45
- 35 62.85714286
46
- 36 61.11111111
47
- 37 62.16216216
48
- 38 60.52631579
49
- 39 61.53846154
50
- 40 60.00000000
51
- 41 60.97560976
52
- 42 59.52380952
53
- 43 58.13953488
54
- 44 56.81818182
55
- 45 57.77777778
56
- 46 58.69565217
57
- 47 59.57446809
58
- 48 60.41666667
59
- 49 61.22448980
60
- 50 60.00000000
61
- 51 58.82352941
62
- 52 59.61538462
63
- 53 58.49056604
64
- 54 57.40740741
65
- 55 56.36363636
66
- 56 57.14285714
67
- 57 57.89473684
68
- 58 58.62068966
69
- 59 59.32203390
70
- 60 60.00000000
71
- 61 60.65573770
72
- 62 59.67741935
73
- 63 60.31746032
74
- 64 60.93750000
75
- 65 60.00000000
76
- 66 60.60606061
77
- 67 61.19402985
78
- 68 61.76470588
79
- 69 62.31884058
80
- 70 61.42857143
81
- 71 60.56338028
82
- 72 59.72222222
83
- 73 60.27397260
84
- 74 60.81081081
85
- 75 61.33333333
86
- 76 61.84210526
87
- 77 62.33766234
88
- 78 62.82051282
89
- 79 62.02531646
90
- 80 62.50000000
91
- 81 61.72839506
92
- 82 62.19512195
93
- 83 61.44578313
94
- 84 61.90476190
95
- 85 62.35294118
96
- 86 62.79069767
97
- 87 63.21839080
98
- 88 63.63636364
99
- 89 64.04494382
100
- 90 64.44444444
101
- 91 64.83516484
102
- 92 65.21739130
103
- 93 64.51612903
104
- 94 63.82978723
105
- 95 64.21052632
106
- 96 64.58333333
107
- 97 63.91752577
108
- 98 63.26530612
109
- 99 63.63636364
110
- 100 64.00000000
111
- 101 63.36633663
112
- 102 63.72549020
113
- 103 63.10679612
114
- 104 63.46153846
115
- 105 63.80952381
116
- 106 63.20754717
117
- 107 63.55140187
118
- 108 62.96296296
119
- 109 62.38532110
120
- 110 62.72727273
121
- 111 63.06306306
122
- 112 63.39285714
123
- 113 62.83185841
124
- 114 63.15789474
125
- 115 63.47826087
126
- 116 63.79310345
127
- 117 63.24786325
128
- 118 63.55932203
129
- 119 63.02521008
130
- 120 63.33333333
131
- 121 62.80991736
132
- 122 63.11475410
133
- 123 63.41463415
134
- 124 62.90322581
135
- 125 63.20000000
136
- 126 63.49206349
137
- 127 63.77952756
138
- 128 64.06250000
139
- 129 63.56589147
140
- 130 63.07692308
141
- 131 63.35877863
142
- 132 63.63636364
143
- 133 63.90977444
144
- 134 63.43283582
145
- 135 62.96296296
146
- 136 62.50000000
147
- 137 62.77372263
148
- 138 63.04347826
149
- 139 63.30935252
150
- 140 63.57142857
151
- 141 63.12056738
152
- 142 63.38028169
153
- 143 63.63636364
154
- 144 63.19444444
155
- 145 63.44827586
156
- 146 63.01369863
157
- 147 63.26530612
158
- 148 63.51351351
159
- 149 63.75838926
160
- 150 64.00000000
161
- 151 63.57615894
162
- 152 63.81578947
163
- 153 64.05228758
164
- 154 63.63636364
165
- 155 63.87096774
166
- 156 64.10256410
167
- 157 64.33121019
168
- 158 63.92405063
169
- 159 64.15094340
170
- 160 63.75000000
171
- 161 63.97515528
172
- 162 64.19753086
173
- 163 64.41717791
174
- 164 64.63414634
175
- 165 64.84848485
176
- 166 65.06024096
177
- 167 64.67065868
178
- 168 64.88095238
179
- 169 64.49704142
180
- 170 64.70588235
181
- 171 64.91228070
182
- 172 64.53488372
183
- 173 64.73988439
184
- 174 64.94252874
185
- 175 64.57142857
186
- 176 64.77272727
187
- 177 64.97175141
188
- 178 65.16853933
189
- 179 65.36312849
190
- 180 65.00000000
191
- 181 64.64088398
192
- 182 64.83516484
193
- 183 64.48087432
194
- 184 64.13043478
195
- 185 64.32432432
196
- 186 64.51612903
197
- 187 64.17112299
198
- 188 63.82978723
199
- 189 63.49206349
200
- 190 63.15789474
201
- 191 63.35078534
202
- 192 63.54166667
203
- 193 63.73056995
204
- 194 63.91752577
205
- 195 64.10256410
206
- 196 64.28571429
207
- 197 64.46700508
208
- 198 64.64646465
209
- 199 64.32160804
210
- 200 64.50000000
211
- 201 64.67661692
212
- 202 64.35643564
213
- 203 64.53201970
214
- 204 64.70588235
215
- 205 64.39024390
216
- 206 64.07766990
217
- 207 63.76811594
218
- 208 63.94230769
219
- 209 64.11483254
220
- 210 64.28571429
221
- 211 64.45497630
222
- 212 64.15094340
223
- 213 64.31924883
224
- 214 64.48598131
225
- 215 64.65116279
226
- 216 64.35185185
227
- 217 64.51612903
228
- 218 64.22018349
229
- 219 64.38356164
230
- 220 64.09090909
231
- 221 64.25339367
232
- 222 63.96396396
233
- 223 63.67713004
234
- 224 63.83928571
235
- 225 63.55555556
236
- 226 63.27433628
237
- 227 63.43612335
238
- 228 63.59649123
239
- 229 63.75545852
240
- 230 63.91304348
241
- 231 63.63636364
242
- 232 63.36206897
243
- 233 63.51931330
244
- 234 63.24786325
245
- 235 62.97872340
246
- 236 62.71186441
247
- 237 62.44725738
248
- 238 62.60504202
249
- 239 62.76150628
250
- 240 62.91666667
251
- 241 63.07053942
252
- 242 63.22314050
253
- 243 62.96296296
254
- 244 62.70491803
255
- 245 62.85714286
256
- 246 63.00813008
257
- 247 63.15789474
258
- 248 63.30645161
259
- 249 63.05220884
260
- 250 63.20000000
261
- 251 63.34661355
262
- 252 63.49206349
263
- 253 63.63636364
264
- 254 63.38582677
265
- 255 63.52941176
266
- 256 63.67187500
267
- 257 63.42412451
268
- 258 63.56589147
269
- 259 63.32046332
270
- 260 63.46153846
271
- 261 63.60153257
272
- 262 63.74045802
273
- 263 63.87832700
274
- 264 64.01515152
275
- 265 63.77358491
276
- 266 63.53383459
277
- 267 63.67041199
278
- 268 63.80597015
279
- 269 63.94052045
280
- 270 64.07407407
281
- 271 64.20664207
282
- 272 63.97058824
283
- 273 63.73626374
284
- 274 63.50364964
285
- 275 63.27272727
286
- 276 63.40579710
287
- 277 63.53790614
288
- 278 63.30935252
289
- 279 63.44086022
290
- 280 63.57142857
291
- 281 63.70106762
292
- 282 63.82978723
293
- 283 63.60424028
294
- 284 63.73239437
295
- 285 63.85964912
296
- 286 63.98601399
297
- 287 63.76306620
298
- 288 63.88888889
299
- 289 63.66782007
300
- 290 63.79310345
301
- 291 63.91752577
302
- 292 64.04109589
303
- 293 64.16382253
304
- 294 63.94557823
305
- 295 64.06779661
306
- 296 64.18918919
307
- 297 64.30976431
308
- 298 64.42953020
309
- 299 64.54849498
310
- 300 64.66666667
311
- 301 64.45182724
312
- 302 64.56953642
313
- 303 64.68646865
314
- 304 64.47368421
315
- 305 64.59016393
316
- 306 64.37908497
317
- 307 64.49511401
318
- 308 64.61038961
319
- 309 64.72491909
320
- 310 64.83870968
321
- 311 64.95176849
322
- 312 65.06410256
323
- 313 65.17571885
324
- 314 64.96815287
325
- 315 64.76190476
326
- 316 64.87341772
327
- 317 64.98422713
328
- 318 64.77987421
329
- 319 64.57680251
330
- 320 64.37500000
331
- 321 64.48598131
332
- 322 64.59627329
333
- 323 64.70588235
334
- 324 64.50617284
335
- 325 64.61538462
336
- 326 64.72392638
337
- 327 64.52599388
338
- 328 64.63414634
339
- 329 64.74164134
340
- 330 64.54545455
341
- 331 64.65256798
342
- 332 64.75903614
343
- 333 64.86486486
344
- 334 64.97005988
345
- 335 64.77611940
346
- 336 64.88095238
347
- 337 64.98516320
348
- 338 64.79289941
349
- 339 64.89675516
350
- 340 65.00000000
351
- 341 65.10263930
352
- 342 64.91228070
353
- 343 65.01457726
354
- 344 65.11627907
355
- 345 64.92753623
356
- 346 65.02890173
357
- 347 64.84149856
358
- 348 64.65517241
359
- 349 64.75644699
360
- 350 64.57142857
361
- 351 64.67236467
362
- 352 64.77272727
363
- 353 64.87252125
364
- 354 64.68926554
365
- 355 64.50704225
366
- 356 64.60674157
367
- 357 64.70588235
368
- 358 64.80446927
369
- 359 64.62395543
370
- 360 64.72222222
371
- 361 64.81994460
372
- 362 64.91712707
373
- 363 64.73829201
374
- 364 64.56043956
375
- 365 64.65753425
376
- 366 64.48087432
377
- 367 64.30517711
378
- 368 64.40217391
379
- 369 64.49864499
380
- 370 64.59459459
381
- 371 64.69002695
382
- 372 64.51612903
383
- 373 64.61126005
384
- 374 64.70588235
385
- 375 64.53333333
386
- 376 64.36170213
387
- 377 64.19098143
388
- 378 64.28571429
389
- 379 64.11609499
390
- 380 64.21052632
391
- 381 64.04199475
392
- 382 63.87434555
393
- 383 63.70757180
394
- 384 63.80208333
395
- 385 63.89610390
396
- 386 63.98963731
397
- 387 63.82428941
398
- 388 63.65979381
399
- 389 63.75321337
400
- 390 63.58974359
401
- 391 63.68286445
402
- 392 63.77551020
403
- 393 63.86768448
404
- 394 63.95939086
405
- 395 64.05063291
406
- 396 63.88888889
407
- 397 63.97984887
408
- 398 63.81909548
409
- 399 63.90977444
410
- 400 64.00000000
411
- 401 63.84039900
412
- 402 63.68159204
413
- 403 63.77171216
414
- 404 63.86138614
415
- 405 63.95061728
416
- 406 64.03940887
417
- 407 64.12776413
418
- 408 64.21568627
419
- 409 64.05867971
420
- 410 64.14634146
421
- 411 64.23357664
422
- 412 64.32038835
423
- 413 64.16464891
424
- 414 64.25120773
425
- 415 64.09638554
426
- 416 64.18269231
427
- 417 64.02877698
428
- 418 64.11483254
429
- 419 63.96181384
430
- 420 64.04761905
431
- 421 63.89548694
432
- 422 63.98104265
433
- 423 64.06619385
434
- 424 63.91509434
435
- 425 64.00000000
436
- 426 63.84976526
437
- 427 63.93442623
438
- 428 63.78504673
439
- 429 63.86946387
440
- 430 63.72093023
441
- 431 63.80510441
442
- 432 63.88888889
443
- 433 63.74133949
444
- 434 63.82488479
445
- 435 63.90804598
446
- 436 63.99082569
447
- 437 64.07322654
448
- 438 63.92694064
449
- 439 64.00911162
450
- 440 63.86363636
451
- 441 63.71882086
452
- 442 63.57466063
453
- 443 63.43115124
454
- 444 63.28828829
455
- 445 63.37078652
456
- 446 63.45291480
457
- 447 63.53467562
458
- 448 63.61607143
459
- 449 63.69710468
460
- 450 63.55555556
461
- 451 63.41463415
462
- 452 63.49557522
463
- 453 63.35540839
464
- 454 63.43612335
465
- 455 63.51648352
466
- 456 63.59649123
467
- 457 63.67614880
468
- 458 63.75545852
469
- 459 63.83442266
470
- 460 63.69565217
471
- 461 63.77440347
472
- 462 63.85281385
473
- 463 63.71490281
474
- 464 63.57758621
475
- 465 63.65591398
476
- 466 63.73390558
477
- 467 63.81156317
478
- 468 63.88888889
479
- 469 63.75266525
480
- 470 63.82978723
481
- 471 63.90658174
482
- 472 63.77118644
483
- 473 63.63636364
484
- 474 63.71308017
485
- 475 63.78947368
486
- 476 63.86554622
487
- 477 63.94129979
488
- 478 64.01673640
489
- 479 64.09185804
490
- 480 64.16666667
491
- 481 64.03326403
492
- 482 64.10788382
493
- 483 64.18219462
494
- 484 64.25619835
495
- 485 64.32989691
496
- 486 64.40329218
497
- 487 64.27104723
498
- 488 64.34426230
499
- 489 64.21267894
500
- 490 64.08163265
501
- 491 63.95112016
502
- 492 64.02439024
503
- 493 63.89452333
504
- 494 63.96761134
505
- 495 63.83838384
506
- 496 63.91129032
507
- 497 63.78269618
508
- 498 63.85542169
509
- 499 63.92785571
510
- 500 64.00000000
511
- 501 63.87225549
512
- 502 63.94422311
513
- 503 64.01590457
514
- 504 64.08730159
515
- 505 64.15841584
516
- 506 64.03162055
517
- 507 63.90532544
518
- 508 63.97637795
519
- 509 63.85068762
520
- 510 63.72549020
521
- 511 63.60078278
522
- 512 63.67187500
523
- 513 63.54775828
524
- 514 63.61867704
525
- 515 63.68932039
526
- 516 63.75968992
527
- 517 63.82978723
528
- 518 63.89961390
529
- 519 63.96917148
530
- 520 63.84615385
531
- 521 63.91554702
532
- 522 63.98467433
533
- 523 63.86233270
534
- 524 63.74045802
535
- 525 63.61904762
536
- 526 63.49809886
537
- 527 63.37760911
538
- 528 63.44696970
539
- 529 63.32703214
540
- 530 63.39622642
541
- 531 63.27683616
542
- 532 63.34586466
543
- 533 63.41463415
544
- 534 63.29588015
545
- 535 63.17757009
546
- 536 63.24626866
547
- 537 63.31471136
548
- 538 63.38289963
549
- 539 63.45083488
550
- 540 63.33333333
551
- 541 63.40110906
552
- 542 63.46863469
553
- 543 63.35174954
554
- 544 63.41911765
555
- 545 63.48623853
556
- 546 63.36996337
557
- 547 63.25411335
558
- 548 63.32116788
559
- 549 63.38797814
560
- 550 63.45454545
561
- 551 63.33938294
562
- 552 63.22463768
563
- 553 63.29113924
564
- 554 63.35740072
565
- 555 63.42342342
566
- 556 63.30935252
567
- 557 63.37522442
568
- 558 63.26164875
569
- 559 63.32737030
570
- 560 63.39285714
571
- 561 63.45811052
572
- 562 63.52313167
573
- 563 63.58792185
574
- 564 63.65248227
575
- 565 63.53982301
576
- 566 63.60424028
577
- 567 63.49206349
578
- 568 63.55633803
579
- 569 63.44463972
580
- 570 63.33333333
581
- 571 63.39754816
582
- 572 63.28671329
583
- 573 63.35078534
584
- 574 63.41463415
585
- 575 63.30434783
586
- 576 63.36805556
587
- 577 63.43154246
588
- 578 63.49480969
589
- 579 63.55785838
590
- 580 63.44827586
591
- 581 63.33907057
592
- 582 63.23024055
593
- 583 63.29331046
594
- 584 63.35616438
595
- 585 63.41880342
596
- 586 63.48122867
597
- 587 63.54344123
598
- 588 63.60544218
599
- 589 63.66723260
600
- 590 63.72881356
601
- 591 63.79018613
602
- 592 63.68243243
603
- 593 63.57504216
604
- 594 63.63636364
605
- 595 63.52941176
606
- 596 63.42281879
607
- 597 63.31658291
608
- 598 63.21070234
609
- 599 63.27212020
610
- 600 63.16666667
611
- 601 63.22795341
612
- 602 63.12292359
613
- 603 63.01824212
614
- 604 62.91390728
615
- 605 62.97520661
616
- 606 62.87128713
617
- 607 62.93245470
618
- 608 62.99342105
619
- 609 62.88998358
620
- 610 62.95081967
621
- 611 62.84779051
622
- 612 62.74509804
623
- 613 62.64274062
624
- 614 62.70358306
625
- 615 62.60162602
626
- 616 62.66233766
627
- 617 62.72285251
628
- 618 62.78317152
629
- 619 62.68174475
630
- 620 62.74193548
631
- 621 62.80193237
632
- 622 62.70096463
633
- 623 62.60032103
634
- 624 62.50000000
635
- 625 62.40000000
636
- 626 62.30031949
637
- 627 62.36044657
638
- 628 62.42038217
639
- 629 62.32114467
640
- 630 62.38095238
641
- 631 62.44057052
642
- 632 62.34177215
643
- 633 62.24328594
644
- 634 62.30283912
645
- 635 62.36220472
646
- 636 62.42138365
647
- 637 62.32339089
648
- 638 62.38244514
649
- 639 62.44131455
650
- 640 62.34375000
651
- 641 62.40249610
652
- 642 62.30529595
653
- 643 62.20839813
654
- 644 62.26708075
655
- 645 62.17054264
656
- 646 62.22910217
657
- 647 62.28748068
658
- 648 62.19135802
659
- 649 62.09553159
660
- 650 62.15384615
661
- 651 62.21198157
662
- 652 62.11656442
663
- 653 62.02143951
664
- 654 62.07951070
665
- 655 61.98473282
666
- 656 61.89024390
667
- 657 61.79604262
668
- 658 61.70212766
669
- 659 61.60849772
670
- 660 61.66666667
671
- 661 61.57337368
672
- 662 61.63141994
673
- 663 61.68929110
674
- 664 61.74698795
675
- 665 61.80451128
676
- 666 61.71171171
677
- 667 61.76911544
678
- 668 61.67664671
679
- 669 61.58445441
680
- 670 61.64179104
681
- 671 61.54992548
682
- 672 61.60714286
683
- 673 61.51560178
684
- 674 61.57270030
685
- 675 61.48148148
686
- 676 61.39053254
687
- 677 61.44756278
688
- 678 61.50442478
689
- 679 61.56111929
690
- 680 61.61764706
691
- 681 61.52716593
692
- 682 61.43695015
693
- 683 61.34699854
694
- 684 61.40350877
695
- 685 61.45985401
696
- 686 61.37026239
697
- 687 61.28093159
698
- 688 61.33720930
699
- 689 61.24818578
700
- 690 61.30434783
701
- 691 61.36034732
702
- 692 61.41618497
703
- 693 61.47186147
704
- 694 61.52737752
705
- 695 61.43884892
706
- 696 61.35057471
707
- 697 61.26255380
708
- 698 61.17478510
709
- 699 61.23032904
710
- 700 61.28571429
711
- 701 61.34094151
712
- 702 61.25356125
713
- 703 61.30867710
714
- 704 61.36363636
715
- 705 61.41843972
716
- 706 61.47308782
717
- 707 61.52758133
718
- 708 61.58192090
719
- 709 61.63610719
720
- 710 61.69014085
721
- 711 61.74402250
722
- 712 61.79775281
723
- 713 61.85133240
724
- 714 61.76470588
725
- 715 61.67832168
726
- 716 61.73184358
727
- 717 61.64574616
728
- 718 61.55988858
729
- 719 61.47426982
730
- 720 61.52777778
731
- 721 61.58113731
732
- 722 61.63434903
733
- 723 61.68741355
734
- 724 61.60220994
735
- 725 61.65517241
736
- 726 61.57024793
737
- 727 61.48555708
738
- 728 61.53846154
739
- 729 61.59122085
740
- 730 61.64383562
741
- 731 61.69630643
742
- 732 61.74863388
743
- 733 61.80081855
744
- 734 61.85286104
745
- 735 61.90476190
746
- 736 61.95652174
747
- 737 61.87245590
748
- 738 61.92411924
749
- 739 61.84032476
750
- 740 61.75675676
751
- 741 61.80836707
752
- 742 61.72506739
753
- 743 61.77658143
754
- 744 61.69354839
755
- 745 61.74496644
756
- 746 61.79624665
757
- 747 61.71352075
758
- 748 61.63101604
759
- 749 61.54873164
760
- 750 61.60000000
761
 
762
- Final result: 61.6000 +/- 1.7771
763
- Random chance: 25.0083 +/- 1.5824
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-IQ3_S.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 57.3529 +/- 1.8095
6
+ Random chance: 25.0335 +/- 1.5850
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 1575.89 ms
10
+ llama_perf_context_print: prompt eval time = 159215.12 ms / 36653 tokens ( 4.34 ms per token, 230.21 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 161465.24 ms / 36654 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-iq3_s.hsw CHANGED
@@ -1,760 +1,12 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- hellaswag_score : loaded 10042 tasks from prompt.
6
- ================================= is_spm = 0
7
- hellaswag_score : selecting 750 randomized tasks.
8
- hellaswag_score : calculating hellaswag score over selected tasks.
9
 
10
- task acc_norm
11
- 1 0.00000000
12
- 2 50.00000000
13
- 3 66.66666667
14
- 4 75.00000000
15
- 5 80.00000000
16
- 6 83.33333333
17
- 7 71.42857143
18
- 8 75.00000000
19
- 9 77.77777778
20
- 10 80.00000000
21
- 11 81.81818182
22
- 12 75.00000000
23
- 13 76.92307692
24
- 14 78.57142857
25
- 15 80.00000000
26
- 16 81.25000000
27
- 17 76.47058824
28
- 18 77.77777778
29
- 19 73.68421053
30
- 20 75.00000000
31
- 21 76.19047619
32
- 22 77.27272727
33
- 23 78.26086957
34
- 24 75.00000000
35
- 25 72.00000000
36
- 26 69.23076923
37
- 27 66.66666667
38
- 28 67.85714286
39
- 29 65.51724138
40
- 30 63.33333333
41
- 31 64.51612903
42
- 32 62.50000000
43
- 33 63.63636364
44
- 34 64.70588235
45
- 35 65.71428571
46
- 36 66.66666667
47
- 37 64.86486486
48
- 38 65.78947368
49
- 39 66.66666667
50
- 40 65.00000000
51
- 41 65.85365854
52
- 42 64.28571429
53
- 43 65.11627907
54
- 44 65.90909091
55
- 45 64.44444444
56
- 46 65.21739130
57
- 47 65.95744681
58
- 48 66.66666667
59
- 49 67.34693878
60
- 50 66.00000000
61
- 51 66.66666667
62
- 52 65.38461538
63
- 53 64.15094340
64
- 54 62.96296296
65
- 55 61.81818182
66
- 56 60.71428571
67
- 57 61.40350877
68
- 58 62.06896552
69
- 59 62.71186441
70
- 60 63.33333333
71
- 61 63.93442623
72
- 62 64.51612903
73
- 63 65.07936508
74
- 64 65.62500000
75
- 65 66.15384615
76
- 66 65.15151515
77
- 67 65.67164179
78
- 68 66.17647059
79
- 69 66.66666667
80
- 70 67.14285714
81
- 71 66.19718310
82
- 72 66.66666667
83
- 73 67.12328767
84
- 74 67.56756757
85
- 75 66.66666667
86
- 76 67.10526316
87
- 77 67.53246753
88
- 78 67.94871795
89
- 79 68.35443038
90
- 80 68.75000000
91
- 81 69.13580247
92
- 82 69.51219512
93
- 83 69.87951807
94
- 84 70.23809524
95
- 85 70.58823529
96
- 86 70.93023256
97
- 87 71.26436782
98
- 88 71.59090909
99
- 89 70.78651685
100
- 90 71.11111111
101
- 91 71.42857143
102
- 92 70.65217391
103
- 93 70.96774194
104
- 94 71.27659574
105
- 95 71.57894737
106
- 96 71.87500000
107
- 97 72.16494845
108
- 98 72.44897959
109
- 99 72.72727273
110
- 100 72.00000000
111
- 101 72.27722772
112
- 102 71.56862745
113
- 103 70.87378641
114
- 104 71.15384615
115
- 105 71.42857143
116
- 106 71.69811321
117
- 107 71.96261682
118
- 108 72.22222222
119
- 109 71.55963303
120
- 110 71.81818182
121
- 111 72.07207207
122
- 112 72.32142857
123
- 113 72.56637168
124
- 114 72.80701754
125
- 115 73.04347826
126
- 116 72.41379310
127
- 117 71.79487179
128
- 118 72.03389831
129
- 119 72.26890756
130
- 120 71.66666667
131
- 121 71.90082645
132
- 122 72.13114754
133
- 123 72.35772358
134
- 124 72.58064516
135
- 125 72.80000000
136
- 126 73.01587302
137
- 127 73.22834646
138
- 128 72.65625000
139
- 129 72.86821705
140
- 130 72.30769231
141
- 131 72.51908397
142
- 132 72.72727273
143
- 133 72.93233083
144
- 134 73.13432836
145
- 135 73.33333333
146
- 136 73.52941176
147
- 137 73.72262774
148
- 138 73.91304348
149
- 139 73.38129496
150
- 140 73.57142857
151
- 141 73.75886525
152
- 142 73.94366197
153
- 143 74.12587413
154
- 144 74.30555556
155
- 145 74.48275862
156
- 146 74.65753425
157
- 147 74.82993197
158
- 148 75.00000000
159
- 149 75.16778523
160
- 150 75.33333333
161
- 151 74.83443709
162
- 152 74.34210526
163
- 153 74.50980392
164
- 154 74.67532468
165
- 155 74.83870968
166
- 156 75.00000000
167
- 157 75.15923567
168
- 158 75.31645570
169
- 159 74.84276730
170
- 160 75.00000000
171
- 161 75.15527950
172
- 162 75.30864198
173
- 163 75.46012270
174
- 164 75.60975610
175
- 165 75.75757576
176
- 166 75.90361446
177
- 167 76.04790419
178
- 168 76.19047619
179
- 169 75.73964497
180
- 170 75.88235294
181
- 171 76.02339181
182
- 172 76.16279070
183
- 173 76.30057803
184
- 174 76.43678161
185
- 175 76.00000000
186
- 176 76.13636364
187
- 177 76.27118644
188
- 178 76.40449438
189
- 179 75.97765363
190
- 180 76.11111111
191
- 181 75.69060773
192
- 182 75.82417582
193
- 183 75.95628415
194
- 184 75.54347826
195
- 185 75.67567568
196
- 186 75.26881720
197
- 187 74.86631016
198
- 188 75.00000000
199
- 189 74.60317460
200
- 190 74.73684211
201
- 191 74.86910995
202
- 192 75.00000000
203
- 193 74.61139896
204
- 194 74.74226804
205
- 195 74.35897436
206
- 196 74.48979592
207
- 197 74.11167513
208
- 198 73.73737374
209
- 199 73.86934673
210
- 200 74.00000000
211
- 201 74.12935323
212
- 202 74.25742574
213
- 203 73.89162562
214
- 204 74.01960784
215
- 205 74.14634146
216
- 206 74.27184466
217
- 207 74.39613527
218
- 208 74.03846154
219
- 209 74.16267943
220
- 210 74.28571429
221
- 211 74.40758294
222
- 212 74.52830189
223
- 213 74.64788732
224
- 214 74.76635514
225
- 215 74.41860465
226
- 216 74.53703704
227
- 217 74.19354839
228
- 218 74.31192661
229
- 219 74.42922374
230
- 220 74.54545455
231
- 221 74.20814480
232
- 222 74.32432432
233
- 223 74.43946188
234
- 224 74.55357143
235
- 225 74.66666667
236
- 226 74.77876106
237
- 227 74.44933921
238
- 228 74.56140351
239
- 229 74.67248908
240
- 230 74.78260870
241
- 231 74.45887446
242
- 232 74.13793103
243
- 233 74.24892704
244
- 234 74.35897436
245
- 235 74.04255319
246
- 236 73.72881356
247
- 237 73.83966245
248
- 238 73.94957983
249
- 239 74.05857741
250
- 240 74.16666667
251
- 241 74.27385892
252
- 242 74.38016529
253
- 243 74.48559671
254
- 244 74.18032787
255
- 245 73.87755102
256
- 246 73.57723577
257
- 247 73.68421053
258
- 248 73.79032258
259
- 249 73.89558233
260
- 250 74.00000000
261
- 251 74.10358566
262
- 252 74.20634921
263
- 253 73.91304348
264
- 254 74.01574803
265
- 255 74.11764706
266
- 256 74.21875000
267
- 257 73.92996109
268
- 258 74.03100775
269
- 259 74.13127413
270
- 260 74.23076923
271
- 261 74.32950192
272
- 262 74.04580153
273
- 263 74.14448669
274
- 264 74.24242424
275
- 265 74.33962264
276
- 266 74.43609023
277
- 267 74.53183521
278
- 268 74.62686567
279
- 269 74.72118959
280
- 270 74.81481481
281
- 271 74.53874539
282
- 272 74.63235294
283
- 273 74.35897436
284
- 274 74.45255474
285
- 275 74.18181818
286
- 276 74.27536232
287
- 277 74.36823105
288
- 278 74.46043165
289
- 279 74.55197133
290
- 280 74.64285714
291
- 281 74.73309609
292
- 282 74.82269504
293
- 283 74.91166078
294
- 284 75.00000000
295
- 285 74.73684211
296
- 286 74.82517483
297
- 287 74.91289199
298
- 288 75.00000000
299
- 289 75.08650519
300
- 290 75.17241379
301
- 291 75.25773196
302
- 292 75.00000000
303
- 293 75.08532423
304
- 294 75.17006803
305
- 295 75.25423729
306
- 296 75.33783784
307
- 297 75.42087542
308
- 298 75.50335570
309
- 299 75.25083612
310
- 300 75.00000000
311
- 301 75.08305648
312
- 302 75.16556291
313
- 303 74.91749175
314
- 304 74.67105263
315
- 305 74.75409836
316
- 306 74.83660131
317
- 307 74.59283388
318
- 308 74.67532468
319
- 309 74.75728155
320
- 310 74.83870968
321
- 311 74.91961415
322
- 312 75.00000000
323
- 313 75.07987220
324
- 314 75.15923567
325
- 315 75.23809524
326
- 316 75.31645570
327
- 317 75.07886435
328
- 318 74.84276730
329
- 319 74.60815047
330
- 320 74.68750000
331
- 321 74.45482866
332
- 322 74.22360248
333
- 323 74.30340557
334
- 324 74.38271605
335
- 325 74.46153846
336
- 326 74.53987730
337
- 327 74.31192661
338
- 328 74.39024390
339
- 329 74.46808511
340
- 330 74.54545455
341
- 331 74.62235650
342
- 332 74.69879518
343
- 333 74.77477477
344
- 334 74.55089820
345
- 335 74.62686567
346
- 336 74.70238095
347
- 337 74.48071217
348
- 338 74.26035503
349
- 339 74.33628319
350
- 340 74.41176471
351
- 341 74.48680352
352
- 342 74.56140351
353
- 343 74.63556851
354
- 344 74.41860465
355
- 345 74.49275362
356
- 346 74.27745665
357
- 347 74.35158501
358
- 348 74.13793103
359
- 349 73.92550143
360
- 350 74.00000000
361
- 351 74.07407407
362
- 352 74.14772727
363
- 353 74.22096317
364
- 354 74.29378531
365
- 355 74.36619718
366
- 356 74.43820225
367
- 357 74.50980392
368
- 358 74.58100559
369
- 359 74.65181058
370
- 360 74.72222222
371
- 361 74.79224377
372
- 362 74.86187845
373
- 363 74.93112948
374
- 364 75.00000000
375
- 365 75.06849315
376
- 366 75.13661202
377
- 367 75.20435967
378
- 368 75.27173913
379
- 369 75.33875339
380
- 370 75.40540541
381
- 371 75.47169811
382
- 372 75.53763441
383
- 373 75.33512064
384
- 374 75.40106952
385
- 375 75.46666667
386
- 376 75.53191489
387
- 377 75.33156499
388
- 378 75.39682540
389
- 379 75.46174142
390
- 380 75.52631579
391
- 381 75.59055118
392
- 382 75.65445026
393
- 383 75.71801567
394
- 384 75.78125000
395
- 385 75.84415584
396
- 386 75.90673575
397
- 387 75.96899225
398
- 388 75.77319588
399
- 389 75.83547558
400
- 390 75.89743590
401
- 391 75.70332481
402
- 392 75.76530612
403
- 393 75.82697201
404
- 394 75.88832487
405
- 395 75.94936709
406
- 396 76.01010101
407
- 397 76.07052897
408
- 398 76.13065327
409
- 399 76.19047619
410
- 400 76.25000000
411
- 401 76.30922693
412
- 402 76.36815920
413
- 403 76.17866005
414
- 404 76.23762376
415
- 405 76.29629630
416
- 406 76.35467980
417
- 407 76.41277641
418
- 408 76.47058824
419
- 409 76.52811736
420
- 410 76.34146341
421
- 411 76.15571776
422
- 412 76.21359223
423
- 413 76.27118644
424
- 414 76.32850242
425
- 415 76.14457831
426
- 416 76.20192308
427
- 417 76.25899281
428
- 418 76.31578947
429
- 419 76.37231504
430
- 420 76.42857143
431
- 421 76.48456057
432
- 422 76.54028436
433
- 423 76.59574468
434
- 424 76.65094340
435
- 425 76.70588235
436
- 426 76.76056338
437
- 427 76.58079625
438
- 428 76.63551402
439
- 429 76.45687646
440
- 430 76.51162791
441
- 431 76.33410673
442
- 432 76.15740741
443
- 433 75.98152425
444
- 434 76.03686636
445
- 435 75.86206897
446
- 436 75.91743119
447
- 437 75.74370709
448
- 438 75.79908676
449
- 439 75.85421412
450
- 440 75.68181818
451
- 441 75.73696145
452
- 442 75.79185520
453
- 443 75.84650113
454
- 444 75.90090090
455
- 445 75.95505618
456
- 446 76.00896861
457
- 447 76.06263982
458
- 448 76.11607143
459
- 449 76.16926503
460
- 450 76.22222222
461
- 451 76.05321508
462
- 452 75.88495575
463
- 453 75.93818985
464
- 454 75.99118943
465
- 455 76.04395604
466
- 456 76.09649123
467
- 457 76.14879650
468
- 458 76.20087336
469
- 459 76.25272331
470
- 460 76.30434783
471
- 461 76.35574837
472
- 462 76.40692641
473
- 463 76.45788337
474
- 464 76.50862069
475
- 465 76.34408602
476
- 466 76.39484979
477
- 467 76.44539615
478
- 468 76.28205128
479
- 469 76.11940299
480
- 470 75.95744681
481
- 471 76.00849257
482
- 472 76.05932203
483
- 473 76.10993658
484
- 474 76.16033755
485
- 475 76.21052632
486
- 476 76.26050420
487
- 477 76.10062893
488
- 478 76.15062762
489
- 479 75.99164927
490
- 480 76.04166667
491
- 481 75.88357588
492
- 482 75.72614108
493
- 483 75.77639752
494
- 484 75.61983471
495
- 485 75.46391753
496
- 486 75.51440329
497
- 487 75.56468172
498
- 488 75.40983607
499
- 489 75.46012270
500
- 490 75.51020408
501
- 491 75.56008147
502
- 492 75.60975610
503
- 493 75.65922921
504
- 494 75.70850202
505
- 495 75.75757576
506
- 496 75.80645161
507
- 497 75.85513078
508
- 498 75.90361446
509
- 499 75.75150301
510
- 500 75.80000000
511
- 501 75.64870259
512
- 502 75.69721116
513
- 503 75.74552684
514
- 504 75.79365079
515
- 505 75.84158416
516
- 506 75.88932806
517
- 507 75.93688363
518
- 508 75.98425197
519
- 509 76.03143418
520
- 510 76.07843137
521
- 511 76.12524462
522
- 512 75.97656250
523
- 513 75.82846004
524
- 514 75.87548638
525
- 515 75.92233010
526
- 516 75.77519380
527
- 517 75.62862669
528
- 518 75.67567568
529
- 519 75.72254335
530
- 520 75.76923077
531
- 521 75.62380038
532
- 522 75.67049808
533
- 523 75.71701721
534
- 524 75.76335878
535
- 525 75.80952381
536
- 526 75.66539924
537
- 527 75.71157495
538
- 528 75.75757576
539
- 529 75.80340265
540
- 530 75.84905660
541
- 531 75.89453861
542
- 532 75.93984962
543
- 533 75.98499062
544
- 534 76.02996255
545
- 535 76.07476636
546
- 536 76.11940299
547
- 537 76.16387337
548
- 538 76.20817844
549
- 539 76.06679035
550
- 540 76.11111111
551
- 541 76.15526802
552
- 542 76.19926199
553
- 543 76.05893186
554
- 544 76.10294118
555
- 545 76.14678899
556
- 546 76.19047619
557
- 547 76.23400366
558
- 548 76.27737226
559
- 549 76.32058288
560
- 550 76.18181818
561
- 551 76.22504537
562
- 552 76.08695652
563
- 553 76.13019892
564
- 554 76.17328520
565
- 555 76.03603604
566
- 556 76.07913669
567
- 557 75.94254937
568
- 558 75.80645161
569
- 559 75.84973166
570
- 560 75.89285714
571
- 561 75.93582888
572
- 562 75.97864769
573
- 563 75.84369449
574
- 564 75.88652482
575
- 565 75.75221239
576
- 566 75.61837456
577
- 567 75.66137566
578
- 568 75.70422535
579
- 569 75.57117750
580
- 570 75.61403509
581
- 571 75.65674256
582
- 572 75.69930070
583
- 573 75.74171030
584
- 574 75.78397213
585
- 575 75.82608696
586
- 576 75.86805556
587
- 577 75.90987868
588
- 578 75.95155709
589
- 579 75.99309154
590
- 580 76.03448276
591
- 581 75.90361446
592
- 582 75.94501718
593
- 583 75.98627787
594
- 584 76.02739726
595
- 585 76.06837607
596
- 586 76.10921502
597
- 587 76.14991482
598
- 588 76.02040816
599
- 589 75.89134126
600
- 590 75.93220339
601
- 591 75.97292724
602
- 592 76.01351351
603
- 593 75.88532884
604
- 594 75.75757576
605
- 595 75.79831933
606
- 596 75.83892617
607
- 597 75.87939698
608
- 598 75.91973244
609
- 599 75.79298831
610
- 600 75.83333333
611
- 601 75.87354409
612
- 602 75.91362126
613
- 603 75.95356551
614
- 604 75.99337748
615
- 605 76.03305785
616
- 606 75.90759076
617
- 607 75.94728171
618
- 608 75.82236842
619
- 609 75.86206897
620
- 610 75.90163934
621
- 611 75.94108020
622
- 612 75.98039216
623
- 613 75.85644372
624
- 614 75.89576547
625
- 615 75.93495935
626
- 616 75.81168831
627
- 617 75.68881686
628
- 618 75.72815534
629
- 619 75.76736672
630
- 620 75.80645161
631
- 621 75.84541063
632
- 622 75.88424437
633
- 623 75.92295345
634
- 624 75.96153846
635
- 625 76.00000000
636
- 626 76.03833866
637
- 627 76.07655502
638
- 628 76.11464968
639
- 629 75.99364070
640
- 630 75.87301587
641
- 631 75.75277338
642
- 632 75.63291139
643
- 633 75.51342812
644
- 634 75.55205047
645
- 635 75.59055118
646
- 636 75.62893082
647
- 637 75.66718995
648
- 638 75.70532915
649
- 639 75.74334898
650
- 640 75.62500000
651
- 641 75.66302652
652
- 642 75.70093458
653
- 643 75.73872473
654
- 644 75.77639752
655
- 645 75.65891473
656
- 646 75.69659443
657
- 647 75.73415765
658
- 648 75.77160494
659
- 649 75.65485362
660
- 650 75.69230769
661
- 651 75.72964670
662
- 652 75.76687117
663
- 653 75.80398162
664
- 654 75.84097859
665
- 655 75.87786260
666
- 656 75.91463415
667
- 657 75.95129376
668
- 658 75.98784195
669
- 659 75.87253414
670
- 660 75.75757576
671
- 661 75.79425113
672
- 662 75.83081571
673
- 663 75.86726998
674
- 664 75.90361446
675
- 665 75.93984962
676
- 666 75.97597598
677
- 667 76.01199400
678
- 668 76.04790419
679
- 669 76.08370703
680
- 670 76.11940299
681
- 671 76.15499255
682
- 672 76.19047619
683
- 673 76.22585438
684
- 674 76.26112760
685
- 675 76.29629630
686
- 676 76.33136095
687
- 677 76.36632201
688
- 678 76.40117994
689
- 679 76.43593520
690
- 680 76.47058824
691
- 681 76.50513950
692
- 682 76.53958944
693
- 683 76.57393851
694
- 684 76.60818713
695
- 685 76.64233577
696
- 686 76.67638484
697
- 687 76.71033479
698
- 688 76.74418605
699
- 689 76.77793904
700
- 690 76.81159420
701
- 691 76.84515195
702
- 692 76.87861272
703
- 693 76.76767677
704
- 694 76.80115274
705
- 695 76.83453237
706
- 696 76.86781609
707
- 697 76.90100430
708
- 698 76.93409742
709
- 699 76.96709585
710
- 700 77.00000000
711
- 701 77.03281027
712
- 702 77.06552707
713
- 703 77.09815078
714
- 704 77.13068182
715
- 705 77.16312057
716
- 706 77.19546742
717
- 707 77.22772277
718
- 708 77.25988701
719
- 709 77.29196051
720
- 710 77.32394366
721
- 711 77.21518987
722
- 712 77.24719101
723
- 713 77.27910238
724
- 714 77.31092437
725
- 715 77.20279720
726
- 716 77.23463687
727
- 717 77.26638773
728
- 718 77.29805014
729
- 719 77.32962448
730
- 720 77.36111111
731
- 721 77.39251040
732
- 722 77.42382271
733
- 723 77.45504841
734
- 724 77.48618785
735
- 725 77.51724138
736
- 726 77.54820937
737
- 727 77.57909216
738
- 728 77.60989011
739
- 729 77.64060357
740
- 730 77.67123288
741
- 731 77.70177839
742
- 732 77.59562842
743
- 733 77.62619372
744
- 734 77.65667575
745
- 735 77.55102041
746
- 736 77.58152174
747
- 737 77.47625509
748
- 738 77.37127371
749
- 739 77.26657645
750
- 740 77.29729730
751
- 741 77.32793522
752
- 742 77.35849057
753
- 743 77.38896366
754
- 744 77.41935484
755
- 745 77.31543624
756
- 746 77.34584450
757
- 747 77.24230254
758
- 748 77.27272727
759
- 749 77.30307076
760
- 750 77.33333333
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-IQ3_S.gguf (version GGUF V3 (latest))
4
 
5
+ 750 77.20000000
 
 
 
 
6
 
7
+
8
+ llama_perf_context_print: load time = 298.56 ms
9
+ llama_perf_context_print: prompt eval time = 430279.23 ms / 124462 tokens ( 3.46 ms per token, 289.26 tokens per second)
10
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
11
+ llama_perf_context_print: total time = 436034.64 ms / 124463 tokens
12
+ ggml_metal_free: deallocating
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scores/Watt-Tool-8B-iq3_s.mmlu CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 1548 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 1548 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 50.00000000
13
- 3 33.33333333
14
- 4 50.00000000
15
- 5 40.00000000
16
- 6 33.33333333
17
- 7 42.85714286
18
- 8 50.00000000
19
- 9 44.44444444
20
- 10 50.00000000
21
- 11 45.45454545
22
- 12 41.66666667
23
- 13 38.46153846
24
- 14 35.71428571
25
- 15 40.00000000
26
- 16 43.75000000
27
- 17 47.05882353
28
- 18 44.44444444
29
- 19 47.36842105
30
- 20 50.00000000
31
- 21 52.38095238
32
- 22 50.00000000
33
- 23 47.82608696
34
- 24 50.00000000
35
- 25 52.00000000
36
- 26 50.00000000
37
- 27 51.85185185
38
- 28 50.00000000
39
- 29 51.72413793
40
- 30 53.33333333
41
- 31 54.83870968
42
- 32 56.25000000
43
- 33 54.54545455
44
- 34 55.88235294
45
- 35 57.14285714
46
- 36 55.55555556
47
- 37 54.05405405
48
- 38 52.63157895
49
- 39 51.28205128
50
- 40 50.00000000
51
- 41 48.78048780
52
- 42 47.61904762
53
- 43 48.83720930
54
- 44 47.72727273
55
- 45 48.88888889
56
- 46 47.82608696
57
- 47 46.80851064
58
- 48 47.91666667
59
- 49 46.93877551
60
- 50 46.00000000
61
- 51 45.09803922
62
- 52 44.23076923
63
- 53 43.39622642
64
- 54 44.44444444
65
- 55 45.45454545
66
- 56 46.42857143
67
- 57 47.36842105
68
- 58 48.27586207
69
- 59 47.45762712
70
- 60 46.66666667
71
- 61 47.54098361
72
- 62 46.77419355
73
- 63 46.03174603
74
- 64 45.31250000
75
- 65 44.61538462
76
- 66 43.93939394
77
- 67 43.28358209
78
- 68 42.64705882
79
- 69 42.02898551
80
- 70 42.85714286
81
- 71 43.66197183
82
- 72 43.05555556
83
- 73 42.46575342
84
- 74 43.24324324
85
- 75 44.00000000
86
- 76 44.73684211
87
- 77 45.45454545
88
- 78 46.15384615
89
- 79 45.56962025
90
- 80 46.25000000
91
- 81 46.91358025
92
- 82 46.34146341
93
- 83 46.98795181
94
- 84 47.61904762
95
- 85 47.05882353
96
- 86 46.51162791
97
- 87 47.12643678
98
- 88 47.72727273
99
- 89 47.19101124
100
- 90 46.66666667
101
- 91 46.15384615
102
- 92 45.65217391
103
- 93 45.16129032
104
- 94 45.74468085
105
- 95 45.26315789
106
- 96 45.83333333
107
- 97 45.36082474
108
- 98 44.89795918
109
- 99 44.44444444
110
- 100 44.00000000
111
- 101 44.55445545
112
- 102 44.11764706
113
- 103 43.68932039
114
- 104 44.23076923
115
- 105 44.76190476
116
- 106 45.28301887
117
- 107 44.85981308
118
- 108 45.37037037
119
- 109 44.95412844
120
- 110 44.54545455
121
- 111 45.04504505
122
- 112 44.64285714
123
- 113 44.24778761
124
- 114 44.73684211
125
- 115 45.21739130
126
- 116 45.68965517
127
- 117 45.29914530
128
- 118 44.91525424
129
- 119 45.37815126
130
- 120 45.00000000
131
- 121 45.45454545
132
- 122 45.08196721
133
- 123 44.71544715
134
- 124 45.16129032
135
- 125 45.60000000
136
- 126 45.23809524
137
- 127 44.88188976
138
- 128 44.53125000
139
- 129 44.18604651
140
- 130 43.84615385
141
- 131 44.27480916
142
- 132 44.69696970
143
- 133 44.36090226
144
- 134 44.02985075
145
- 135 43.70370370
146
- 136 43.38235294
147
- 137 43.06569343
148
- 138 42.75362319
149
- 139 43.16546763
150
- 140 43.57142857
151
- 141 43.97163121
152
- 142 44.36619718
153
- 143 44.75524476
154
- 144 44.44444444
155
- 145 44.82758621
156
- 146 45.20547945
157
- 147 44.89795918
158
- 148 45.27027027
159
- 149 44.96644295
160
- 150 44.66666667
161
- 151 44.37086093
162
- 152 44.73684211
163
- 153 45.09803922
164
- 154 44.80519481
165
- 155 44.51612903
166
- 156 44.23076923
167
- 157 44.58598726
168
- 158 44.93670886
169
- 159 44.65408805
170
- 160 45.00000000
171
- 161 44.72049689
172
- 162 44.44444444
173
- 163 44.17177914
174
- 164 43.90243902
175
- 165 44.24242424
176
- 166 43.97590361
177
- 167 44.31137725
178
- 168 44.04761905
179
- 169 43.78698225
180
- 170 43.52941176
181
- 171 43.27485380
182
- 172 43.02325581
183
- 173 42.77456647
184
- 174 42.52873563
185
- 175 42.28571429
186
- 176 42.61363636
187
- 177 42.37288136
188
- 178 42.13483146
189
- 179 42.45810056
190
- 180 42.22222222
191
- 181 41.98895028
192
- 182 42.30769231
193
- 183 42.62295082
194
- 184 42.39130435
195
- 185 42.70270270
196
- 186 43.01075269
197
- 187 42.78074866
198
- 188 43.08510638
199
- 189 43.38624339
200
- 190 43.15789474
201
- 191 43.45549738
202
- 192 43.75000000
203
- 193 43.52331606
204
- 194 43.29896907
205
- 195 43.58974359
206
- 196 43.87755102
207
- 197 43.65482234
208
- 198 43.93939394
209
- 199 43.71859296
210
- 200 43.50000000
211
- 201 43.28358209
212
- 202 43.06930693
213
- 203 42.85714286
214
- 204 42.64705882
215
- 205 42.92682927
216
- 206 42.71844660
217
- 207 42.99516908
218
- 208 42.78846154
219
- 209 42.58373206
220
- 210 42.38095238
221
- 211 42.65402844
222
- 212 42.45283019
223
- 213 42.25352113
224
- 214 42.05607477
225
- 215 42.32558140
226
- 216 42.59259259
227
- 217 42.39631336
228
- 218 42.66055046
229
- 219 42.92237443
230
- 220 43.18181818
231
- 221 42.98642534
232
- 222 42.79279279
233
- 223 43.04932735
234
- 224 42.85714286
235
- 225 43.11111111
236
- 226 42.92035398
237
- 227 42.73127753
238
- 228 42.98245614
239
- 229 42.79475983
240
- 230 42.60869565
241
- 231 42.42424242
242
- 232 42.67241379
243
- 233 42.48927039
244
- 234 42.73504274
245
- 235 42.55319149
246
- 236 42.37288136
247
- 237 42.19409283
248
- 238 42.01680672
249
- 239 41.84100418
250
- 240 42.08333333
251
- 241 41.90871369
252
- 242 42.14876033
253
- 243 42.38683128
254
- 244 42.21311475
255
- 245 42.44897959
256
- 246 42.27642276
257
- 247 42.51012146
258
- 248 42.33870968
259
- 249 42.16867470
260
- 250 42.00000000
261
- 251 41.83266932
262
- 252 41.66666667
263
- 253 41.50197628
264
- 254 41.33858268
265
- 255 41.17647059
266
- 256 41.40625000
267
- 257 41.63424125
268
- 258 41.47286822
269
- 259 41.31274131
270
- 260 41.53846154
271
- 261 41.37931034
272
- 262 41.22137405
273
- 263 41.06463878
274
- 264 40.90909091
275
- 265 41.13207547
276
- 266 41.35338346
277
- 267 41.19850187
278
- 268 41.04477612
279
- 269 41.26394052
280
- 270 41.48148148
281
- 271 41.32841328
282
- 272 41.54411765
283
- 273 41.39194139
284
- 274 41.24087591
285
- 275 41.09090909
286
- 276 41.30434783
287
- 277 41.51624549
288
- 278 41.36690647
289
- 279 41.21863799
290
- 280 41.07142857
291
- 281 41.28113879
292
- 282 41.13475177
293
- 283 40.98939929
294
- 284 40.84507042
295
- 285 41.05263158
296
- 286 40.90909091
297
- 287 41.11498258
298
- 288 40.97222222
299
- 289 40.83044983
300
- 290 41.03448276
301
- 291 41.23711340
302
- 292 41.43835616
303
- 293 41.63822526
304
- 294 41.49659864
305
- 295 41.35593220
306
- 296 41.21621622
307
- 297 41.07744108
308
- 298 41.27516779
309
- 299 41.47157191
310
- 300 41.66666667
311
- 301 41.52823920
312
- 302 41.72185430
313
- 303 41.58415842
314
- 304 41.44736842
315
- 305 41.63934426
316
- 306 41.50326797
317
- 307 41.69381107
318
- 308 41.55844156
319
- 309 41.42394822
320
- 310 41.29032258
321
- 311 41.47909968
322
- 312 41.66666667
323
- 313 41.85303514
324
- 314 42.03821656
325
- 315 41.90476190
326
- 316 41.77215190
327
- 317 41.95583596
328
- 318 41.82389937
329
- 319 42.00626959
330
- 320 41.87500000
331
- 321 41.74454829
332
- 322 41.92546584
333
- 323 41.79566563
334
- 324 41.66666667
335
- 325 41.84615385
336
- 326 41.71779141
337
- 327 41.89602446
338
- 328 41.76829268
339
- 329 41.64133739
340
- 330 41.51515152
341
- 331 41.38972810
342
- 332 41.26506024
343
- 333 41.44144144
344
- 334 41.31736527
345
- 335 41.49253731
346
- 336 41.36904762
347
- 337 41.24629080
348
- 338 41.12426036
349
- 339 41.00294985
350
- 340 41.17647059
351
- 341 41.34897361
352
- 342 41.52046784
353
- 343 41.39941691
354
- 344 41.27906977
355
- 345 41.15942029
356
- 346 41.32947977
357
- 347 41.49855908
358
- 348 41.37931034
359
- 349 41.26074499
360
- 350 41.42857143
361
- 351 41.31054131
362
- 352 41.47727273
363
- 353 41.35977337
364
- 354 41.24293785
365
- 355 41.12676056
366
- 356 41.29213483
367
- 357 41.45658263
368
- 358 41.34078212
369
- 359 41.22562674
370
- 360 41.38888889
371
- 361 41.55124654
372
- 362 41.71270718
373
- 363 41.59779614
374
- 364 41.48351648
375
- 365 41.36986301
376
- 366 41.53005464
377
- 367 41.41689373
378
- 368 41.30434783
379
- 369 41.19241192
380
- 370 41.35135135
381
- 371 41.23989218
382
- 372 41.12903226
383
- 373 41.01876676
384
- 374 40.90909091
385
- 375 41.06666667
386
- 376 40.95744681
387
- 377 41.11405836
388
- 378 41.26984127
389
- 379 41.16094987
390
- 380 41.05263158
391
- 381 41.20734908
392
- 382 41.36125654
393
- 383 41.51436031
394
- 384 41.40625000
395
- 385 41.55844156
396
- 386 41.45077720
397
- 387 41.34366925
398
- 388 41.23711340
399
- 389 41.38817481
400
- 390 41.53846154
401
- 391 41.68797954
402
- 392 41.83673469
403
- 393 41.73027990
404
- 394 41.62436548
405
- 395 41.77215190
406
- 396 41.91919192
407
- 397 42.06549118
408
- 398 41.95979899
409
- 399 42.10526316
410
- 400 42.00000000
411
- 401 42.14463840
412
- 402 42.03980100
413
- 403 41.93548387
414
- 404 41.83168317
415
- 405 41.97530864
416
- 406 41.87192118
417
- 407 41.76904177
418
- 408 41.66666667
419
- 409 41.80929095
420
- 410 41.70731707
421
- 411 41.60583942
422
- 412 41.50485437
423
- 413 41.40435835
424
- 414 41.54589372
425
- 415 41.44578313
426
- 416 41.58653846
427
- 417 41.48681055
428
- 418 41.38755981
429
- 419 41.52744630
430
- 420 41.42857143
431
- 421 41.56769596
432
- 422 41.46919431
433
- 423 41.37115839
434
- 424 41.50943396
435
- 425 41.64705882
436
- 426 41.78403756
437
- 427 41.68618267
438
- 428 41.58878505
439
- 429 41.72494172
440
- 430 41.62790698
441
- 431 41.76334107
442
- 432 41.89814815
443
- 433 41.80138568
444
- 434 41.93548387
445
- 435 41.83908046
446
- 436 41.74311927
447
- 437 41.64759725
448
- 438 41.78082192
449
- 439 41.91343964
450
- 440 41.81818182
451
- 441 41.72335601
452
- 442 41.62895928
453
- 443 41.53498871
454
- 444 41.44144144
455
- 445 41.34831461
456
- 446 41.47982063
457
- 447 41.38702461
458
- 448 41.29464286
459
- 449 41.20267261
460
- 450 41.33333333
461
- 451 41.46341463
462
- 452 41.37168142
463
- 453 41.50110375
464
- 454 41.40969163
465
- 455 41.31868132
466
- 456 41.44736842
467
- 457 41.57549234
468
- 458 41.48471616
469
- 459 41.39433551
470
- 460 41.52173913
471
- 461 41.43167028
472
- 462 41.34199134
473
- 463 41.46868251
474
- 464 41.59482759
475
- 465 41.50537634
476
- 466 41.63090129
477
- 467 41.75588865
478
- 468 41.66666667
479
- 469 41.79104478
480
- 470 41.70212766
481
- 471 41.61358811
482
- 472 41.52542373
483
- 473 41.64904863
484
- 474 41.56118143
485
- 475 41.47368421
486
- 476 41.59663866
487
- 477 41.50943396
488
- 478 41.63179916
489
- 479 41.75365344
490
- 480 41.87500000
491
- 481 41.78794179
492
- 482 41.70124481
493
- 483 41.61490683
494
- 484 41.52892562
495
- 485 41.44329897
496
- 486 41.56378601
497
- 487 41.68377823
498
- 488 41.59836066
499
- 489 41.51329243
500
- 490 41.63265306
501
- 491 41.54786151
502
- 492 41.46341463
503
- 493 41.37931034
504
- 494 41.49797571
505
- 495 41.61616162
506
- 496 41.73387097
507
- 497 41.64989940
508
- 498 41.56626506
509
- 499 41.68336673
510
- 500 41.80000000
511
- 501 41.71656687
512
- 502 41.63346614
513
- 503 41.55069583
514
- 504 41.46825397
515
- 505 41.38613861
516
- 506 41.50197628
517
- 507 41.42011834
518
- 508 41.33858268
519
- 509 41.25736739
520
- 510 41.37254902
521
- 511 41.29158513
522
- 512 41.21093750
523
- 513 41.13060429
524
- 514 41.05058366
525
- 515 41.16504854
526
- 516 41.27906977
527
- 517 41.39264990
528
- 518 41.50579151
529
- 519 41.42581888
530
- 520 41.53846154
531
- 521 41.45873321
532
- 522 41.57088123
533
- 523 41.49139579
534
- 524 41.41221374
535
- 525 41.33333333
536
- 526 41.25475285
537
- 527 41.17647059
538
- 528 41.28787879
539
- 529 41.39886578
540
- 530 41.50943396
541
- 531 41.43126177
542
- 532 41.35338346
543
- 533 41.27579737
544
- 534 41.38576779
545
- 535 41.49532710
546
- 536 41.41791045
547
- 537 41.52700186
548
- 538 41.63568773
549
- 539 41.55844156
550
- 540 41.48148148
551
- 541 41.40480591
552
- 542 41.51291513
553
- 543 41.43646409
554
- 544 41.54411765
555
- 545 41.46788991
556
- 546 41.39194139
557
- 547 41.31627057
558
- 548 41.42335766
559
- 549 41.34790528
560
- 550 41.27272727
561
- 551 41.37931034
562
- 552 41.30434783
563
- 553 41.22965642
564
- 554 41.15523466
565
- 555 41.08108108
566
- 556 41.18705036
567
- 557 41.29263914
568
- 558 41.39784946
569
- 559 41.32379249
570
- 560 41.25000000
571
- 561 41.17647059
572
- 562 41.28113879
573
- 563 41.38543517
574
- 564 41.31205674
575
- 565 41.41592920
576
- 566 41.34275618
577
- 567 41.26984127
578
- 568 41.37323944
579
- 569 41.30052724
580
- 570 41.22807018
581
- 571 41.15586690
582
- 572 41.25874126
583
- 573 41.18673647
584
- 574 41.11498258
585
- 575 41.04347826
586
- 576 41.14583333
587
- 577 41.24783362
588
- 578 41.17647059
589
- 579 41.10535406
590
- 580 41.20689655
591
- 581 41.13597246
592
- 582 41.06529210
593
- 583 41.16638079
594
- 584 41.26712329
595
- 585 41.36752137
596
- 586 41.46757679
597
- 587 41.39693356
598
- 588 41.32653061
599
- 589 41.25636672
600
- 590 41.18644068
601
- 591 41.11675127
602
- 592 41.04729730
603
- 593 41.14671164
604
- 594 41.24579125
605
- 595 41.17647059
606
- 596 41.27516779
607
- 597 41.20603015
608
- 598 41.13712375
609
- 599 41.23539232
610
- 600 41.16666667
611
- 601 41.09816972
612
- 602 41.02990033
613
- 603 41.12769486
614
- 604 41.05960265
615
- 605 40.99173554
616
- 606 40.92409241
617
- 607 41.02141680
618
- 608 40.95394737
619
- 609 40.88669951
620
- 610 40.81967213
621
- 611 40.91653028
622
- 612 40.84967320
623
- 613 40.94616639
624
- 614 40.87947883
625
- 615 40.81300813
626
- 616 40.90909091
627
- 617 41.00486224
628
- 618 40.93851133
629
- 619 40.87237480
630
- 620 40.80645161
631
- 621 40.74074074
632
- 622 40.67524116
633
- 623 40.60995185
634
- 624 40.54487179
635
- 625 40.48000000
636
- 626 40.57507987
637
- 627 40.51036683
638
- 628 40.44585987
639
- 629 40.54054054
640
- 630 40.63492063
641
- 631 40.72900158
642
- 632 40.82278481
643
- 633 40.75829384
644
- 634 40.69400631
645
- 635 40.62992126
646
- 636 40.56603774
647
- 637 40.50235479
648
- 638 40.43887147
649
- 639 40.53208138
650
- 640 40.62500000
651
- 641 40.56162246
652
- 642 40.49844237
653
- 643 40.43545879
654
- 644 40.52795031
655
- 645 40.46511628
656
- 646 40.40247678
657
- 647 40.34003091
658
- 648 40.27777778
659
- 649 40.21571649
660
- 650 40.15384615
661
- 651 40.09216590
662
- 652 40.03067485
663
- 653 39.96937213
664
- 654 40.06116208
665
- 655 40.00000000
666
- 656 39.93902439
667
- 657 39.87823440
668
- 658 39.96960486
669
- 659 39.90895296
670
- 660 39.84848485
671
- 661 39.78819970
672
- 662 39.72809668
673
- 663 39.66817496
674
- 664 39.60843373
675
- 665 39.54887218
676
- 666 39.48948949
677
- 667 39.58020990
678
- 668 39.67065868
679
- 669 39.76083707
680
- 670 39.85074627
681
- 671 39.79135618
682
- 672 39.73214286
683
- 673 39.67310550
684
- 674 39.61424332
685
- 675 39.55555556
686
- 676 39.49704142
687
- 677 39.43870015
688
- 678 39.52802360
689
- 679 39.46980854
690
- 680 39.55882353
691
- 681 39.64757709
692
- 682 39.58944282
693
- 683 39.53147877
694
- 684 39.61988304
695
- 685 39.56204380
696
- 686 39.65014577
697
- 687 39.59243086
698
- 688 39.53488372
699
- 689 39.47750363
700
- 690 39.42028986
701
- 691 39.50795948
702
- 692 39.45086705
703
- 693 39.39393939
704
- 694 39.33717579
705
- 695 39.28057554
706
- 696 39.36781609
707
- 697 39.31133429
708
- 698 39.39828080
709
- 699 39.48497854
710
- 700 39.42857143
711
- 701 39.51497860
712
- 702 39.60113960
713
- 703 39.68705548
714
- 704 39.77272727
715
- 705 39.71631206
716
- 706 39.66005666
717
- 707 39.60396040
718
- 708 39.54802260
719
- 709 39.49224260
720
- 710 39.43661972
721
- 711 39.52180028
722
- 712 39.60674157
723
- 713 39.69144460
724
- 714 39.63585434
725
- 715 39.58041958
726
- 716 39.52513966
727
- 717 39.47001395
728
- 718 39.41504178
729
- 719 39.49930459
730
- 720 39.44444444
731
- 721 39.38973648
732
- 722 39.33518006
733
- 723 39.41908714
734
- 724 39.36464088
735
- 725 39.44827586
736
- 726 39.53168044
737
- 727 39.47730399
738
- 728 39.42307692
739
- 729 39.36899863
740
- 730 39.31506849
741
- 731 39.26128591
742
- 732 39.20765027
743
- 733 39.29058663
744
- 734 39.37329700
745
- 735 39.31972789
746
- 736 39.26630435
747
- 737 39.34871099
748
- 738 39.43089431
749
- 739 39.51285521
750
- 740 39.45945946
751
- 741 39.40620783
752
- 742 39.48787062
753
- 743 39.56931359
754
- 744 39.65053763
755
- 745 39.59731544
756
- 746 39.54423592
757
- 747 39.49129853
758
- 748 39.43850267
759
- 749 39.51935915
760
- 750 39.60000000
761
 
762
- Final result: 39.6000 +/- 1.7870
763
- Random chance: 25.0000 +/- 1.5822
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-IQ3_S.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 36.1333 +/- 1.7553
6
+ Random chance: 25.0000 +/- 1.5822
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 279.05 ms
10
+ llama_perf_context_print: prompt eval time = 257292.94 ms / 70079 tokens ( 3.67 ms per token, 272.37 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 260084.41 ms / 70080 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-iq3_s.ppx CHANGED
@@ -1,37 +1,37 @@
1
  ====== Perplexity statistics ======
2
- Mean PPL(Q) : 9.156208 ± 0.059074
3
- Mean PPL(base) : 7.534124 ± 0.048206
4
- Cor(ln(PPL(Q)), ln(PPL(base))): 96.88%
5
- Mean ln(PPL(Q)/PPL(base)) : 0.194990 ± 0.001607
6
- Mean PPL(Q)/PPL(base) : 1.215298 ± 0.001953
7
- Mean PPL(Q)-PPL(base) : 1.622084 ± 0.017205
8
 
9
  ====== KL divergence statistics ======
10
- Mean KLD: 0.176807 ± 0.000516
11
- Maximum KLD: 7.850962
12
- 99.9% KLD: 2.159336
13
- 99.0% KLD: 0.866296
14
- 99.0% KLD: 0.866296
15
- Median KLD: 0.144583
16
- 10.0% KLD: 0.010968
17
- 5.0% KLD: 0.003578
18
- 1.0% KLD: 0.000530
19
- Minimum KLD: 0.000004
20
 
21
  ====== Token probability statistics ======
22
- Mean Δp: -4.620 ± 0.031 %
23
- Maximum Δp: 81.902%
24
- 99.9% Δp: 38.750%
25
- 99.0% Δp: 22.129%
26
- 95.0% Δp: 9.587%
27
- 90.0% Δp: 4.291%
28
- 75.0% Δp: 0.093%
29
- Median Δp: -1.101%
30
- 25.0% Δp: -8.157%
31
- 10.0% Δp: -19.324%
32
- 5.0% Δp: -27.170%
33
- 1.0% Δp: -44.631%
34
- 0.1% Δp: -74.280%
35
- Minimum Δp: -96.184%
36
- RMS Δp : 12.577 ± 0.047 %
37
- Same top p: 77.814 ± 0.110 %
 
1
  ====== Perplexity statistics ======
2
+ Mean PPL(Q) : 9.032577 ± 0.058532
3
+ Mean PPL(base) : 7.237090 ± 0.045539
4
+ Cor(ln(PPL(Q)), ln(PPL(base))): 95.96%
5
+ Mean ln(PPL(Q)/PPL(base)) : 0.221619 ± 0.001825
6
+ Mean PPL(Q)/PPL(base) : 1.248095 ± 0.002278
7
+ Mean PPL(Q)-PPL(base) : 1.795488 ± 0.019603
8
 
9
  ====== KL divergence statistics ======
10
+ Mean KLD: 0.204862 ± 0.000758
11
+ Maximum KLD: 9.152626
12
+ 99.9% KLD: 3.342306
13
+ 99.0% KLD: 1.358416
14
+ 99.0% KLD: 1.358416
15
+ Median KLD: 0.143007
16
+ 10.0% KLD: 0.013411
17
+ 5.0% KLD: 0.004899
18
+ 1.0% KLD: 0.000896
19
+ Minimum KLD: 0.000000
20
 
21
  ====== Token probability statistics ======
22
+ Mean Δp: -4.675 ± 0.034 %
23
+ Maximum Δp: 83.272%
24
+ 99.9% Δp: 42.833%
25
+ 99.0% Δp: 23.669%
26
+ 95.0% Δp: 10.247%
27
+ 90.0% Δp: 4.840%
28
+ 75.0% Δp: 0.184%
29
+ Median Δp: -1.059%
30
+ 25.0% Δp: -7.819%
31
+ 10.0% Δp: -19.392%
32
+ 5.0% Δp: -28.470%
33
+ 1.0% Δp: -53.110%
34
+ 0.1% Δp: -86.125%
35
+ Minimum Δp: -99.905%
36
+ RMS Δp : 13.678 ± 0.058 %
37
+ Same top p: 78.198 ± 0.109 %
scores/Watt-Tool-8B-iq3_s.tqa CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 817 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 817 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 50.00000000
13
- 3 33.33333333
14
- 4 25.00000000
15
- 5 20.00000000
16
- 6 16.66666667
17
- 7 14.28571429
18
- 8 12.50000000
19
- 9 11.11111111
20
- 10 10.00000000
21
- 11 9.09090909
22
- 12 16.66666667
23
- 13 23.07692308
24
- 14 21.42857143
25
- 15 20.00000000
26
- 16 25.00000000
27
- 17 23.52941176
28
- 18 22.22222222
29
- 19 21.05263158
30
- 20 20.00000000
31
- 21 19.04761905
32
- 22 18.18181818
33
- 23 17.39130435
34
- 24 20.83333333
35
- 25 20.00000000
36
- 26 19.23076923
37
- 27 18.51851852
38
- 28 17.85714286
39
- 29 20.68965517
40
- 30 20.00000000
41
- 31 22.58064516
42
- 32 25.00000000
43
- 33 27.27272727
44
- 34 26.47058824
45
- 35 25.71428571
46
- 36 25.00000000
47
- 37 27.02702703
48
- 38 26.31578947
49
- 39 28.20512821
50
- 40 27.50000000
51
- 41 26.82926829
52
- 42 26.19047619
53
- 43 27.90697674
54
- 44 27.27272727
55
- 45 28.88888889
56
- 46 28.26086957
57
- 47 29.78723404
58
- 48 31.25000000
59
- 49 30.61224490
60
- 50 32.00000000
61
- 51 31.37254902
62
- 52 30.76923077
63
- 53 30.18867925
64
- 54 29.62962963
65
- 55 30.90909091
66
- 56 30.35714286
67
- 57 29.82456140
68
- 58 29.31034483
69
- 59 30.50847458
70
- 60 31.66666667
71
- 61 31.14754098
72
- 62 32.25806452
73
- 63 31.74603175
74
- 64 31.25000000
75
- 65 30.76923077
76
- 66 30.30303030
77
- 67 29.85074627
78
- 68 29.41176471
79
- 69 30.43478261
80
- 70 30.00000000
81
- 71 29.57746479
82
- 72 30.55555556
83
- 73 30.13698630
84
- 74 29.72972973
85
- 75 29.33333333
86
- 76 28.94736842
87
- 77 28.57142857
88
- 78 28.20512821
89
- 79 27.84810127
90
- 80 28.75000000
91
- 81 28.39506173
92
- 82 29.26829268
93
- 83 28.91566265
94
- 84 28.57142857
95
- 85 28.23529412
96
- 86 27.90697674
97
- 87 27.58620690
98
- 88 27.27272727
99
- 89 28.08988764
100
- 90 27.77777778
101
- 91 27.47252747
102
- 92 27.17391304
103
- 93 26.88172043
104
- 94 26.59574468
105
- 95 27.36842105
106
- 96 28.12500000
107
- 97 28.86597938
108
- 98 28.57142857
109
- 99 29.29292929
110
- 100 30.00000000
111
- 101 30.69306931
112
- 102 30.39215686
113
- 103 30.09708738
114
- 104 29.80769231
115
- 105 29.52380952
116
- 106 29.24528302
117
- 107 29.90654206
118
- 108 30.55555556
119
- 109 30.27522936
120
- 110 30.90909091
121
- 111 30.63063063
122
- 112 31.25000000
123
- 113 31.85840708
124
- 114 32.45614035
125
- 115 32.17391304
126
- 116 31.89655172
127
- 117 31.62393162
128
- 118 31.35593220
129
- 119 31.09243697
130
- 120 31.66666667
131
- 121 32.23140496
132
- 122 31.96721311
133
- 123 31.70731707
134
- 124 32.25806452
135
- 125 32.80000000
136
- 126 32.53968254
137
- 127 32.28346457
138
- 128 32.03125000
139
- 129 32.55813953
140
- 130 33.07692308
141
- 131 32.82442748
142
- 132 32.57575758
143
- 133 32.33082707
144
- 134 32.08955224
145
- 135 32.59259259
146
- 136 32.35294118
147
- 137 32.84671533
148
- 138 33.33333333
149
- 139 33.81294964
150
- 140 33.57142857
151
- 141 33.33333333
152
- 142 33.09859155
153
- 143 32.86713287
154
- 144 33.33333333
155
- 145 33.10344828
156
- 146 32.87671233
157
- 147 33.33333333
158
- 148 33.10810811
159
- 149 32.88590604
160
- 150 32.66666667
161
- 151 33.11258278
162
- 152 32.89473684
163
- 153 32.67973856
164
- 154 32.46753247
165
- 155 32.90322581
166
- 156 33.33333333
167
- 157 33.12101911
168
- 158 33.54430380
169
- 159 33.33333333
170
- 160 33.12500000
171
- 161 32.91925466
172
- 162 32.71604938
173
- 163 32.51533742
174
- 164 32.31707317
175
- 165 32.72727273
176
- 166 32.53012048
177
- 167 32.33532934
178
- 168 32.73809524
179
- 169 32.54437870
180
- 170 32.35294118
181
- 171 32.16374269
182
- 172 31.97674419
183
- 173 31.79190751
184
- 174 32.18390805
185
- 175 32.00000000
186
- 176 32.38636364
187
- 177 32.20338983
188
- 178 32.02247191
189
- 179 32.40223464
190
- 180 32.77777778
191
- 181 32.59668508
192
- 182 32.41758242
193
- 183 32.24043716
194
- 184 32.06521739
195
- 185 31.89189189
196
- 186 32.25806452
197
- 187 32.08556150
198
- 188 31.91489362
199
- 189 31.74603175
200
- 190 31.57894737
201
- 191 31.41361257
202
- 192 31.25000000
203
- 193 31.08808290
204
- 194 31.44329897
205
- 195 31.28205128
206
- 196 31.12244898
207
- 197 31.47208122
208
- 198 31.81818182
209
- 199 31.65829146
210
- 200 31.50000000
211
- 201 31.34328358
212
- 202 31.18811881
213
- 203 31.03448276
214
- 204 30.88235294
215
- 205 31.21951220
216
- 206 31.06796117
217
- 207 31.40096618
218
- 208 31.25000000
219
- 209 31.10047847
220
- 210 30.95238095
221
- 211 30.80568720
222
- 212 30.66037736
223
- 213 30.51643192
224
- 214 30.37383178
225
- 215 30.23255814
226
- 216 30.09259259
227
- 217 30.41474654
228
- 218 30.73394495
229
- 219 30.59360731
230
- 220 30.45454545
231
- 221 30.31674208
232
- 222 30.18018018
233
- 223 30.49327354
234
- 224 30.80357143
235
- 225 30.66666667
236
- 226 30.53097345
237
- 227 30.83700441
238
- 228 30.70175439
239
- 229 30.56768559
240
- 230 30.43478261
241
- 231 30.30303030
242
- 232 30.17241379
243
- 233 30.47210300
244
- 234 30.76923077
245
- 235 30.63829787
246
- 236 30.93220339
247
- 237 31.22362869
248
- 238 31.51260504
249
- 239 31.38075314
250
- 240 31.25000000
251
- 241 31.12033195
252
- 242 30.99173554
253
- 243 30.86419753
254
- 244 30.73770492
255
- 245 30.61224490
256
- 246 30.48780488
257
- 247 30.76923077
258
- 248 30.64516129
259
- 249 30.92369478
260
- 250 30.80000000
261
- 251 31.07569721
262
- 252 30.95238095
263
- 253 30.83003953
264
- 254 30.70866142
265
- 255 30.58823529
266
- 256 30.46875000
267
- 257 30.35019455
268
- 258 30.23255814
269
- 259 30.11583012
270
- 260 30.00000000
271
- 261 30.26819923
272
- 262 30.15267176
273
- 263 30.03802281
274
- 264 30.30303030
275
- 265 30.56603774
276
- 266 30.45112782
277
- 267 30.33707865
278
- 268 30.22388060
279
- 269 30.11152416
280
- 270 30.00000000
281
- 271 29.88929889
282
- 272 29.77941176
283
- 273 30.03663004
284
- 274 29.92700730
285
- 275 29.81818182
286
- 276 29.71014493
287
- 277 29.60288809
288
- 278 29.49640288
289
- 279 29.39068100
290
- 280 29.28571429
291
- 281 29.53736655
292
- 282 29.43262411
293
- 283 29.68197880
294
- 284 29.57746479
295
- 285 29.47368421
296
- 286 29.72027972
297
- 287 29.96515679
298
- 288 29.86111111
299
- 289 29.75778547
300
- 290 29.65517241
301
- 291 29.89690722
302
- 292 29.79452055
303
- 293 30.03412969
304
- 294 29.93197279
305
- 295 30.16949153
306
- 296 30.40540541
307
- 297 30.63973064
308
- 298 30.53691275
309
- 299 30.43478261
310
- 300 30.33333333
311
- 301 30.23255814
312
- 302 30.13245033
313
- 303 30.03300330
314
- 304 29.93421053
315
- 305 29.83606557
316
- 306 29.73856209
317
- 307 29.64169381
318
- 308 29.87012987
319
- 309 29.77346278
320
- 310 29.67741935
321
- 311 29.58199357
322
- 312 29.80769231
323
- 313 29.71246006
324
- 314 29.61783439
325
- 315 29.52380952
326
- 316 29.43037975
327
- 317 29.33753943
328
- 318 29.24528302
329
- 319 29.15360502
330
- 320 29.06250000
331
- 321 28.97196262
332
- 322 29.19254658
333
- 323 29.10216718
334
- 324 29.01234568
335
- 325 28.92307692
336
- 326 28.83435583
337
- 327 29.05198777
338
- 328 29.26829268
339
- 329 29.17933131
340
- 330 29.39393939
341
- 331 29.30513595
342
- 332 29.51807229
343
- 333 29.72972973
344
- 334 29.64071856
345
- 335 29.85074627
346
- 336 30.05952381
347
- 337 29.97032641
348
- 338 29.88165680
349
- 339 30.08849558
350
- 340 30.00000000
351
- 341 30.20527859
352
- 342 30.40935673
353
- 343 30.32069971
354
- 344 30.52325581
355
- 345 30.72463768
356
- 346 30.63583815
357
- 347 30.83573487
358
- 348 30.74712644
359
- 349 30.65902579
360
- 350 30.57142857
361
- 351 30.48433048
362
- 352 30.39772727
363
- 353 30.31161473
364
- 354 30.22598870
365
- 355 30.42253521
366
- 356 30.33707865
367
- 357 30.25210084
368
- 358 30.16759777
369
- 359 30.08356546
370
- 360 30.00000000
371
- 361 29.91689751
372
- 362 29.83425414
373
- 363 30.02754821
374
- 364 29.94505495
375
- 365 29.86301370
376
- 366 29.78142077
377
- 367 29.97275204
378
- 368 29.89130435
379
- 369 29.81029810
380
- 370 29.72972973
381
- 371 29.64959569
382
- 372 29.83870968
383
- 373 29.75871314
384
- 374 29.94652406
385
- 375 29.86666667
386
- 376 29.78723404
387
- 377 29.97347480
388
- 378 29.89417989
389
- 379 29.81530343
390
- 380 30.00000000
391
- 381 29.92125984
392
- 382 29.84293194
393
- 383 29.76501305
394
- 384 29.68750000
395
- 385 29.61038961
396
- 386 29.53367876
397
- 387 29.45736434
398
- 388 29.63917526
399
- 389 29.56298201
400
- 390 29.48717949
401
- 391 29.66751918
402
- 392 29.59183673
403
- 393 29.77099237
404
- 394 29.94923858
405
- 395 29.87341772
406
- 396 30.05050505
407
- 397 30.22670025
408
- 398 30.15075377
409
- 399 30.07518797
410
- 400 30.00000000
411
- 401 30.17456359
412
- 402 30.34825871
413
- 403 30.27295285
414
- 404 30.19801980
415
- 405 30.12345679
416
- 406 30.04926108
417
- 407 29.97542998
418
- 408 29.90196078
419
- 409 29.82885086
420
- 410 29.75609756
421
- 411 29.68369830
422
- 412 29.85436893
423
- 413 29.78208232
424
- 414 29.95169082
425
- 415 30.12048193
426
- 416 30.04807692
427
- 417 30.21582734
428
- 418 30.14354067
429
- 419 30.07159905
430
- 420 30.23809524
431
- 421 30.40380048
432
- 422 30.33175355
433
- 423 30.49645390
434
- 424 30.42452830
435
- 425 30.35294118
436
- 426 30.28169014
437
- 427 30.21077283
438
- 428 30.14018692
439
- 429 30.06993007
440
- 430 30.00000000
441
- 431 29.93039443
442
- 432 30.09259259
443
- 433 30.25404157
444
- 434 30.41474654
445
- 435 30.34482759
446
- 436 30.27522936
447
- 437 30.20594966
448
- 438 30.36529680
449
- 439 30.52391800
450
- 440 30.45454545
451
- 441 30.38548753
452
- 442 30.31674208
453
- 443 30.24830700
454
- 444 30.40540541
455
- 445 30.33707865
456
- 446 30.26905830
457
- 447 30.20134228
458
- 448 30.13392857
459
- 449 30.28953229
460
- 450 30.22222222
461
- 451 30.15521064
462
- 452 30.08849558
463
- 453 30.24282561
464
- 454 30.17621145
465
- 455 30.10989011
466
- 456 30.04385965
467
- 457 29.97811816
468
- 458 29.91266376
469
- 459 29.84749455
470
- 460 29.78260870
471
- 461 29.71800434
472
- 462 29.87012987
473
- 463 30.02159827
474
- 464 30.17241379
475
- 465 30.32258065
476
- 466 30.25751073
477
- 467 30.19271949
478
- 468 30.34188034
479
- 469 30.27718550
480
- 470 30.21276596
481
- 471 30.36093418
482
- 472 30.50847458
483
- 473 30.44397463
484
- 474 30.59071730
485
- 475 30.73684211
486
- 476 30.67226891
487
- 477 30.60796646
488
- 478 30.54393305
489
- 479 30.48016701
490
- 480 30.41666667
491
- 481 30.35343035
492
- 482 30.49792531
493
- 483 30.43478261
494
- 484 30.37190083
495
- 485 30.51546392
496
- 486 30.45267490
497
- 487 30.59548255
498
- 488 30.53278689
499
- 489 30.47034765
500
- 490 30.61224490
501
- 491 30.75356415
502
- 492 30.89430894
503
- 493 30.83164300
504
- 494 30.97165992
505
- 495 30.90909091
506
- 496 30.84677419
507
- 497 30.78470825
508
- 498 30.72289157
509
- 499 30.66132265
510
- 500 30.60000000
511
- 501 30.73852295
512
- 502 30.67729084
513
- 503 30.61630219
514
- 504 30.75396825
515
- 505 30.69306931
516
- 506 30.63241107
517
- 507 30.57199211
518
- 508 30.70866142
519
- 509 30.84479371
520
- 510 30.98039216
521
- 511 30.91976517
522
- 512 30.85937500
523
- 513 30.79922027
524
- 514 30.73929961
525
- 515 30.87378641
526
- 516 30.81395349
527
- 517 30.94777563
528
- 518 30.88803089
529
- 519 30.82851638
530
- 520 30.76923077
531
- 521 30.71017274
532
- 522 30.65134100
533
- 523 30.59273423
534
- 524 30.53435115
535
- 525 30.47619048
536
- 526 30.41825095
537
- 527 30.55028463
538
- 528 30.49242424
539
- 529 30.62381853
540
- 530 30.56603774
541
- 531 30.50847458
542
- 532 30.45112782
543
- 533 30.58161351
544
- 534 30.52434457
545
- 535 30.46728972
546
- 536 30.41044776
547
- 537 30.54003724
548
- 538 30.48327138
549
- 539 30.42671614
550
- 540 30.55555556
551
- 541 30.49907579
552
- 542 30.44280443
553
- 543 30.38674033
554
- 544 30.51470588
555
- 545 30.45871560
556
- 546 30.40293040
557
- 547 30.53016453
558
- 548 30.65693431
559
- 549 30.60109290
560
- 550 30.54545455
561
- 551 30.49001815
562
- 552 30.61594203
563
- 553 30.74141049
564
- 554 30.68592058
565
- 555 30.63063063
566
- 556 30.57553957
567
- 557 30.52064632
568
- 558 30.46594982
569
- 559 30.41144902
570
- 560 30.53571429
571
- 561 30.48128342
572
- 562 30.42704626
573
- 563 30.37300178
574
- 564 30.49645390
575
- 565 30.61946903
576
- 566 30.74204947
577
- 567 30.68783069
578
- 568 30.80985915
579
- 569 30.93145870
580
- 570 30.87719298
581
- 571 30.99824869
582
- 572 30.94405594
583
- 573 30.89005236
584
- 574 31.01045296
585
- 575 31.13043478
586
- 576 31.25000000
587
- 577 31.36915078
588
- 578 31.48788927
589
- 579 31.43350604
590
- 580 31.55172414
591
- 581 31.49741824
592
- 582 31.61512027
593
- 583 31.56089194
594
- 584 31.67808219
595
- 585 31.62393162
596
- 586 31.74061433
597
- 587 31.68654174
598
- 588 31.63265306
599
- 589 31.57894737
600
- 590 31.52542373
601
- 591 31.47208122
602
- 592 31.41891892
603
- 593 31.36593592
604
- 594 31.48148148
605
- 595 31.42857143
606
- 596 31.37583893
607
- 597 31.32328308
608
- 598 31.27090301
609
- 599 31.21869783
610
- 600 31.16666667
611
- 601 31.28119800
612
- 602 31.39534884
613
- 603 31.50912106
614
- 604 31.45695364
615
- 605 31.40495868
616
- 606 31.51815182
617
- 607 31.46622735
618
- 608 31.41447368
619
- 609 31.36288998
620
- 610 31.31147541
621
- 611 31.26022913
622
- 612 31.37254902
623
- 613 31.32137031
624
- 614 31.27035831
625
- 615 31.21951220
626
- 616 31.33116883
627
- 617 31.44246353
628
- 618 31.39158576
629
- 619 31.50242326
630
- 620 31.45161290
631
- 621 31.40096618
632
- 622 31.51125402
633
- 623 31.46067416
634
- 624 31.41025641
635
- 625 31.36000000
636
- 626 31.30990415
637
- 627 31.25996810
638
- 628 31.36942675
639
- 629 31.31955485
640
- 630 31.26984127
641
- 631 31.37876387
642
- 632 31.32911392
643
- 633 31.43759874
644
- 634 31.54574132
645
- 635 31.49606299
646
- 636 31.44654088
647
- 637 31.39717425
648
- 638 31.34796238
649
- 639 31.29890454
650
- 640 31.40625000
651
- 641 31.35725429
652
- 642 31.46417445
653
- 643 31.41524106
654
- 644 31.36645963
655
- 645 31.47286822
656
- 646 31.42414861
657
- 647 31.37557960
658
- 648 31.48148148
659
- 649 31.43297381
660
- 650 31.53846154
661
- 651 31.64362519
662
- 652 31.74846626
663
- 653 31.85298622
664
- 654 31.80428135
665
- 655 31.75572519
666
- 656 31.85975610
667
- 657 31.96347032
668
- 658 31.91489362
669
- 659 32.01820941
670
- 660 31.96969697
671
- 661 31.92133132
672
- 662 31.87311178
673
- 663 31.82503771
674
- 664 31.77710843
675
- 665 31.72932331
676
- 666 31.83183183
677
- 667 31.78410795
678
- 668 31.73652695
679
- 669 31.68908819
680
- 670 31.79104478
681
- 671 31.74366617
682
- 672 31.69642857
683
- 673 31.64933135
684
- 674 31.60237389
685
- 675 31.70370370
686
- 676 31.80473373
687
- 677 31.75775480
688
- 678 31.71091445
689
- 679 31.66421208
690
- 680 31.61764706
691
- 681 31.57121880
692
- 682 31.52492669
693
- 683 31.47877013
694
- 684 31.57894737
695
- 685 31.53284672
696
- 686 31.48688047
697
- 687 31.44104803
698
- 688 31.39534884
699
- 689 31.49492017
700
- 690 31.44927536
701
- 691 31.54848046
702
- 692 31.64739884
703
- 693 31.74603175
704
- 694 31.70028818
705
- 695 31.65467626
706
- 696 31.75287356
707
- 697 31.70731707
708
- 698 31.66189112
709
- 699 31.61659514
710
- 700 31.71428571
711
- 701 31.66904422
712
- 702 31.76638177
713
- 703 31.72119488
714
- 704 31.67613636
715
- 705 31.63120567
716
- 706 31.72804533
717
- 707 31.82461103
718
- 708 31.77966102
719
- 709 31.73483780
720
- 710 31.83098592
721
- 711 31.78621660
722
- 712 31.88202247
723
- 713 31.83730715
724
- 714 31.79271709
725
- 715 31.88811189
726
- 716 31.84357542
727
- 717 31.93863319
728
- 718 32.03342618
729
- 719 32.12795549
730
- 720 32.08333333
731
- 721 32.03883495
732
- 722 31.99445983
733
- 723 31.95020747
734
- 724 32.04419890
735
- 725 32.00000000
736
- 726 32.09366391
737
- 727 32.04951857
738
- 728 32.14285714
739
- 729 32.23593964
740
- 730 32.19178082
741
- 731 32.14774282
742
- 732 32.24043716
743
- 733 32.19645293
744
- 734 32.15258856
745
- 735 32.10884354
746
- 736 32.06521739
747
- 737 32.15739484
748
- 738 32.11382114
749
- 739 32.07036536
750
- 740 32.02702703
751
- 741 32.11875843
752
- 742 32.07547170
753
- 743 32.03230148
754
- 744 32.12365591
755
- 745 32.08053691
756
- 746 32.17158177
757
- 747 32.26238286
758
- 748 32.21925134
759
- 749 32.30974633
760
- 750 32.40000000
761
 
762
- Final result: 32.4000 +/- 1.7100
763
- Random chance: 19.8992 +/- 1.4588
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-IQ3_S.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 35.5346 +/- 2.6882
6
+ Random chance: 28.4691 +/- 2.5346
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 277.68 ms
10
+ llama_perf_context_print: prompt eval time = 74951.02 ms / 17379 tokens ( 4.31 ms per token, 231.87 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 76169.50 ms / 17380 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-iq3_s.wng CHANGED
@@ -1,760 +1,11 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- winogrande_score : loaded 1267 tasks from prompt.
6
- winogrande_score : selecting 750 random tasks
7
- winogrande_score : tokenizing selected tasks
8
- winogrande_score : calculating winogrande score over selected tasks.
9
- 1 100.0000 -3.016338 -3.700354 1 1
10
- 2 100.0000 -1.808596 -1.980557 1 1
11
- 3 100.0000 -3.131317 -2.796099 2 2
12
- 4 100.0000 -4.825441 -4.855082 1 1
13
- 5 80.0000 -2.619177 -3.165452 1 2
14
- 6 83.3333 -4.821982 -4.904912 1 1
15
- 7 85.7143 -4.433872 -2.916471 2 2
16
- 8 87.5000 -3.379602 -2.686197 2 2
17
- 9 88.8889 -5.214768 -6.180322 1 1
18
- 10 90.0000 -1.893882 -1.759326 2 2
19
- 11 90.9091 -8.759066 -5.295569 2 2
20
- 12 91.6667 -3.340694 -2.248925 2 2
21
- 13 92.3077 -3.033750 -4.799620 1 1
22
- 14 92.8571 -3.211276 -3.775937 1 1
23
- 15 86.6667 -4.163718 -5.934842 1 2
24
- 16 87.5000 -4.286625 -3.884195 2 2
25
- 17 82.3529 -6.512271 -7.020851 1 2
26
- 18 83.3333 -2.008840 -3.820330 1 1
27
- 19 84.2105 -3.436059 -5.403902 1 1
28
- 20 80.0000 -2.866882 -2.648930 2 1
29
- 21 80.9524 -1.785276 -3.246026 1 1
30
- 22 77.2727 -2.473239 -2.550555 1 2
31
- 23 78.2609 -4.686691 -4.973099 1 1
32
- 24 75.0000 -2.832604 -2.382859 2 1
33
- 25 72.0000 -1.953768 -2.042300 1 2
34
- 26 73.0769 -3.764557 -4.265749 1 1
35
- 27 70.3704 -5.192149 -5.276355 1 2
36
- 28 71.4286 -4.873934 -4.984968 1 1
37
- 29 68.9655 -3.985773 -3.728333 2 1
38
- 30 70.0000 -2.051205 -1.796867 2 2
39
- 31 70.9677 -5.437857 -5.553694 1 1
40
- 32 71.8750 -4.323391 -6.020934 1 1
41
- 33 72.7273 -4.550635 -5.975172 1 1
42
- 34 73.5294 -2.652501 -2.824857 1 1
43
- 35 74.2857 -4.781831 -4.707762 2 2
44
- 36 75.0000 -3.965620 -3.821334 2 2
45
- 37 75.6757 -3.191013 -3.465207 1 1
46
- 38 76.3158 -3.650637 -3.811018 1 1
47
- 39 76.9231 -5.681862 -4.553236 2 2
48
- 40 75.0000 -5.421961 -6.063195 1 2
49
- 41 75.6098 -2.182501 -2.710452 1 1
50
- 42 76.1905 -2.114013 -3.218512 1 1
51
- 43 76.7442 -3.187092 -3.172217 2 2
52
- 44 77.2727 -2.588753 -1.863370 2 2
53
- 45 77.7778 -0.106876 -11.138974 1 1
54
- 46 78.2609 -2.607622 -2.796153 1 1
55
- 47 76.5957 -3.819999 -4.028642 1 2
56
- 48 77.0833 -3.936981 -3.778211 2 2
57
- 49 75.5102 -3.072092 -1.922831 2 1
58
- 50 76.0000 -2.641234 -2.608571 2 2
59
- 51 76.4706 -2.266083 -2.675310 1 1
60
- 52 76.9231 -3.531030 -2.433093 2 2
61
- 53 77.3585 -4.172545 -4.448242 1 1
62
- 54 77.7778 -2.976446 -2.610279 2 2
63
- 55 78.1818 -3.020611 -2.910297 2 2
64
- 56 76.7857 -3.631793 -4.560276 1 2
65
- 57 77.1930 -3.354012 -2.957499 2 2
66
- 58 77.5862 -3.719047 -4.396271 1 1
67
- 59 77.9661 -3.532394 -3.525620 2 2
68
- 60 78.3333 -4.863557 -4.487694 2 2
69
- 61 78.6885 -3.180577 -3.588455 1 1
70
- 62 79.0323 -5.721639 -6.120710 1 1
71
- 63 77.7778 -4.519791 -4.914109 1 2
72
- 64 78.1250 -3.610022 -3.708459 1 1
73
- 65 78.4615 -5.470369 -4.514688 2 2
74
- 66 78.7879 -3.562167 -3.975618 1 1
75
- 67 79.1045 -3.204804 -2.559672 2 2
76
- 68 79.4118 -5.308523 -5.173804 2 2
77
- 69 79.7101 -5.020135 -6.761971 1 1
78
- 70 80.0000 -3.663180 -3.183629 2 2
79
- 71 78.8732 -3.490359 -2.565171 2 1
80
- 72 79.1667 -4.032802 -3.492890 2 2
81
- 73 78.0822 -2.692882 -2.536222 2 1
82
- 74 78.3784 -2.404188 -2.709121 1 1
83
- 75 78.6667 -1.971937 -2.363230 1 1
84
- 76 78.9474 -2.349376 -4.887977 1 1
85
- 77 79.2208 -2.701798 -2.147785 2 2
86
- 78 79.4872 -0.873406 -1.716274 1 1
87
- 79 78.4810 -2.705854 -3.544493 1 2
88
- 80 78.7500 -2.186554 -3.549894 1 1
89
- 81 79.0123 -5.611124 -4.273670 2 2
90
- 82 78.0488 -3.291211 -3.920670 1 2
91
- 83 78.3133 -6.148896 -4.837275 2 2
92
- 84 78.5714 -2.445466 -2.295965 2 2
93
- 85 78.8235 -4.874400 -5.384687 1 1
94
- 86 77.9070 -5.820992 -6.057772 1 2
95
- 87 77.0115 -2.155953 -2.030491 2 1
96
- 88 76.1364 -4.467140 -4.279336 2 1
97
- 89 76.4045 -3.666130 -4.126385 1 1
98
- 90 76.6667 -4.390772 -3.737889 2 2
99
- 91 75.8242 -4.997620 -4.771589 2 1
100
- 92 76.0870 -5.239377 -4.354457 2 2
101
- 93 75.2688 -3.104731 -3.162530 1 2
102
- 94 75.5319 -5.131552 -3.763434 2 2
103
- 95 75.7895 -4.123584 -4.389645 1 1
104
- 96 76.0417 -1.979463 -2.732426 1 1
105
- 97 76.2887 -5.267792 -5.772199 1 1
106
- 98 76.5306 -3.909905 -3.804626 2 2
107
- 99 75.7576 -2.819341 -2.574439 2 1
108
- 100 76.0000 -4.994117 -4.146884 2 2
109
- 101 76.2376 -4.740726 -4.712138 2 2
110
- 102 76.4706 -3.622826 -3.883577 1 1
111
- 103 76.6990 -2.767721 -2.555997 2 2
112
- 104 76.9231 -7.528219 -7.202917 2 2
113
- 105 77.1429 -1.537127 -2.416052 1 1
114
- 106 77.3585 -2.948394 -2.747480 2 2
115
- 107 77.5701 -2.528286 -2.270074 2 2
116
- 108 77.7778 -4.260145 -3.196341 2 2
117
- 109 77.0642 -4.477525 -5.230292 1 2
118
- 110 77.2727 -4.662229 -3.702001 2 2
119
- 111 77.4775 -3.119897 -2.804967 2 2
120
- 112 76.7857 -4.120982 -3.589855 2 1
121
- 113 76.1062 -5.291877 -6.359853 1 2
122
- 114 76.3158 -4.485590 -5.071085 1 1
123
- 115 76.5217 -1.417128 -2.336694 1 1
124
- 116 76.7241 -3.356763 -2.928499 2 2
125
- 117 76.0684 -4.511835 -4.728785 1 2
126
- 118 76.2712 -1.582574 -1.975489 1 1
127
- 119 76.4706 -1.846684 -2.742284 1 1
128
- 120 75.8333 -3.097177 -2.701705 2 1
129
- 121 76.0331 -1.023098 -0.704142 2 2
130
- 122 76.2295 -6.310169 -5.418572 2 2
131
- 123 76.4228 -3.346503 -3.895095 1 1
132
- 124 76.6129 -2.881857 -2.560783 2 2
133
- 125 76.8000 -2.481758 -2.467880 2 2
134
- 126 76.9841 -6.851459 -6.650091 2 2
135
- 127 77.1654 -2.301178 -1.533842 2 2
136
- 128 77.3438 -1.573644 -1.741959 1 1
137
- 129 77.5194 -2.385813 -1.689343 2 2
138
- 130 77.6923 -6.244230 -5.721936 2 2
139
- 131 77.0992 -3.480256 -3.570452 1 2
140
- 132 77.2727 -2.177109 -1.721857 2 2
141
- 133 77.4436 -2.347383 -4.405723 1 1
142
- 134 77.6119 -4.851100 -4.454564 2 2
143
- 135 77.7778 -4.670312 -5.412760 1 1
144
- 136 77.9412 -3.597807 -2.706506 2 2
145
- 137 78.1022 -5.102621 -5.366832 1 1
146
- 138 77.5362 -3.705326 -3.975037 1 2
147
- 139 77.6978 -4.791965 -4.837171 1 1
148
- 140 77.8571 -1.344836 -1.144453 2 2
149
- 141 78.0142 -5.388006 -4.149069 2 2
150
- 142 78.1690 -3.527023 -2.900241 2 2
151
- 143 78.3217 -4.244773 -7.631731 1 1
152
- 144 77.7778 -2.393973 -2.306131 2 1
153
- 145 77.2414 -1.096721 -13.769015 1 2
154
- 146 76.7123 -3.964808 -4.148097 1 2
155
- 147 76.8707 -5.031253 -4.492764 2 2
156
- 148 77.0270 -1.941122 -2.353145 1 1
157
- 149 77.1812 -4.352716 -4.337131 2 2
158
- 150 77.3333 -4.510396 -3.736581 2 2
159
- 151 76.8212 -1.984834 -1.928483 2 1
160
- 152 76.3158 -4.080500 -4.451440 1 2
161
- 153 76.4706 -2.665499 -3.273902 1 1
162
- 154 76.6234 -3.614988 -3.714048 1 1
163
- 155 76.1290 -2.817345 -3.017010 1 2
164
- 156 76.2821 -2.021692 -2.293266 1 1
165
- 157 76.4331 -2.671659 -3.964378 1 1
166
- 158 76.5823 -4.721590 -3.954857 2 2
167
- 159 76.7296 -3.882096 -4.918319 1 1
168
- 160 76.8750 -3.890642 -3.553253 2 2
169
- 161 77.0186 -1.831558 -2.321880 1 1
170
- 162 76.5432 -0.052503 -12.035725 1 2
171
- 163 76.6871 -2.502203 -2.684366 1 1
172
- 164 76.8293 -5.402290 -4.363969 2 2
173
- 165 76.9697 -4.058267 -3.930931 2 2
174
- 166 77.1084 -4.726624 -4.083379 2 2
175
- 167 76.6467 -4.667902 -3.992162 2 1
176
- 168 76.7857 -5.000994 -4.137993 2 2
177
- 169 76.3314 -4.380780 -5.095354 1 2
178
- 170 76.4706 -2.377985 -3.213492 1 1
179
- 171 76.6082 -3.703186 -4.419187 1 1
180
- 172 76.7442 -4.210624 -5.467400 1 1
181
- 173 76.8786 -2.938626 -2.716955 2 2
182
- 174 77.0115 -5.671019 -5.575806 2 2
183
- 175 76.5714 -4.447015 -4.438485 2 1
184
- 176 76.7045 -2.268280 -4.301332 1 1
185
- 177 76.8362 -6.369696 -6.173134 2 2
186
- 178 76.4045 -2.612391 -2.356549 2 1
187
- 179 76.5363 -2.330755 -2.821667 1 1
188
- 180 76.6667 -6.222091 -5.660235 2 2
189
- 181 76.7956 -3.971676 -2.716136 2 2
190
- 182 76.9231 -4.130423 -3.222575 2 2
191
- 183 77.0492 -5.253687 -5.046474 2 2
192
- 184 77.1739 -1.482690 -2.592338 1 1
193
- 185 76.7568 -4.493372 -4.686529 1 2
194
- 186 76.8817 -3.003799 -4.025673 1 1
195
- 187 77.0053 -4.863554 -4.417139 2 2
196
- 188 76.5957 -4.482434 -3.211960 2 1
197
- 189 76.1905 -3.941740 -4.406484 1 2
198
- 190 75.7895 -4.929833 -4.957886 1 2
199
- 191 75.9162 -3.319017 -3.992764 1 1
200
- 192 76.0417 -4.657423 -3.656500 2 2
201
- 193 75.6477 -4.139796 -4.189611 1 2
202
- 194 75.7732 -1.990885 -1.601310 2 2
203
- 195 75.3846 -3.000168 -2.866001 2 1
204
- 196 75.5102 -3.632234 -3.332855 2 2
205
- 197 75.6345 -1.318555 -4.626215 1 1
206
- 198 75.7576 -4.626329 -7.933516 1 1
207
- 199 75.8794 -5.274441 -4.231371 2 2
208
- 200 76.0000 -2.965725 -2.765537 2 2
209
- 201 76.1194 -2.849717 -3.179926 1 1
210
- 202 76.2376 -8.196797 -7.697220 2 2
211
- 203 76.3547 -3.250502 -2.740949 2 2
212
- 204 76.4706 -2.353652 -3.056568 1 1
213
- 205 76.5854 -3.568216 -2.443234 2 2
214
- 206 76.2136 -4.444749 -4.791353 1 2
215
- 207 76.3285 -3.281698 -3.765573 1 1
216
- 208 76.4423 -3.237730 -2.760785 2 2
217
- 209 76.5550 -4.400598 -3.104133 2 2
218
- 210 76.6667 -6.599264 -6.586576 2 2
219
- 211 76.3033 -5.055387 -4.524460 2 1
220
- 212 76.4151 -4.948793 -4.192063 2 2
221
- 213 76.0563 -3.732406 -4.834161 1 2
222
- 214 76.1682 -3.168198 -3.218878 1 1
223
- 215 76.2791 -4.136448 -3.821901 2 2
224
- 216 75.9259 -3.205242 -3.044082 2 1
225
- 217 76.0369 -1.367905 -2.145892 1 1
226
- 218 76.1468 -3.056241 -2.805653 2 2
227
- 219 76.2557 -3.222014 -3.345023 1 1
228
- 220 75.9091 -4.160766 -4.443542 1 2
229
- 221 76.0181 -4.045054 -8.125311 1 1
230
- 222 75.6757 -0.423356 -9.317116 1 2
231
- 223 75.3363 -3.467071 -3.626925 1 2
232
- 224 75.4464 -5.370069 -4.669429 2 2
233
- 225 75.1111 -2.897599 -3.092002 1 2
234
- 226 75.2212 -2.797588 -3.322037 1 1
235
- 227 75.3304 -2.985317 -3.151291 1 1
236
- 228 75.4386 -1.397187 -1.880324 1 1
237
- 229 75.5459 -3.442293 -3.275531 2 2
238
- 230 75.6522 -3.204458 -3.529769 1 1
239
- 231 75.7576 -2.561928 -3.193932 1 1
240
- 232 75.8621 -2.210146 -1.883842 2 2
241
- 233 75.9657 -1.301673 -1.952306 1 1
242
- 234 76.0684 -3.718387 -3.352203 2 2
243
- 235 76.1702 -3.097804 -2.346395 2 2
244
- 236 75.8475 -1.689962 -1.425384 2 1
245
- 237 75.9494 -3.149614 -3.164965 1 1
246
- 238 76.0504 -2.955509 -2.551168 2 2
247
- 239 75.7322 -2.717539 -2.861351 1 2
248
- 240 75.4167 -7.294895 -7.315476 1 2
249
- 241 75.1037 -2.609038 -3.570657 1 2
250
- 242 75.2066 -5.877479 -5.586966 2 2
251
- 243 74.8971 -4.015631 -3.515743 2 1
252
- 244 75.0000 -1.953313 -3.805146 1 1
253
- 245 75.1020 -4.121024 -4.703038 1 1
254
- 246 75.2033 -2.684841 -3.175685 1 1
255
- 247 75.3036 -3.135708 -3.130586 2 2
256
- 248 75.4032 -3.529809 -3.206077 2 2
257
- 249 75.5020 -3.091428 -2.187722 2 2
258
- 250 75.6000 -2.239947 -2.733392 1 1
259
- 251 75.2988 -1.927274 -1.895754 2 1
260
- 252 75.3968 -6.831152 -6.328040 2 2
261
- 253 75.0988 -5.388623 -5.129042 2 1
262
- 254 75.1969 -4.774097 -4.605206 2 2
263
- 255 75.2941 -3.405265 -4.338165 1 1
264
- 256 75.0000 -2.843083 -2.725087 2 1
265
- 257 74.7082 -2.812201 -2.086976 2 1
266
- 258 74.4186 -3.952132 -4.011015 1 2
267
- 259 74.5174 -5.116317 -5.648463 1 1
268
- 260 74.6154 -5.720558 -5.814836 1 1
269
- 261 74.3295 -3.748696 -3.789469 1 2
270
- 262 74.4275 -4.823616 -4.422381 2 2
271
- 263 74.1445 -4.422177 -4.381124 2 1
272
- 264 74.2424 -4.259415 -3.686740 2 2
273
- 265 74.3396 -1.274685 -2.200294 1 1
274
- 266 74.4361 -5.108119 -6.740411 1 1
275
- 267 74.1573 -3.314351 -3.123540 2 1
276
- 268 74.2537 -3.013178 -3.111267 1 1
277
- 269 74.3494 -2.928288 -2.778250 2 2
278
- 270 74.4444 -2.432161 -3.083442 1 1
279
- 271 74.5387 -3.142792 -2.787878 2 2
280
- 272 74.6324 -3.874478 -3.598253 2 2
281
- 273 74.7253 -3.535646 -3.455873 2 2
282
- 274 74.8175 -3.466761 -3.374979 2 2
283
- 275 74.5455 -3.007794 -2.783713 2 1
284
- 276 74.2754 -2.827646 -2.528085 2 1
285
- 277 74.3682 -2.790845 -3.488371 1 1
286
- 278 74.4604 -2.648247 -2.655356 1 1
287
- 279 74.5520 -5.790143 -3.488042 2 2
288
- 280 74.6429 -4.185615 -3.799936 2 2
289
- 281 74.7331 -2.662693 -1.830334 2 2
290
- 282 74.8227 -4.527612 -4.051562 2 2
291
- 283 74.5583 -3.612467 -3.995129 1 2
292
- 284 74.6479 -2.424048 -3.090434 1 1
293
- 285 74.7368 -3.383094 -4.897220 1 1
294
- 286 74.4755 -3.573808 -3.482643 2 1
295
- 287 74.5645 -3.199462 -4.420720 1 1
296
- 288 74.6528 -3.306305 -3.160614 2 2
297
- 289 74.7405 -1.811557 -2.354283 1 1
298
- 290 74.8276 -3.886954 -4.573462 1 1
299
- 291 74.9141 -2.068356 -3.238176 1 1
300
- 292 75.0000 -3.098855 -2.399107 2 2
301
- 293 75.0853 -3.862068 -3.387666 2 2
302
- 294 75.1701 -3.970066 -4.279181 1 1
303
- 295 74.9153 -1.933856 -1.925757 2 1
304
- 296 75.0000 -5.428074 -5.166237 2 2
305
- 297 74.7475 -1.529383 -1.856233 1 2
306
- 298 74.8322 -5.256212 -5.128955 2 2
307
- 299 74.9164 -2.479348 -2.365204 2 2
308
- 300 75.0000 -2.132623 -2.447510 1 1
309
- 301 75.0831 -4.120558 -4.251523 1 1
310
- 302 74.8344 -4.071233 -4.114149 1 2
311
- 303 74.5875 -8.941860 -8.877957 2 1
312
- 304 74.6711 -4.392438 -3.601057 2 2
313
- 305 74.7541 -3.628976 -3.161942 2 2
314
- 306 74.8366 -2.249700 -1.939124 2 2
315
- 307 74.9186 -2.948709 -3.333120 1 1
316
- 308 75.0000 -5.701199 -4.380801 2 2
317
- 309 75.0809 -3.944882 -4.751744 1 1
318
- 310 75.1613 -5.792801 -5.567377 2 2
319
- 311 75.2412 -0.613918 -4.599301 1 1
320
- 312 75.3205 -4.188595 -4.376330 1 1
321
- 313 75.0799 -4.530135 -4.899605 1 2
322
- 314 75.1592 -4.410392 -4.000660 2 2
323
- 315 75.2381 -1.852219 -2.968207 1 1
324
- 316 75.3165 -7.466243 -5.335471 2 2
325
- 317 75.0789 -5.036376 -4.936187 2 1
326
- 318 75.1572 -4.923122 -4.550493 2 2
327
- 319 75.2351 -1.366071 -1.505218 1 1
328
- 320 75.3125 -3.974855 -5.235562 1 1
329
- 321 75.3894 -2.979057 -2.944241 2 2
330
- 322 75.1553 -2.272531 -2.634435 1 2
331
- 323 75.2322 -3.811519 -4.016639 1 1
332
- 324 75.0000 -4.383833 -4.204798 2 1
333
- 325 75.0769 -5.433412 -4.692913 2 2
334
- 326 75.1534 -2.595671 -1.867640 2 2
335
- 327 75.2294 -2.500896 -3.019386 1 1
336
- 328 75.0000 -4.333834 -4.291324 2 1
337
- 329 75.0760 -3.263256 -7.756055 1 1
338
- 330 75.1515 -1.810918 -2.009795 1 1
339
- 331 74.9245 -3.972511 -3.409039 2 1
340
- 332 75.0000 -2.142272 -3.261034 1 1
341
- 333 75.0751 -3.869663 -2.023973 2 2
342
- 334 75.1497 -4.200250 -5.881537 1 1
343
- 335 74.9254 -5.407447 -4.847479 2 1
344
- 336 74.7024 -3.713417 -3.905933 1 2
345
- 337 74.7774 -2.424819 -1.493909 2 2
346
- 338 74.5562 -3.279458 -2.991822 2 1
347
- 339 74.3363 -5.555740 -5.944225 1 2
348
- 340 74.4118 -4.743134 -5.514631 1 1
349
- 341 74.4868 -3.862589 -5.225331 1 1
350
- 342 74.5614 -5.290017 -4.666895 2 2
351
- 343 74.6356 -3.112321 -3.721474 1 1
352
- 344 74.7093 -1.976001 -3.170472 1 1
353
- 345 74.7826 -0.961024 -3.658190 1 1
354
- 346 74.8555 -6.073458 -5.895138 2 2
355
- 347 74.9280 -3.885418 -3.424571 2 2
356
- 348 75.0000 -0.749392 -1.753700 1 1
357
- 349 74.7851 -3.321259 -4.162577 1 2
358
- 350 74.5714 -1.942024 -2.717645 1 2
359
- 351 74.6439 -2.778664 -3.482575 1 1
360
- 352 74.7159 -3.593745 -2.984382 2 2
361
- 353 74.7875 -2.401397 -2.672335 1 1
362
- 354 74.8588 -5.572083 -3.855017 2 2
363
- 355 74.9296 -2.739659 -2.344515 2 2
364
- 356 74.7191 -3.292941 -3.535381 1 2
365
- 357 74.7899 -4.690139 -4.296035 2 2
366
- 358 74.8603 -2.357273 -2.088365 2 2
367
- 359 74.9304 -5.058109 -2.923642 2 2
368
- 360 75.0000 -3.825522 -3.632410 2 2
369
- 361 75.0693 -3.311461 -1.724215 2 2
370
- 362 75.1381 -5.628792 -5.231765 2 2
371
- 363 74.9311 -2.890128 -2.629011 2 1
372
- 364 75.0000 -5.759220 -5.986688 1 1
373
- 365 75.0685 -5.429910 -5.493495 1 1
374
- 366 74.8634 -2.775193 -7.446714 1 2
375
- 367 74.9319 -4.925971 -5.152369 1 1
376
- 368 75.0000 -4.097889 -3.902129 2 2
377
- 369 74.7967 -4.073771 -4.241682 1 2
378
- 370 74.5946 -5.169896 -4.752865 2 1
379
- 371 74.6631 -2.194654 -3.152217 1 1
380
- 372 74.7312 -2.412963 -2.752394 1 1
381
- 373 74.5308 -4.338666 -3.367977 2 1
382
- 374 74.5989 -2.342344 -2.667943 1 1
383
- 375 74.6667 -3.892800 -3.155823 2 2
384
- 376 74.7340 -4.635585 -3.574522 2 2
385
- 377 74.8011 -3.717031 -3.204594 2 2
386
- 378 74.6032 -3.123170 -3.403888 1 2
387
- 379 74.4063 -0.643810 -2.413502 1 2
388
- 380 74.4737 -5.640286 -6.300222 1 1
389
- 381 74.5407 -10.068798 -9.278888 2 2
390
- 382 74.6073 -3.026389 -2.963873 2 2
391
- 383 74.6736 -2.363845 -2.063510 2 2
392
- 384 74.4792 -3.360389 -2.943766 2 1
393
- 385 74.2857 -1.165229 -1.093572 2 1
394
- 386 74.3523 -2.182988 -13.623173 1 1
395
- 387 74.4186 -2.024201 -2.497538 1 1
396
- 388 74.4845 -5.676381 -5.523682 2 2
397
- 389 74.2931 -2.952336 -2.952426 1 2
398
- 390 74.3590 -3.226230 -3.351560 1 1
399
- 391 74.1688 -4.357555 -4.662426 1 2
400
- 392 73.9796 -4.581877 -3.842105 2 1
401
- 393 74.0458 -6.145604 -5.683577 2 2
402
- 394 74.1117 -2.415315 -3.915883 1 1
403
- 395 74.1772 -3.854992 -3.301882 2 2
404
- 396 74.2424 -2.968475 -3.524409 1 1
405
- 397 74.3073 -6.674735 -7.470665 1 1
406
- 398 74.3719 -2.820813 -4.979408 1 1
407
- 399 74.4361 -2.508104 -3.660016 1 1
408
- 400 74.5000 -1.397590 -6.655550 1 1
409
- 401 74.5636 -3.494135 -3.938664 1 1
410
- 402 74.6269 -3.562679 -3.491322 2 2
411
- 403 74.4417 -5.622849 -6.181046 1 2
412
- 404 74.5050 -4.199457 -3.349139 2 2
413
- 405 74.5679 -3.571401 -4.490305 1 1
414
- 406 74.6305 -7.616042 -8.512033 1 1
415
- 407 74.4472 -4.780979 -4.877293 1 2
416
- 408 74.2647 -3.355747 -3.507684 1 2
417
- 409 74.3276 -4.023485 -4.663177 1 1
418
- 410 74.3902 -3.671190 -3.949585 1 1
419
- 411 74.4526 -3.863642 -5.203405 1 1
420
- 412 74.5146 -3.910854 -3.715435 2 2
421
- 413 74.5763 -3.529945 -2.755156 2 2
422
- 414 74.3961 -4.070851 -4.369953 1 2
423
- 415 74.4578 -3.700975 -3.415781 2 2
424
- 416 74.5192 -4.329809 -1.961706 2 2
425
- 417 74.5803 -4.042063 -5.009366 1 1
426
- 418 74.6411 -4.941821 -5.193530 1 1
427
- 419 74.7017 -4.095130 -3.840902 2 2
428
- 420 74.5238 -2.966457 -3.231584 1 2
429
- 421 74.5843 -4.202862 -3.927149 2 2
430
- 422 74.6445 -2.180738 -2.322403 1 1
431
- 423 74.4681 -3.563339 -3.453216 2 1
432
- 424 74.5283 -4.932109 -5.657740 1 1
433
- 425 74.5882 -5.152693 -3.205211 2 2
434
- 426 74.6479 -2.064699 -1.890589 2 2
435
- 427 74.7073 -3.744425 -4.419004 1 1
436
- 428 74.5327 -4.211808 -4.092133 2 1
437
- 429 74.3590 -3.123073 -3.125029 1 2
438
- 430 74.4186 -5.017125 -5.564210 1 1
439
- 431 74.4780 -3.788020 -5.436461 1 1
440
- 432 74.3056 -3.325112 -3.916176 1 2
441
- 433 74.3649 -2.407068 -1.799452 2 2
442
- 434 74.4240 -7.098004 -7.699830 1 1
443
- 435 74.4828 -3.625166 -3.744595 1 1
444
- 436 74.5413 -2.178165 -2.293220 1 1
445
- 437 74.5995 -4.403235 -3.874478 2 2
446
- 438 74.6575 -3.847316 -3.735227 2 2
447
- 439 74.7153 -2.894490 -3.226710 1 1
448
- 440 74.7727 -3.608408 -4.055033 1 1
449
- 441 74.8299 -3.647294 -4.611561 1 1
450
- 442 74.8869 -4.222510 -4.299705 1 1
451
- 443 74.9436 -1.925195 -3.330413 1 1
452
- 444 75.0000 -5.857247 -5.711429 2 2
453
- 445 75.0562 -1.624226 -2.265275 1 1
454
- 446 75.1121 -4.234089 -2.676817 2 2
455
- 447 74.9441 -3.262027 -2.830692 2 1
456
- 448 75.0000 -4.798672 -4.537140 2 2
457
- 449 75.0557 -2.475442 -2.106851 2 2
458
- 450 75.1111 -6.144877 -5.083982 2 2
459
- 451 75.1663 -4.036001 -3.544187 2 2
460
- 452 75.0000 -4.442327 -4.093553 2 1
461
- 453 75.0552 -4.347972 -3.687747 2 2
462
- 454 75.1101 -3.597656 -3.952366 1 1
463
- 455 74.9451 -1.925199 -10.485233 1 2
464
- 456 74.7807 -4.291330 -4.300808 1 2
465
- 457 74.6171 -8.138121 -7.907802 2 1
466
- 458 74.6725 -3.615497 -2.846944 2 2
467
- 459 74.7277 -4.020659 -3.929885 2 2
468
- 460 74.7826 -0.706159 -9.449612 1 1
469
- 461 74.8373 -2.733495 -3.597960 1 1
470
- 462 74.6753 -5.497791 -5.230321 2 1
471
- 463 74.7300 -1.430749 -1.690695 1 1
472
- 464 74.7845 -2.139597 -2.497597 1 1
473
- 465 74.8387 -3.104791 -2.915266 2 2
474
- 466 74.6781 -5.536346 -5.547518 1 2
475
- 467 74.5182 -4.858192 -4.717556 2 1
476
- 468 74.5726 -4.311841 -3.060905 2 2
477
- 469 74.4136 -1.850627 -1.607777 2 1
478
- 470 74.4681 -2.016903 -1.795577 2 2
479
- 471 74.5223 -2.642876 -3.068224 1 1
480
- 472 74.5763 -3.394247 -3.366230 2 2
481
- 473 74.6300 -4.117832 -3.232960 2 2
482
- 474 74.6835 -4.207540 -5.519671 1 1
483
- 475 74.7368 -2.061728 -1.836252 2 2
484
- 476 74.5798 -4.004561 -4.333869 1 2
485
- 477 74.4235 -4.317048 -4.800623 1 2
486
- 478 74.4770 -2.593099 -2.681657 1 1
487
- 479 74.3215 -3.116304 -3.097697 2 1
488
- 480 74.3750 -6.029042 -5.759176 2 2
489
- 481 74.4283 -2.186204 -2.522891 1 1
490
- 482 74.2739 -4.318773 -4.953063 1 2
491
- 483 74.3271 -4.505262 -4.992292 1 1
492
- 484 74.1736 -2.809197 -2.354123 2 1
493
- 485 74.2268 -1.438840 -2.090367 1 1
494
- 486 74.2798 -2.851235 -3.353595 1 1
495
- 487 74.3326 -3.779319 -3.215051 2 2
496
- 488 74.3852 -4.461856 -3.369376 2 2
497
- 489 74.4376 -2.362956 -2.863886 1 1
498
- 490 74.4898 -2.778020 -3.201678 1 1
499
- 491 74.5418 -5.541337 -4.463844 2 2
500
- 492 74.5935 -5.145911 -4.282646 2 2
501
- 493 74.6450 -1.477142 -2.749208 1 1
502
- 494 74.6964 -3.635319 -2.924104 2 2
503
- 495 74.5455 -3.949721 -3.930644 2 1
504
- 496 74.5968 -3.302192 -3.159831 2 2
505
- 497 74.6479 -4.990888 -5.096939 1 1
506
- 498 74.4980 -1.979564 -2.152777 1 2
507
- 499 74.5491 -3.034977 -3.520262 1 1
508
- 500 74.6000 -5.481223 -4.822967 2 2
509
- 501 74.6507 -2.462127 -3.085704 1 1
510
- 502 74.7012 -1.485708 -2.555435 1 1
511
- 503 74.7515 -3.725702 -3.335722 2 2
512
- 504 74.8016 -2.624752 -2.801450 1 1
513
- 505 74.6535 -7.203042 -7.378334 1 2
514
- 506 74.7036 -4.109410 -4.960081 1 1
515
- 507 74.7535 -3.203793 -3.470226 1 1
516
- 508 74.8031 -4.568336 -3.861395 2 2
517
- 509 74.8527 -5.927052 -9.077837 1 1
518
- 510 74.9020 -3.791780 -4.283448 1 1
519
- 511 74.9511 -5.841024 -4.459714 2 2
520
- 512 75.0000 -3.377311 -3.526109 1 1
521
- 513 75.0487 -3.024160 -2.997895 2 2
522
- 514 74.9027 -4.831834 -3.844434 2 1
523
- 515 74.9515 -5.883479 -8.243164 1 1
524
- 516 75.0000 -4.313364 -4.572339 1 1
525
- 517 75.0484 -1.736265 -2.233045 1 1
526
- 518 75.0965 -2.735844 -2.711997 2 2
527
- 519 74.9518 -5.797417 -6.211676 1 2
528
- 520 75.0000 -2.628856 -2.547703 2 2
529
- 521 75.0480 -2.207117 -2.215101 1 1
530
- 522 75.0958 -3.967665 -2.624802 2 2
531
- 523 75.1434 -3.774526 -3.130207 2 2
532
- 524 75.1908 -4.124199 -4.603506 1 1
533
- 525 75.2381 -4.027994 -1.936259 2 2
534
- 526 75.2852 -0.684569 -14.584228 1 1
535
- 527 75.3321 -6.149704 -4.025888 2 2
536
- 528 75.3788 -6.345269 -5.961727 2 2
537
- 529 75.4253 -5.099360 -5.710513 1 1
538
- 530 75.4717 -4.605446 -2.929898 2 2
539
- 531 75.5179 -3.596594 -3.831368 1 1
540
- 532 75.5639 -2.045439 -3.137911 1 1
541
- 533 75.6098 -4.039925 -2.668880 2 2
542
- 534 75.6554 -2.362336 -2.862948 1 1
543
- 535 75.5140 -3.672946 -3.772023 1 2
544
- 536 75.3731 -2.414235 -2.203349 2 1
545
- 537 75.4190 -5.150235 -6.013234 1 1
546
- 538 75.4647 -1.818094 -4.967626 1 1
547
- 539 75.5102 -3.742584 -1.825052 2 2
548
- 540 75.5556 -2.897302 -2.711947 2 2
549
- 541 75.6007 -2.664831 -3.252761 1 1
550
- 542 75.4613 -3.337824 -4.692550 1 2
551
- 543 75.5064 -5.645246 -4.055596 2 2
552
- 544 75.5515 -3.053798 -3.789311 1 1
553
- 545 75.4128 -2.256422 -4.145020 1 2
554
- 546 75.4579 -2.724989 -3.946668 1 1
555
- 547 75.5027 -5.853046 -4.667560 2 2
556
- 548 75.5474 -4.725450 -4.753620 1 1
557
- 549 75.5920 -5.213475 -5.544143 1 1
558
- 550 75.6364 -2.738129 -2.634528 2 2
559
- 551 75.4991 -1.018525 -12.086659 1 2
560
- 552 75.5435 -2.377233 -2.384735 1 1
561
- 553 75.5877 -5.150899 -4.994988 2 2
562
- 554 75.6318 -4.818916 -4.704148 2 2
563
- 555 75.6757 -2.010890 -1.615438 2 2
564
- 556 75.7194 -5.339697 -5.160274 2 2
565
- 557 75.5835 -4.738886 -4.197691 2 1
566
- 558 75.6272 -1.875037 -3.121289 1 1
567
- 559 75.6708 -2.526915 -2.469661 2 2
568
- 560 75.5357 -2.663487 -2.567836 2 1
569
- 561 75.5793 -2.031263 -2.664051 1 1
570
- 562 75.6228 -0.622257 -1.691906 1 1
571
- 563 75.4885 -1.563125 -2.502928 1 2
572
- 564 75.5319 -2.844138 -3.208428 1 1
573
- 565 75.5752 -4.832521 -4.401260 2 2
574
- 566 75.6184 -4.694997 -5.010533 1 1
575
- 567 75.6614 -2.091695 -1.737015 2 2
576
- 568 75.7042 -7.576469 -2.953415 2 2
577
- 569 75.5712 -5.359783 -4.597651 2 1
578
- 570 75.6140 -4.103200 -5.580116 1 1
579
- 571 75.6567 -4.276189 -3.055103 2 2
580
- 572 75.6993 -2.706305 -2.232418 2 2
581
- 573 75.5672 -3.927538 -4.158408 1 2
582
- 574 75.6098 -3.990731 -3.608039 2 2
583
- 575 75.6522 -5.365208 -5.315892 2 2
584
- 576 75.6944 -3.498107 -4.164121 1 1
585
- 577 75.7366 -4.075049 -3.702802 2 2
586
- 578 75.7785 -2.970797 -2.809494 2 2
587
- 579 75.6477 -4.175112 -4.626372 1 2
588
- 580 75.6897 -2.159041 -2.614806 1 1
589
- 581 75.5594 -6.300266 -6.042942 2 1
590
- 582 75.4296 -5.307058 -5.443231 1 2
591
- 583 75.4717 -5.052415 -4.671195 2 2
592
- 584 75.3425 -4.450569 -4.120255 2 1
593
- 585 75.3846 -0.451843 -1.417040 1 1
594
- 586 75.4266 -3.643149 -3.656776 1 1
595
- 587 75.2981 -5.181589 -5.026025 2 1
596
- 588 75.3401 -3.987187 -2.860708 2 2
597
- 589 75.2122 -3.906131 -4.079319 1 2
598
- 590 75.2542 -4.194465 -3.414926 2 2
599
- 591 75.2961 -3.898033 -3.621221 2 2
600
- 592 75.3378 -2.979693 -3.839550 1 1
601
- 593 75.3794 -3.820930 -3.166747 2 2
602
- 594 75.2525 -3.991677 -3.803577 2 1
603
- 595 75.1261 -1.636322 -1.684349 1 2
604
- 596 75.1678 -5.316608 -5.177227 2 2
605
- 597 75.2094 -1.879422 -2.089728 1 1
606
- 598 75.2508 -2.936712 -3.743055 1 1
607
- 599 75.2922 -3.051193 -3.120804 1 1
608
- 600 75.3333 -4.212932 -3.982070 2 2
609
- 601 75.3744 -3.297436 -3.151707 2 2
610
- 602 75.4153 -2.164534 -3.906807 1 1
611
- 603 75.4561 -5.729845 -5.682674 2 2
612
- 604 75.3311 -3.104517 -3.246167 1 2
613
- 605 75.2066 -2.658667 -2.463872 2 1
614
- 606 75.2475 -3.281253 -2.428075 2 2
615
- 607 75.1236 -2.389994 -1.958902 2 1
616
- 608 75.1645 -2.419189 -2.746315 1 1
617
- 609 75.2053 -5.561723 -6.412402 1 1
618
- 610 75.0820 -2.776464 -2.658938 2 1
619
- 611 75.1227 -4.896749 -4.960549 1 1
620
- 612 75.1634 -3.826475 -5.497705 1 1
621
- 613 75.2039 -2.550734 -2.457320 2 2
622
- 614 75.2443 -2.871862 -1.809686 2 2
623
- 615 75.2846 -2.589114 -2.523511 2 2
624
- 616 75.3247 -3.508700 -3.365285 2 2
625
- 617 75.3647 -11.402687 -3.376035 2 2
626
- 618 75.2427 -3.914923 -4.151593 1 2
627
- 619 75.2827 -3.344285 -2.704345 2 2
628
- 620 75.3226 -4.519379 -4.802140 1 1
629
- 621 75.3623 -3.777889 -3.079480 2 2
630
- 622 75.4019 -2.425511 -2.511834 1 1
631
- 623 75.2809 -3.003730 -2.766911 2 1
632
- 624 75.3205 -4.424657 -3.800578 2 2
633
- 625 75.3600 -3.536587 -5.016929 1 1
634
- 626 75.3994 -2.569373 -2.968881 1 1
635
- 627 75.2791 -3.827440 -3.723844 2 1
636
- 628 75.3185 -2.532845 -2.292835 2 2
637
- 629 75.3577 -1.704787 -2.340892 1 1
638
- 630 75.3968 -1.270356 -1.900736 1 1
639
- 631 75.4358 -1.465610 -2.765382 1 1
640
- 632 75.4747 -1.798835 -1.321187 2 2
641
- 633 75.5134 -2.969589 -2.414291 2 2
642
- 634 75.5521 -2.177664 -2.833552 1 1
643
- 635 75.4331 -4.135893 -4.474928 1 2
644
- 636 75.4717 -2.543812 -4.186953 1 1
645
- 637 75.5102 -3.114409 -5.335104 1 1
646
- 638 75.5486 -4.820964 -5.239189 1 1
647
- 639 75.5869 -1.433183 -1.718281 1 1
648
- 640 75.4688 -1.336847 -2.054662 1 2
649
- 641 75.5070 -3.910844 -4.868577 1 1
650
- 642 75.5452 -4.328723 -2.479299 2 2
651
- 643 75.5832 -0.885112 -0.549491 2 2
652
- 644 75.6211 -4.714597 -4.999483 1 1
653
- 645 75.6589 -1.419659 -2.281091 1 1
654
- 646 75.5418 -4.256945 -3.998688 2 1
655
- 647 75.5796 -4.274268 -4.460077 1 1
656
- 648 75.6173 -3.308492 -3.188788 2 2
657
- 649 75.6549 -2.386477 -2.986150 1 1
658
- 650 75.6923 -3.163456 -3.875068 1 1
659
- 651 75.7296 -5.474017 -5.444322 2 2
660
- 652 75.7669 -5.288219 -5.122352 2 2
661
- 653 75.8040 -6.528839 -5.690201 2 2
662
- 654 75.6881 -3.351024 -3.544878 1 2
663
- 655 75.7252 -2.712922 -2.584755 2 2
664
- 656 75.7622 -3.764951 -1.733579 2 2
665
- 657 75.6469 -5.204409 -5.661041 1 2
666
- 658 75.6839 -5.366095 -3.781115 2 2
667
- 659 75.7208 -2.265442 -1.914578 2 2
668
- 660 75.7576 -3.520226 -2.335560 2 2
669
- 661 75.7943 -2.631767 -3.016934 1 1
670
- 662 75.8308 -3.595957 -3.755254 1 1
671
- 663 75.8673 -6.413270 -4.928877 2 2
672
- 664 75.9036 -2.867244 -2.927661 1 1
673
- 665 75.9398 -3.887688 -3.501133 2 2
674
- 666 75.9760 -5.003187 -5.135930 1 1
675
- 667 75.8621 -4.752621 -9.615022 1 2
676
- 668 75.8982 -2.790062 -3.399313 1 1
677
- 669 75.9342 -3.273346 -3.633488 1 1
678
- 670 75.8209 -3.487094 -3.080981 2 1
679
- 671 75.8569 -3.723062 -4.722219 1 1
680
- 672 75.8929 -2.843087 -2.645352 2 2
681
- 673 75.7801 -1.813552 -1.847092 1 2
682
- 674 75.8160 -6.012554 -6.130527 1 1
683
- 675 75.7037 -4.712497 -4.687178 2 1
684
- 676 75.5917 -1.994161 -2.454943 1 2
685
- 677 75.6278 -3.470867 -4.110757 1 1
686
- 678 75.6637 -5.092277 -5.027576 2 2
687
- 679 75.6996 -5.437482 -4.886851 2 2
688
- 680 75.7353 -2.948081 -4.916172 1 1
689
- 681 75.6241 -3.659360 -2.929616 2 1
690
- 682 75.5132 -2.136437 -2.318466 1 2
691
- 683 75.5490 -1.655803 -2.765518 1 1
692
- 684 75.5848 -4.513469 -8.008779 1 1
693
- 685 75.6204 -4.121778 -1.702858 2 2
694
- 686 75.6560 -5.324989 -5.272547 2 2
695
- 687 75.6914 -2.686729 -1.891066 2 2
696
- 688 75.7267 -6.972888 -7.646428 1 1
697
- 689 75.7620 -2.853402 -2.498782 2 2
698
- 690 75.6522 -2.467654 -2.474524 1 2
699
- 691 75.5427 -3.519656 -3.161650 2 1
700
- 692 75.5780 -2.004331 -2.890412 1 1
701
- 693 75.6133 -4.723198 -4.848408 1 1
702
- 694 75.6484 -3.364899 -3.123396 2 2
703
- 695 75.6835 -4.822385 -5.896009 1 1
704
- 696 75.5747 -4.902782 -4.736121 2 1
705
- 697 75.6098 -2.662646 -2.825220 1 1
706
- 698 75.6447 -4.326837 -3.966126 2 2
707
- 699 75.6795 -3.857300 -5.670707 1 1
708
- 700 75.7143 -3.501438 -2.673927 2 2
709
- 701 75.6063 -2.050769 -1.626799 2 1
710
- 702 75.4986 -4.376371 -4.914997 1 2
711
- 703 75.3912 -4.750635 -3.433849 2 1
712
- 704 75.2841 -2.051587 -1.919922 2 1
713
- 705 75.3191 -2.629596 -2.202023 2 2
714
- 706 75.3541 -2.881210 -2.648880 2 2
715
- 707 75.2475 -5.045852 -4.905766 2 1
716
- 708 75.2825 -3.822015 -3.541684 2 2
717
- 709 75.3173 -3.617289 -2.788389 2 2
718
- 710 75.3521 -2.745065 -3.851634 1 1
719
- 711 75.3868 -2.771745 -2.208492 2 2
720
- 712 75.2809 -2.972170 -4.048257 1 2
721
- 713 75.1753 -3.592868 -3.708959 1 2
722
- 714 75.2101 -3.148624 -2.759390 2 2
723
- 715 75.2448 -5.535668 -4.008753 2 2
724
- 716 75.2793 -4.673175 -3.577783 2 2
725
- 717 75.3138 -2.450007 -0.901205 2 2
726
- 718 75.3482 -3.466274 -3.893165 1 1
727
- 719 75.2434 -4.985331 -4.764683 2 1
728
- 720 75.1389 -0.865437 -4.601235 1 2
729
- 721 75.0347 -5.211213 -5.236653 1 2
730
- 722 75.0693 -2.372424 -2.461527 1 1
731
- 723 75.1037 -2.565556 -2.871064 1 1
732
- 724 75.0000 -3.357473 -3.636106 1 2
733
- 725 74.8966 -3.274843 -3.480422 1 2
734
- 726 74.7934 -4.234673 -4.248022 1 2
735
- 727 74.8281 -3.296737 -2.656152 2 2
736
- 728 74.7253 -4.599780 -4.479354 2 1
737
- 729 74.7599 -2.025480 -2.938724 1 1
738
- 730 74.6575 -3.411519 -3.248773 2 1
739
- 731 74.6922 -3.435143 -4.325244 1 1
740
- 732 74.7268 -3.254907 -2.411543 2 2
741
- 733 74.6248 -3.761533 -3.953408 1 2
742
- 734 74.6594 -0.112278 -7.793087 1 1
743
- 735 74.6939 -2.787906 -2.817590 1 1
744
- 736 74.7283 -3.204641 -3.657746 1 1
745
- 737 74.6269 -4.095687 -3.696146 2 1
746
- 738 74.6612 -4.614668 -4.432005 2 2
747
- 739 74.6955 -0.386722 -9.220013 1 1
748
- 740 74.7297 -4.550553 -3.602751 2 2
749
- 741 74.6289 -4.433225 -4.269165 2 1
750
- 742 74.6631 -2.320354 -1.598068 2 2
751
- 743 74.5626 -3.654797 -3.655552 1 2
752
- 744 74.5968 -5.188527 -4.211945 2 2
753
- 745 74.6309 -4.109009 -4.858491 1 1
754
- 746 74.6649 -5.117740 -5.339579 1 1
755
- 747 74.5649 -5.658653 -5.006263 2 1
756
- 748 74.5989 -3.570308 -3.434863 2 2
757
- 749 74.6328 -3.302384 -3.768939 1 1
758
- 750 74.6667 -1.428684 -9.071774 1 1
759
 
760
- Final Winogrande score(750 tasks): 74.6667 +/- 1.5892
 
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-IQ3_S.gguf (version GGUF V3 (latest))
4
 
5
+ Final Winogrande score(750 tasks): 70.2667 +/- 1.6702
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
 
7
+ llama_perf_context_print: load time = 277.86 ms
8
+ llama_perf_context_print: prompt eval time = 88618.25 ms / 22199 tokens ( 3.99 ms per token, 250.50 tokens per second)
9
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
10
+ llama_perf_context_print: total time = 89806.41 ms / 22200 tokens
11
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-iq4_nl.arc CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 869 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 869 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 100.00000000
13
- 3 100.00000000
14
- 4 75.00000000
15
- 5 80.00000000
16
- 6 83.33333333
17
- 7 71.42857143
18
- 8 75.00000000
19
- 9 66.66666667
20
- 10 70.00000000
21
- 11 63.63636364
22
- 12 66.66666667
23
- 13 69.23076923
24
- 14 71.42857143
25
- 15 73.33333333
26
- 16 75.00000000
27
- 17 76.47058824
28
- 18 72.22222222
29
- 19 73.68421053
30
- 20 70.00000000
31
- 21 71.42857143
32
- 22 72.72727273
33
- 23 69.56521739
34
- 24 70.83333333
35
- 25 68.00000000
36
- 26 65.38461538
37
- 27 66.66666667
38
- 28 64.28571429
39
- 29 65.51724138
40
- 30 63.33333333
41
- 31 61.29032258
42
- 32 59.37500000
43
- 33 57.57575758
44
- 34 55.88235294
45
- 35 57.14285714
46
- 36 55.55555556
47
- 37 56.75675676
48
- 38 55.26315789
49
- 39 56.41025641
50
- 40 57.50000000
51
- 41 58.53658537
52
- 42 57.14285714
53
- 43 58.13953488
54
- 44 56.81818182
55
- 45 57.77777778
56
- 46 58.69565217
57
- 47 59.57446809
58
- 48 60.41666667
59
- 49 61.22448980
60
- 50 60.00000000
61
- 51 60.78431373
62
- 52 61.53846154
63
- 53 60.37735849
64
- 54 59.25925926
65
- 55 58.18181818
66
- 56 58.92857143
67
- 57 59.64912281
68
- 58 60.34482759
69
- 59 61.01694915
70
- 60 61.66666667
71
- 61 62.29508197
72
- 62 61.29032258
73
- 63 61.90476190
74
- 64 62.50000000
75
- 65 61.53846154
76
- 66 62.12121212
77
- 67 62.68656716
78
- 68 63.23529412
79
- 69 63.76811594
80
- 70 62.85714286
81
- 71 61.97183099
82
- 72 61.11111111
83
- 73 61.64383562
84
- 74 62.16216216
85
- 75 62.66666667
86
- 76 61.84210526
87
- 77 62.33766234
88
- 78 62.82051282
89
- 79 62.02531646
90
- 80 62.50000000
91
- 81 61.72839506
92
- 82 62.19512195
93
- 83 61.44578313
94
- 84 61.90476190
95
- 85 62.35294118
96
- 86 62.79069767
97
- 87 63.21839080
98
- 88 63.63636364
99
- 89 64.04494382
100
- 90 64.44444444
101
- 91 64.83516484
102
- 92 65.21739130
103
- 93 64.51612903
104
- 94 63.82978723
105
- 95 64.21052632
106
- 96 64.58333333
107
- 97 63.91752577
108
- 98 64.28571429
109
- 99 64.64646465
110
- 100 65.00000000
111
- 101 64.35643564
112
- 102 64.70588235
113
- 103 64.07766990
114
- 104 63.46153846
115
- 105 63.80952381
116
- 106 63.20754717
117
- 107 63.55140187
118
- 108 62.96296296
119
- 109 62.38532110
120
- 110 62.72727273
121
- 111 63.06306306
122
- 112 63.39285714
123
- 113 63.71681416
124
- 114 63.15789474
125
- 115 63.47826087
126
- 116 63.79310345
127
- 117 63.24786325
128
- 118 63.55932203
129
- 119 63.02521008
130
- 120 63.33333333
131
- 121 62.80991736
132
- 122 63.11475410
133
- 123 63.41463415
134
- 124 62.90322581
135
- 125 63.20000000
136
- 126 63.49206349
137
- 127 63.77952756
138
- 128 64.06250000
139
- 129 63.56589147
140
- 130 63.84615385
141
- 131 64.12213740
142
- 132 64.39393939
143
- 133 64.66165414
144
- 134 64.17910448
145
- 135 63.70370370
146
- 136 63.23529412
147
- 137 63.50364964
148
- 138 63.76811594
149
- 139 64.02877698
150
- 140 64.28571429
151
- 141 63.82978723
152
- 142 64.08450704
153
- 143 64.33566434
154
- 144 64.58333333
155
- 145 64.82758621
156
- 146 64.38356164
157
- 147 64.62585034
158
- 148 64.86486486
159
- 149 65.10067114
160
- 150 65.33333333
161
- 151 64.90066225
162
- 152 65.13157895
163
- 153 65.35947712
164
- 154 64.93506494
165
- 155 65.16129032
166
- 156 65.38461538
167
- 157 65.60509554
168
- 158 65.18987342
169
- 159 65.40880503
170
- 160 65.00000000
171
- 161 65.21739130
172
- 162 65.43209877
173
- 163 65.64417178
174
- 164 65.85365854
175
- 165 65.45454545
176
- 166 65.66265060
177
- 167 65.26946108
178
- 168 65.47619048
179
- 169 65.08875740
180
- 170 65.29411765
181
- 171 65.49707602
182
- 172 65.69767442
183
- 173 65.89595376
184
- 174 66.09195402
185
- 175 65.71428571
186
- 176 65.90909091
187
- 177 66.10169492
188
- 178 66.29213483
189
- 179 66.48044693
190
- 180 66.66666667
191
- 181 66.29834254
192
- 182 66.48351648
193
- 183 66.12021858
194
- 184 66.30434783
195
- 185 66.48648649
196
- 186 66.66666667
197
- 187 66.31016043
198
- 188 65.95744681
199
- 189 65.60846561
200
- 190 65.26315789
201
- 191 65.44502618
202
- 192 65.62500000
203
- 193 65.28497409
204
- 194 65.46391753
205
- 195 65.64102564
206
- 196 65.81632653
207
- 197 65.98984772
208
- 198 65.65656566
209
- 199 65.32663317
210
- 200 65.50000000
211
- 201 65.67164179
212
- 202 65.34653465
213
- 203 65.51724138
214
- 204 65.68627451
215
- 205 65.36585366
216
- 206 65.04854369
217
- 207 65.21739130
218
- 208 65.38461538
219
- 209 65.55023923
220
- 210 65.71428571
221
- 211 65.87677725
222
- 212 65.56603774
223
- 213 65.72769953
224
- 214 65.88785047
225
- 215 66.04651163
226
- 216 65.74074074
227
- 217 65.89861751
228
- 218 65.59633028
229
- 219 65.75342466
230
- 220 65.90909091
231
- 221 66.06334842
232
- 222 66.21621622
233
- 223 65.91928251
234
- 224 66.07142857
235
- 225 65.77777778
236
- 226 65.92920354
237
- 227 66.07929515
238
- 228 66.22807018
239
- 229 66.37554585
240
- 230 66.52173913
241
- 231 66.23376623
242
- 232 65.94827586
243
- 233 66.09442060
244
- 234 66.23931624
245
- 235 65.95744681
246
- 236 65.67796610
247
- 237 65.82278481
248
- 238 65.96638655
249
- 239 66.10878661
250
- 240 66.25000000
251
- 241 66.39004149
252
- 242 66.52892562
253
- 243 66.25514403
254
- 244 65.98360656
255
- 245 66.12244898
256
- 246 66.26016260
257
- 247 66.39676113
258
- 248 66.53225806
259
- 249 66.26506024
260
- 250 66.40000000
261
- 251 66.53386454
262
- 252 66.66666667
263
- 253 66.79841897
264
- 254 66.53543307
265
- 255 66.66666667
266
- 256 66.40625000
267
- 257 66.14785992
268
- 258 66.27906977
269
- 259 66.02316602
270
- 260 66.15384615
271
- 261 66.28352490
272
- 262 66.41221374
273
- 263 66.53992395
274
- 264 66.66666667
275
- 265 66.79245283
276
- 266 66.54135338
277
- 267 66.66666667
278
- 268 66.79104478
279
- 269 66.91449814
280
- 270 67.03703704
281
- 271 67.15867159
282
- 272 66.91176471
283
- 273 66.66666667
284
- 274 66.42335766
285
- 275 66.18181818
286
- 276 66.30434783
287
- 277 66.42599278
288
- 278 66.18705036
289
- 279 66.30824373
290
- 280 66.42857143
291
- 281 66.54804270
292
- 282 66.66666667
293
- 283 66.43109541
294
- 284 66.54929577
295
- 285 66.66666667
296
- 286 66.78321678
297
- 287 66.89895470
298
- 288 67.01388889
299
- 289 66.78200692
300
- 290 66.89655172
301
- 291 67.01030928
302
- 292 67.12328767
303
- 293 67.23549488
304
- 294 67.00680272
305
- 295 67.11864407
306
- 296 67.22972973
307
- 297 67.34006734
308
- 298 67.44966443
309
- 299 67.55852843
310
- 300 67.66666667
311
- 301 67.44186047
312
- 302 67.54966887
313
- 303 67.65676568
314
- 304 67.76315789
315
- 305 67.86885246
316
- 306 67.64705882
317
- 307 67.75244300
318
- 308 67.85714286
319
- 309 67.96116505
320
- 310 68.06451613
321
- 311 68.16720257
322
- 312 68.26923077
323
- 313 68.37060703
324
- 314 68.15286624
325
- 315 68.25396825
326
- 316 68.35443038
327
- 317 68.45425868
328
- 318 68.23899371
329
- 319 68.02507837
330
- 320 67.81250000
331
- 321 67.91277259
332
- 322 68.01242236
333
- 323 68.11145511
334
- 324 67.90123457
335
- 325 68.00000000
336
- 326 68.09815951
337
- 327 67.88990826
338
- 328 67.98780488
339
- 329 68.08510638
340
- 330 67.87878788
341
- 331 67.97583082
342
- 332 68.07228916
343
- 333 68.16816817
344
- 334 68.26347305
345
- 335 68.05970149
346
- 336 68.15476190
347
- 337 68.24925816
348
- 338 68.04733728
349
- 339 68.14159292
350
- 340 68.23529412
351
- 341 68.32844575
352
- 342 68.42105263
353
- 343 68.51311953
354
- 344 68.60465116
355
- 345 68.40579710
356
- 346 68.49710983
357
- 347 68.29971182
358
- 348 68.10344828
359
- 349 68.19484241
360
- 350 68.00000000
361
- 351 68.09116809
362
- 352 68.18181818
363
- 353 68.27195467
364
- 354 68.07909605
365
- 355 67.88732394
366
- 356 67.97752809
367
- 357 67.78711485
368
- 358 67.87709497
369
- 359 67.96657382
370
- 360 68.05555556
371
- 361 68.14404432
372
- 362 68.23204420
373
- 363 68.04407713
374
- 364 67.85714286
375
- 365 67.94520548
376
- 366 67.75956284
377
- 367 67.57493188
378
- 368 67.39130435
379
- 369 67.47967480
380
- 370 67.56756757
381
- 371 67.65498652
382
- 372 67.74193548
383
- 373 67.82841823
384
- 374 67.91443850
385
- 375 67.73333333
386
- 376 67.81914894
387
- 377 67.63925729
388
- 378 67.72486772
389
- 379 67.54617414
390
- 380 67.63157895
391
- 381 67.45406824
392
- 382 67.27748691
393
- 383 67.10182768
394
- 384 67.18750000
395
- 385 67.01298701
396
- 386 67.09844560
397
- 387 67.18346253
398
- 388 67.01030928
399
- 389 67.09511568
400
- 390 66.92307692
401
- 391 66.75191816
402
- 392 66.83673469
403
- 393 66.92111959
404
- 394 67.00507614
405
- 395 67.08860759
406
- 396 66.91919192
407
- 397 67.00251889
408
- 398 66.83417085
409
- 399 66.91729323
410
- 400 66.75000000
411
- 401 66.83291771
412
- 402 66.66666667
413
- 403 66.74937965
414
- 404 66.83168317
415
- 405 66.91358025
416
- 406 66.99507389
417
- 407 67.07616708
418
- 408 67.15686275
419
- 409 66.99266504
420
- 410 67.07317073
421
- 411 67.15328467
422
- 412 67.23300971
423
- 413 67.31234867
424
- 414 67.39130435
425
- 415 67.22891566
426
- 416 67.30769231
427
- 417 67.14628297
428
- 418 67.22488038
429
- 419 67.06443914
430
- 420 67.14285714
431
- 421 67.22090261
432
- 422 67.29857820
433
- 423 67.37588652
434
- 424 67.21698113
435
- 425 67.29411765
436
- 426 67.13615023
437
- 427 67.21311475
438
- 428 67.05607477
439
- 429 67.13286713
440
- 430 67.20930233
441
- 431 67.28538283
442
- 432 67.36111111
443
- 433 67.20554273
444
- 434 67.28110599
445
- 435 67.35632184
446
- 436 67.43119266
447
- 437 67.50572082
448
- 438 67.35159817
449
- 439 67.42596811
450
- 440 67.27272727
451
- 441 67.34693878
452
- 442 67.19457014
453
- 443 67.04288939
454
- 444 66.89189189
455
- 445 66.96629213
456
- 446 67.04035874
457
- 447 67.11409396
458
- 448 67.18750000
459
- 449 67.26057906
460
- 450 67.33333333
461
- 451 67.40576497
462
- 452 67.47787611
463
- 453 67.54966887
464
- 454 67.62114537
465
- 455 67.47252747
466
- 456 67.54385965
467
- 457 67.61487965
468
- 458 67.68558952
469
- 459 67.75599129
470
- 460 67.60869565
471
- 461 67.67895879
472
- 462 67.74891775
473
- 463 67.60259179
474
- 464 67.45689655
475
- 465 67.52688172
476
- 466 67.59656652
477
- 467 67.66595289
478
- 468 67.73504274
479
- 469 67.59061834
480
- 470 67.65957447
481
- 471 67.72823779
482
- 472 67.79661017
483
- 473 67.65327696
484
- 474 67.72151899
485
- 475 67.57894737
486
- 476 67.43697479
487
- 477 67.50524109
488
- 478 67.57322176
489
- 479 67.64091858
490
- 480 67.70833333
491
- 481 67.77546778
492
- 482 67.84232365
493
- 483 67.90890269
494
- 484 67.97520661
495
- 485 68.04123711
496
- 486 68.10699588
497
- 487 67.96714579
498
- 488 68.03278689
499
- 489 68.09815951
500
- 490 67.95918367
501
- 491 67.82077393
502
- 492 67.88617886
503
- 493 67.74847870
504
- 494 67.81376518
505
- 495 67.67676768
506
- 496 67.74193548
507
- 497 67.80684105
508
- 498 67.87148594
509
- 499 67.93587174
510
- 500 68.00000000
511
- 501 67.86427146
512
- 502 67.92828685
513
- 503 67.99204771
514
- 504 68.05555556
515
- 505 68.11881188
516
- 506 67.98418972
517
- 507 67.85009862
518
- 508 67.91338583
519
- 509 67.77996071
520
- 510 67.64705882
521
- 511 67.51467710
522
- 512 67.57812500
523
- 513 67.44639376
524
- 514 67.50972763
525
- 515 67.57281553
526
- 516 67.63565891
527
- 517 67.69825919
528
- 518 67.76061776
529
- 519 67.82273603
530
- 520 67.69230769
531
- 521 67.75431862
532
- 522 67.81609195
533
- 523 67.68642447
534
- 524 67.55725191
535
- 525 67.42857143
536
- 526 67.30038023
537
- 527 67.17267552
538
- 528 67.04545455
539
- 529 66.91871456
540
- 530 66.98113208
541
- 531 67.04331450
542
- 532 67.10526316
543
- 533 67.16697936
544
- 534 67.04119850
545
- 535 66.91588785
546
- 536 66.97761194
547
- 537 67.03910615
548
- 538 67.10037175
549
- 539 67.16141002
550
- 540 67.03703704
551
- 541 67.09796673
552
- 542 67.15867159
553
- 543 67.03499079
554
- 544 67.09558824
555
- 545 67.15596330
556
- 546 67.03296703
557
- 547 66.91042048
558
- 548 66.97080292
559
- 549 67.03096539
560
- 550 67.09090909
561
- 551 66.96914701
562
- 552 66.84782609
563
- 553 66.90777577
564
- 554 66.96750903
565
- 555 67.02702703
566
- 556 66.90647482
567
- 557 66.96588869
568
- 558 66.84587814
569
- 559 66.90518784
570
- 560 66.96428571
571
- 561 67.02317291
572
- 562 67.08185053
573
- 563 67.14031972
574
- 564 67.02127660
575
- 565 66.90265487
576
- 566 66.96113074
577
- 567 66.84303351
578
- 568 66.90140845
579
- 569 66.78383128
580
- 570 66.66666667
581
- 571 66.72504378
582
- 572 66.60839161
583
- 573 66.66666667
584
- 574 66.72473868
585
- 575 66.78260870
586
- 576 66.84027778
587
- 577 66.89774697
588
- 578 66.95501730
589
- 579 67.01208981
590
- 580 67.06896552
591
- 581 66.95352840
592
- 582 67.01030928
593
- 583 67.06689537
594
- 584 67.12328767
595
- 585 67.17948718
596
- 586 67.06484642
597
- 587 67.12095400
598
- 588 67.17687075
599
- 589 67.23259762
600
- 590 67.28813559
601
- 591 67.34348562
602
- 592 67.39864865
603
- 593 67.28499157
604
- 594 67.34006734
605
- 595 67.39495798
606
- 596 67.28187919
607
- 597 67.16917923
608
- 598 67.05685619
609
- 599 67.11185309
610
- 600 67.00000000
611
- 601 67.05490849
612
- 602 66.94352159
613
- 603 66.83250415
614
- 604 66.88741722
615
- 605 66.77685950
616
- 606 66.83168317
617
- 607 66.88632619
618
- 608 66.94078947
619
- 609 66.83087028
620
- 610 66.88524590
621
- 611 66.77577741
622
- 612 66.83006536
623
- 613 66.72104405
624
- 614 66.77524430
625
- 615 66.82926829
626
- 616 66.72077922
627
- 617 66.77471637
628
- 618 66.82847896
629
- 619 66.72051696
630
- 620 66.77419355
631
- 621 66.82769726
632
- 622 66.72025723
633
- 623 66.77367576
634
- 624 66.66666667
635
- 625 66.56000000
636
- 626 66.45367412
637
- 627 66.50717703
638
- 628 66.56050955
639
- 629 66.45468998
640
- 630 66.50793651
641
- 631 66.56101426
642
- 632 66.45569620
643
- 633 66.35071090
644
- 634 66.40378549
645
- 635 66.45669291
646
- 636 66.50943396
647
- 637 66.40502355
648
- 638 66.45768025
649
- 639 66.51017214
650
- 640 66.40625000
651
- 641 66.45865835
652
- 642 66.35514019
653
- 643 66.25194401
654
- 644 66.30434783
655
- 645 66.20155039
656
- 646 66.25386997
657
- 647 66.30602782
658
- 648 66.20370370
659
- 649 66.10169492
660
- 650 66.15384615
661
- 651 66.20583717
662
- 652 66.10429448
663
- 653 66.00306279
664
- 654 65.90214067
665
- 655 65.80152672
666
- 656 65.70121951
667
- 657 65.60121766
668
- 658 65.50151976
669
- 659 65.40212443
670
- 660 65.45454545
671
- 661 65.35552194
672
- 662 65.40785498
673
- 663 65.46003017
674
- 664 65.51204819
675
- 665 65.56390977
676
- 666 65.46546547
677
- 667 65.51724138
678
- 668 65.41916168
679
- 669 65.32137519
680
- 670 65.37313433
681
- 671 65.27570790
682
- 672 65.32738095
683
- 673 65.23031204
684
- 674 65.13353116
685
- 675 65.03703704
686
- 676 65.08875740
687
- 677 65.14032496
688
- 678 65.19174041
689
- 679 65.24300442
690
- 680 65.14705882
691
- 681 65.05139501
692
- 682 64.95601173
693
- 683 64.86090776
694
- 684 64.91228070
695
- 685 64.96350365
696
- 686 64.86880466
697
- 687 64.77438137
698
- 688 64.82558140
699
- 689 64.73149492
700
- 690 64.78260870
701
- 691 64.83357453
702
- 692 64.88439306
703
- 693 64.93506494
704
- 694 64.98559078
705
- 695 64.89208633
706
- 696 64.79885057
707
- 697 64.70588235
708
- 698 64.61318052
709
- 699 64.66380544
710
- 700 64.71428571
711
- 701 64.76462197
712
- 702 64.81481481
713
- 703 64.86486486
714
- 704 64.91477273
715
- 705 64.96453901
716
- 706 65.01416431
717
- 707 65.06364922
718
- 708 65.11299435
719
- 709 65.16220028
720
- 710 65.21126761
721
- 711 65.26019691
722
- 712 65.30898876
723
- 713 65.35764376
724
- 714 65.26610644
725
- 715 65.17482517
726
- 716 65.22346369
727
- 717 65.13249651
728
- 718 65.04178273
729
- 719 64.95132128
730
- 720 65.00000000
731
- 721 65.04854369
732
- 722 65.09695291
733
- 723 65.14522822
734
- 724 65.05524862
735
- 725 65.10344828
736
- 726 65.01377410
737
- 727 64.92434663
738
- 728 64.97252747
739
- 729 65.02057613
740
- 730 65.06849315
741
- 731 65.11627907
742
- 732 65.16393443
743
- 733 65.21145975
744
- 734 65.25885559
745
- 735 65.30612245
746
- 736 65.35326087
747
- 737 65.26458616
748
- 738 65.31165312
749
- 739 65.22327470
750
- 740 65.13513514
751
- 741 65.18218623
752
- 742 65.09433962
753
- 743 65.14131898
754
- 744 65.05376344
755
- 745 65.10067114
756
- 746 65.14745308
757
- 747 65.06024096
758
- 748 64.97326203
759
- 749 64.88651535
760
- 750 64.93333333
761
 
762
- Final result: 64.9333 +/- 1.7436
763
- Random chance: 25.0083 +/- 1.5824
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-IQ4_NL.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 64.7925 +/- 1.7487
6
+ Random chance: 25.0251 +/- 1.5859
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 1990.26 ms
10
+ llama_perf_context_print: prompt eval time = 156702.59 ms / 36807 tokens ( 4.26 ms per token, 234.88 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 158963.12 ms / 36808 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-iq4_nl.hsw CHANGED
@@ -1,760 +1,12 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- hellaswag_score : loaded 10042 tasks from prompt.
6
- ================================= is_spm = 0
7
- hellaswag_score : selecting 750 randomized tasks.
8
- hellaswag_score : calculating hellaswag score over selected tasks.
9
 
10
- task acc_norm
11
- 1 0.00000000
12
- 2 50.00000000
13
- 3 66.66666667
14
- 4 75.00000000
15
- 5 80.00000000
16
- 6 83.33333333
17
- 7 85.71428571
18
- 8 87.50000000
19
- 9 88.88888889
20
- 10 90.00000000
21
- 11 90.90909091
22
- 12 83.33333333
23
- 13 84.61538462
24
- 14 85.71428571
25
- 15 86.66666667
26
- 16 87.50000000
27
- 17 88.23529412
28
- 18 88.88888889
29
- 19 84.21052632
30
- 20 85.00000000
31
- 21 85.71428571
32
- 22 86.36363636
33
- 23 86.95652174
34
- 24 83.33333333
35
- 25 80.00000000
36
- 26 76.92307692
37
- 27 74.07407407
38
- 28 75.00000000
39
- 29 75.86206897
40
- 30 76.66666667
41
- 31 77.41935484
42
- 32 75.00000000
43
- 33 75.75757576
44
- 34 76.47058824
45
- 35 77.14285714
46
- 36 77.77777778
47
- 37 75.67567568
48
- 38 76.31578947
49
- 39 74.35897436
50
- 40 72.50000000
51
- 41 73.17073171
52
- 42 71.42857143
53
- 43 72.09302326
54
- 44 72.72727273
55
- 45 71.11111111
56
- 46 71.73913043
57
- 47 72.34042553
58
- 48 72.91666667
59
- 49 73.46938776
60
- 50 72.00000000
61
- 51 72.54901961
62
- 52 71.15384615
63
- 53 69.81132075
64
- 54 68.51851852
65
- 55 69.09090909
66
- 56 67.85714286
67
- 57 68.42105263
68
- 58 68.96551724
69
- 59 69.49152542
70
- 60 70.00000000
71
- 61 70.49180328
72
- 62 70.96774194
73
- 63 71.42857143
74
- 64 71.87500000
75
- 65 70.76923077
76
- 66 69.69696970
77
- 67 70.14925373
78
- 68 70.58823529
79
- 69 71.01449275
80
- 70 71.42857143
81
- 71 70.42253521
82
- 72 70.83333333
83
- 73 71.23287671
84
- 74 71.62162162
85
- 75 70.66666667
86
- 76 71.05263158
87
- 77 71.42857143
88
- 78 71.79487179
89
- 79 72.15189873
90
- 80 72.50000000
91
- 81 72.83950617
92
- 82 73.17073171
93
- 83 73.49397590
94
- 84 73.80952381
95
- 85 74.11764706
96
- 86 74.41860465
97
- 87 74.71264368
98
- 88 75.00000000
99
- 89 74.15730337
100
- 90 74.44444444
101
- 91 74.72527473
102
- 92 73.91304348
103
- 93 74.19354839
104
- 94 74.46808511
105
- 95 74.73684211
106
- 96 75.00000000
107
- 97 75.25773196
108
- 98 75.51020408
109
- 99 75.75757576
110
- 100 75.00000000
111
- 101 75.24752475
112
- 102 74.50980392
113
- 103 73.78640777
114
- 104 74.03846154
115
- 105 74.28571429
116
- 106 74.52830189
117
- 107 74.76635514
118
- 108 75.00000000
119
- 109 74.31192661
120
- 110 74.54545455
121
- 111 73.87387387
122
- 112 74.10714286
123
- 113 74.33628319
124
- 114 74.56140351
125
- 115 74.78260870
126
- 116 74.13793103
127
- 117 74.35897436
128
- 118 74.57627119
129
- 119 74.78991597
130
- 120 74.16666667
131
- 121 74.38016529
132
- 122 74.59016393
133
- 123 74.79674797
134
- 124 75.00000000
135
- 125 75.20000000
136
- 126 75.39682540
137
- 127 75.59055118
138
- 128 75.00000000
139
- 129 75.19379845
140
- 130 74.61538462
141
- 131 74.80916031
142
- 132 75.00000000
143
- 133 75.18796992
144
- 134 75.37313433
145
- 135 75.55555556
146
- 136 75.73529412
147
- 137 75.91240876
148
- 138 76.08695652
149
- 139 75.53956835
150
- 140 75.71428571
151
- 141 75.88652482
152
- 142 76.05633803
153
- 143 76.22377622
154
- 144 76.38888889
155
- 145 76.55172414
156
- 146 76.71232877
157
- 147 76.87074830
158
- 148 77.02702703
159
- 149 77.18120805
160
- 150 77.33333333
161
- 151 76.82119205
162
- 152 76.31578947
163
- 153 76.47058824
164
- 154 76.62337662
165
- 155 76.77419355
166
- 156 76.28205128
167
- 157 75.79617834
168
- 158 75.94936709
169
- 159 75.47169811
170
- 160 75.62500000
171
- 161 75.77639752
172
- 162 75.92592593
173
- 163 76.07361963
174
- 164 76.21951220
175
- 165 76.36363636
176
- 166 76.50602410
177
- 167 76.64670659
178
- 168 76.78571429
179
- 169 76.33136095
180
- 170 76.47058824
181
- 171 76.60818713
182
- 172 76.74418605
183
- 173 76.87861272
184
- 174 77.01149425
185
- 175 76.57142857
186
- 176 76.70454545
187
- 177 76.83615819
188
- 178 76.96629213
189
- 179 76.53631285
190
- 180 76.66666667
191
- 181 76.24309392
192
- 182 76.37362637
193
- 183 76.50273224
194
- 184 76.63043478
195
- 185 76.75675676
196
- 186 76.34408602
197
- 187 75.93582888
198
- 188 75.53191489
199
- 189 75.66137566
200
- 190 75.78947368
201
- 191 75.91623037
202
- 192 76.04166667
203
- 193 75.64766839
204
- 194 75.77319588
205
- 195 75.38461538
206
- 196 75.51020408
207
- 197 75.12690355
208
- 198 74.74747475
209
- 199 74.87437186
210
- 200 75.00000000
211
- 201 75.12437811
212
- 202 75.24752475
213
- 203 74.87684729
214
- 204 75.00000000
215
- 205 75.12195122
216
- 206 75.24271845
217
- 207 75.36231884
218
- 208 75.00000000
219
- 209 75.11961722
220
- 210 75.23809524
221
- 211 75.35545024
222
- 212 75.47169811
223
- 213 75.58685446
224
- 214 75.70093458
225
- 215 75.34883721
226
- 216 75.46296296
227
- 217 75.11520737
228
- 218 75.22935780
229
- 219 75.34246575
230
- 220 75.45454545
231
- 221 75.11312217
232
- 222 75.22522523
233
- 223 75.33632287
234
- 224 75.44642857
235
- 225 75.55555556
236
- 226 75.66371681
237
- 227 75.33039648
238
- 228 75.43859649
239
- 229 75.54585153
240
- 230 75.65217391
241
- 231 75.32467532
242
- 232 75.00000000
243
- 233 75.10729614
244
- 234 75.21367521
245
- 235 74.89361702
246
- 236 74.57627119
247
- 237 74.68354430
248
- 238 74.78991597
249
- 239 74.89539749
250
- 240 75.00000000
251
- 241 75.10373444
252
- 242 75.20661157
253
- 243 75.30864198
254
- 244 75.40983607
255
- 245 75.10204082
256
- 246 75.20325203
257
- 247 75.30364372
258
- 248 75.40322581
259
- 249 75.50200803
260
- 250 75.60000000
261
- 251 75.69721116
262
- 252 75.79365079
263
- 253 75.49407115
264
- 254 75.59055118
265
- 255 75.68627451
266
- 256 75.78125000
267
- 257 75.48638132
268
- 258 75.58139535
269
- 259 75.67567568
270
- 260 75.76923077
271
- 261 75.86206897
272
- 262 75.57251908
273
- 263 75.66539924
274
- 264 75.75757576
275
- 265 75.84905660
276
- 266 75.56390977
277
- 267 75.65543071
278
- 268 75.74626866
279
- 269 75.83643123
280
- 270 75.92592593
281
- 271 75.64575646
282
- 272 75.73529412
283
- 273 75.45787546
284
- 274 75.54744526
285
- 275 75.27272727
286
- 276 75.36231884
287
- 277 75.45126354
288
- 278 75.53956835
289
- 279 75.62724014
290
- 280 75.71428571
291
- 281 75.80071174
292
- 282 75.88652482
293
- 283 75.61837456
294
- 284 75.70422535
295
- 285 75.43859649
296
- 286 75.52447552
297
- 287 75.60975610
298
- 288 75.69444444
299
- 289 75.77854671
300
- 290 75.86206897
301
- 291 75.94501718
302
- 292 75.68493151
303
- 293 75.76791809
304
- 294 75.85034014
305
- 295 75.93220339
306
- 296 76.01351351
307
- 297 76.09427609
308
- 298 76.17449664
309
- 299 75.91973244
310
- 300 75.66666667
311
- 301 75.74750831
312
- 302 75.82781457
313
- 303 75.57755776
314
- 304 75.32894737
315
- 305 75.40983607
316
- 306 75.49019608
317
- 307 75.24429967
318
- 308 75.32467532
319
- 309 75.40453074
320
- 310 75.48387097
321
- 311 75.56270096
322
- 312 75.64102564
323
- 313 75.71884984
324
- 314 75.79617834
325
- 315 75.87301587
326
- 316 75.94936709
327
- 317 76.02523659
328
- 318 75.78616352
329
- 319 75.54858934
330
- 320 75.62500000
331
- 321 75.38940810
332
- 322 75.15527950
333
- 323 75.23219814
334
- 324 75.30864198
335
- 325 75.38461538
336
- 326 75.46012270
337
- 327 75.22935780
338
- 328 75.30487805
339
- 329 75.37993921
340
- 330 75.45454545
341
- 331 75.52870091
342
- 332 75.60240964
343
- 333 75.67567568
344
- 334 75.44910180
345
- 335 75.52238806
346
- 336 75.59523810
347
- 337 75.37091988
348
- 338 75.14792899
349
- 339 75.22123894
350
- 340 75.29411765
351
- 341 75.36656891
352
- 342 75.43859649
353
- 343 75.51020408
354
- 344 75.29069767
355
- 345 75.36231884
356
- 346 75.14450867
357
- 347 75.21613833
358
- 348 75.00000000
359
- 349 74.78510029
360
- 350 74.85714286
361
- 351 74.92877493
362
- 352 75.00000000
363
- 353 75.07082153
364
- 354 75.14124294
365
- 355 75.21126761
366
- 356 75.28089888
367
- 357 75.35014006
368
- 358 75.41899441
369
- 359 75.48746518
370
- 360 75.27777778
371
- 361 75.34626039
372
- 362 75.41436464
373
- 363 75.48209366
374
- 364 75.54945055
375
- 365 75.61643836
376
- 366 75.68306011
377
- 367 75.74931880
378
- 368 75.81521739
379
- 369 75.88075881
380
- 370 75.94594595
381
- 371 76.01078167
382
- 372 76.07526882
383
- 373 75.87131367
384
- 374 75.93582888
385
- 375 76.00000000
386
- 376 76.06382979
387
- 377 75.86206897
388
- 378 75.92592593
389
- 379 75.98944591
390
- 380 76.05263158
391
- 381 76.11548556
392
- 382 76.17801047
393
- 383 76.24020888
394
- 384 76.30208333
395
- 385 76.36363636
396
- 386 76.42487047
397
- 387 76.48578811
398
- 388 76.28865979
399
- 389 76.34961440
400
- 390 76.41025641
401
- 391 76.21483376
402
- 392 76.27551020
403
- 393 76.33587786
404
- 394 76.39593909
405
- 395 76.45569620
406
- 396 76.51515152
407
- 397 76.57430730
408
- 398 76.63316583
409
- 399 76.69172932
410
- 400 76.75000000
411
- 401 76.80798005
412
- 402 76.86567164
413
- 403 76.67493797
414
- 404 76.73267327
415
- 405 76.54320988
416
- 406 76.60098522
417
- 407 76.65847666
418
- 408 76.71568627
419
- 409 76.77261614
420
- 410 76.82926829
421
- 411 76.88564477
422
- 412 76.94174757
423
- 413 76.99757869
424
- 414 77.05314010
425
- 415 76.86746988
426
- 416 76.92307692
427
- 417 76.97841727
428
- 418 77.03349282
429
- 419 77.08830549
430
- 420 77.14285714
431
- 421 77.19714964
432
- 422 77.25118483
433
- 423 77.30496454
434
- 424 77.35849057
435
- 425 77.41176471
436
- 426 77.46478873
437
- 427 77.28337237
438
- 428 77.33644860
439
- 429 77.38927739
440
- 430 77.44186047
441
- 431 77.26218097
442
- 432 77.08333333
443
- 433 77.13625866
444
- 434 77.18894009
445
- 435 77.01149425
446
- 436 77.06422018
447
- 437 76.88787185
448
- 438 76.94063927
449
- 439 76.99316629
450
- 440 77.04545455
451
- 441 77.09750567
452
- 442 77.14932127
453
- 443 77.20090293
454
- 444 77.25225225
455
- 445 77.30337079
456
- 446 77.35426009
457
- 447 77.40492170
458
- 448 77.45535714
459
- 449 77.50556793
460
- 450 77.55555556
461
- 451 77.38359202
462
- 452 77.43362832
463
- 453 77.48344371
464
- 454 77.53303965
465
- 455 77.58241758
466
- 456 77.63157895
467
- 457 77.68052516
468
- 458 77.72925764
469
- 459 77.77777778
470
- 460 77.82608696
471
- 461 77.87418655
472
- 462 77.92207792
473
- 463 77.96976242
474
- 464 78.01724138
475
- 465 77.84946237
476
- 466 77.89699571
477
- 467 77.94432548
478
- 468 77.77777778
479
- 469 77.82515991
480
- 470 77.65957447
481
- 471 77.70700637
482
- 472 77.75423729
483
- 473 77.80126850
484
- 474 77.84810127
485
- 475 77.89473684
486
- 476 77.73109244
487
- 477 77.56813417
488
- 478 77.61506276
489
- 479 77.66179541
490
- 480 77.70833333
491
- 481 77.75467775
492
- 482 77.59336100
493
- 483 77.63975155
494
- 484 77.47933884
495
- 485 77.31958763
496
- 486 77.36625514
497
- 487 77.41273101
498
- 488 77.25409836
499
- 489 77.30061350
500
- 490 77.34693878
501
- 491 77.39307536
502
- 492 77.43902439
503
- 493 77.48478702
504
- 494 77.53036437
505
- 495 77.57575758
506
- 496 77.62096774
507
- 497 77.66599598
508
- 498 77.71084337
509
- 499 77.55511022
510
- 500 77.60000000
511
- 501 77.64471058
512
- 502 77.68924303
513
- 503 77.73359841
514
- 504 77.77777778
515
- 505 77.82178218
516
- 506 77.86561265
517
- 507 77.90927022
518
- 508 77.95275591
519
- 509 77.99607073
520
- 510 78.03921569
521
- 511 78.08219178
522
- 512 77.92968750
523
- 513 77.97270955
524
- 514 78.01556420
525
- 515 78.05825243
526
- 516 77.90697674
527
- 517 77.75628627
528
- 518 77.79922780
529
- 519 77.84200385
530
- 520 77.88461538
531
- 521 77.73512476
532
- 522 77.77777778
533
- 523 77.82026769
534
- 524 77.86259542
535
- 525 77.90476190
536
- 526 77.75665399
537
- 527 77.60910816
538
- 528 77.65151515
539
- 529 77.69376181
540
- 530 77.73584906
541
- 531 77.77777778
542
- 532 77.81954887
543
- 533 77.86116323
544
- 534 77.90262172
545
- 535 77.94392523
546
- 536 77.98507463
547
- 537 78.02607076
548
- 538 78.06691450
549
- 539 77.92207792
550
- 540 77.96296296
551
- 541 78.00369686
552
- 542 78.04428044
553
- 543 78.08471455
554
- 544 78.12500000
555
- 545 78.16513761
556
- 546 78.20512821
557
- 547 78.24497258
558
- 548 78.28467153
559
- 549 78.32422587
560
- 550 78.18181818
561
- 551 78.22141561
562
- 552 78.07971014
563
- 553 78.11934901
564
- 554 78.15884477
565
- 555 78.01801802
566
- 556 78.05755396
567
- 557 77.91741472
568
- 558 77.77777778
569
- 559 77.81753131
570
- 560 77.85714286
571
- 561 77.89661319
572
- 562 77.75800712
573
- 563 77.79751332
574
- 564 77.83687943
575
- 565 77.69911504
576
- 566 77.56183746
577
- 567 77.60141093
578
- 568 77.64084507
579
- 569 77.50439367
580
- 570 77.54385965
581
- 571 77.58318739
582
- 572 77.62237762
583
- 573 77.66143106
584
- 574 77.70034843
585
- 575 77.73913043
586
- 576 77.77777778
587
- 577 77.64298094
588
- 578 77.68166090
589
- 579 77.72020725
590
- 580 77.75862069
591
- 581 77.62478485
592
- 582 77.66323024
593
- 583 77.70154374
594
- 584 77.73972603
595
- 585 77.77777778
596
- 586 77.81569966
597
- 587 77.85349233
598
- 588 77.72108844
599
- 589 77.58913413
600
- 590 77.62711864
601
- 591 77.66497462
602
- 592 77.70270270
603
- 593 77.57166948
604
- 594 77.44107744
605
- 595 77.47899160
606
- 596 77.51677852
607
- 597 77.55443886
608
- 598 77.59197324
609
- 599 77.46243740
610
- 600 77.50000000
611
- 601 77.53743760
612
- 602 77.57475083
613
- 603 77.61194030
614
- 604 77.64900662
615
- 605 77.68595041
616
- 606 77.55775578
617
- 607 77.59472817
618
- 608 77.46710526
619
- 609 77.50410509
620
- 610 77.54098361
621
- 611 77.57774141
622
- 612 77.45098039
623
- 613 77.32463295
624
- 614 77.36156352
625
- 615 77.39837398
626
- 616 77.27272727
627
- 617 77.14748784
628
- 618 77.18446602
629
- 619 77.22132472
630
- 620 77.25806452
631
- 621 77.29468599
632
- 622 77.33118971
633
- 623 77.36757624
634
- 624 77.40384615
635
- 625 77.44000000
636
- 626 77.47603834
637
- 627 77.51196172
638
- 628 77.54777070
639
- 629 77.42448331
640
- 630 77.46031746
641
- 631 77.49603803
642
- 632 77.37341772
643
- 633 77.25118483
644
- 634 77.28706625
645
- 635 77.32283465
646
- 636 77.35849057
647
- 637 77.39403454
648
- 638 77.42946708
649
- 639 77.46478873
650
- 640 77.34375000
651
- 641 77.37909516
652
- 642 77.41433022
653
- 643 77.44945568
654
- 644 77.48447205
655
- 645 77.36434109
656
- 646 77.39938080
657
- 647 77.43431221
658
- 648 77.46913580
659
- 649 77.34976888
660
- 650 77.38461538
661
- 651 77.41935484
662
- 652 77.45398773
663
- 653 77.48851455
664
- 654 77.52293578
665
- 655 77.55725191
666
- 656 77.59146341
667
- 657 77.62557078
668
- 658 77.65957447
669
- 659 77.54172989
670
- 660 77.42424242
671
- 661 77.45839637
672
- 662 77.49244713
673
- 663 77.52639517
674
- 664 77.56024096
675
- 665 77.59398496
676
- 666 77.62762763
677
- 667 77.66116942
678
- 668 77.69461078
679
- 669 77.72795217
680
- 670 77.76119403
681
- 671 77.79433681
682
- 672 77.82738095
683
- 673 77.86032689
684
- 674 77.89317507
685
- 675 77.92592593
686
- 676 77.95857988
687
- 677 77.99113737
688
- 678 78.02359882
689
- 679 78.05596465
690
- 680 78.08823529
691
- 681 78.12041116
692
- 682 78.15249267
693
- 683 78.18448023
694
- 684 78.21637427
695
- 685 78.24817518
696
- 686 78.27988338
697
- 687 78.31149927
698
- 688 78.34302326
699
- 689 78.37445573
700
- 690 78.40579710
701
- 691 78.43704776
702
- 692 78.46820809
703
- 693 78.35497835
704
- 694 78.38616715
705
- 695 78.41726619
706
- 696 78.44827586
707
- 697 78.47919656
708
- 698 78.51002865
709
- 699 78.54077253
710
- 700 78.57142857
711
- 701 78.60199715
712
- 702 78.63247863
713
- 703 78.66287340
714
- 704 78.69318182
715
- 705 78.72340426
716
- 706 78.75354108
717
- 707 78.78359264
718
- 708 78.81355932
719
- 709 78.84344147
720
- 710 78.87323944
721
- 711 78.90295359
722
- 712 78.93258427
723
- 713 78.96213184
724
- 714 78.99159664
725
- 715 78.88111888
726
- 716 78.91061453
727
- 717 78.94002789
728
- 718 78.96935933
729
- 719 78.99860918
730
- 720 79.02777778
731
- 721 79.05686546
732
- 722 79.08587258
733
- 723 79.11479945
734
- 724 79.14364641
735
- 725 79.17241379
736
- 726 79.06336088
737
- 727 79.09215956
738
- 728 79.12087912
739
- 729 79.14951989
740
- 730 79.17808219
741
- 731 79.20656635
742
- 732 79.09836066
743
- 733 78.99045020
744
- 734 79.01907357
745
- 735 78.91156463
746
- 736 78.94021739
747
- 737 78.83310719
748
- 738 78.72628726
749
- 739 78.61975643
750
- 740 78.64864865
751
- 741 78.67746289
752
- 742 78.70619946
753
- 743 78.73485868
754
- 744 78.76344086
755
- 745 78.79194631
756
- 746 78.82037534
757
- 747 78.71485944
758
- 748 78.74331551
759
- 749 78.77169559
760
- 750 78.80000000
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-IQ4_NL.gguf (version GGUF V3 (latest))
4
 
5
+ 750 77.86666667
 
 
 
 
6
 
7
+
8
+ llama_perf_context_print: load time = 285.97 ms
9
+ llama_perf_context_print: prompt eval time = 426128.64 ms / 126096 tokens ( 3.38 ms per token, 295.91 tokens per second)
10
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
11
+ llama_perf_context_print: total time = 431845.15 ms / 126097 tokens
12
+ ggml_metal_free: deallocating
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scores/Watt-Tool-8B-iq4_nl.mmlu CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 1548 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 1548 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 50.00000000
13
- 3 33.33333333
14
- 4 50.00000000
15
- 5 40.00000000
16
- 6 33.33333333
17
- 7 42.85714286
18
- 8 50.00000000
19
- 9 55.55555556
20
- 10 60.00000000
21
- 11 54.54545455
22
- 12 58.33333333
23
- 13 53.84615385
24
- 14 50.00000000
25
- 15 53.33333333
26
- 16 56.25000000
27
- 17 58.82352941
28
- 18 55.55555556
29
- 19 57.89473684
30
- 20 60.00000000
31
- 21 61.90476190
32
- 22 59.09090909
33
- 23 56.52173913
34
- 24 58.33333333
35
- 25 60.00000000
36
- 26 57.69230769
37
- 27 59.25925926
38
- 28 60.71428571
39
- 29 62.06896552
40
- 30 63.33333333
41
- 31 61.29032258
42
- 32 62.50000000
43
- 33 60.60606061
44
- 34 61.76470588
45
- 35 62.85714286
46
- 36 61.11111111
47
- 37 59.45945946
48
- 38 60.52631579
49
- 39 58.97435897
50
- 40 57.50000000
51
- 41 56.09756098
52
- 42 54.76190476
53
- 43 55.81395349
54
- 44 54.54545455
55
- 45 55.55555556
56
- 46 54.34782609
57
- 47 53.19148936
58
- 48 54.16666667
59
- 49 55.10204082
60
- 50 54.00000000
61
- 51 52.94117647
62
- 52 53.84615385
63
- 53 52.83018868
64
- 54 51.85185185
65
- 55 52.72727273
66
- 56 53.57142857
67
- 57 54.38596491
68
- 58 53.44827586
69
- 59 52.54237288
70
- 60 51.66666667
71
- 61 52.45901639
72
- 62 51.61290323
73
- 63 50.79365079
74
- 64 50.00000000
75
- 65 49.23076923
76
- 66 48.48484848
77
- 67 47.76119403
78
- 68 47.05882353
79
- 69 46.37681159
80
- 70 47.14285714
81
- 71 47.88732394
82
- 72 47.22222222
83
- 73 46.57534247
84
- 74 47.29729730
85
- 75 48.00000000
86
- 76 48.68421053
87
- 77 49.35064935
88
- 78 50.00000000
89
- 79 49.36708861
90
- 80 50.00000000
91
- 81 50.61728395
92
- 82 50.00000000
93
- 83 50.60240964
94
- 84 51.19047619
95
- 85 50.58823529
96
- 86 50.00000000
97
- 87 50.57471264
98
- 88 51.13636364
99
- 89 50.56179775
100
- 90 50.00000000
101
- 91 49.45054945
102
- 92 48.91304348
103
- 93 48.38709677
104
- 94 48.93617021
105
- 95 48.42105263
106
- 96 48.95833333
107
- 97 48.45360825
108
- 98 47.95918367
109
- 99 47.47474747
110
- 100 47.00000000
111
- 101 47.52475248
112
- 102 47.05882353
113
- 103 46.60194175
114
- 104 47.11538462
115
- 105 47.61904762
116
- 106 48.11320755
117
- 107 47.66355140
118
- 108 48.14814815
119
- 109 48.62385321
120
- 110 49.09090909
121
- 111 49.54954955
122
- 112 49.10714286
123
- 113 48.67256637
124
- 114 49.12280702
125
- 115 49.56521739
126
- 116 50.00000000
127
- 117 49.57264957
128
- 118 49.15254237
129
- 119 49.57983193
130
- 120 49.16666667
131
- 121 49.58677686
132
- 122 49.18032787
133
- 123 48.78048780
134
- 124 48.38709677
135
- 125 48.80000000
136
- 126 48.41269841
137
- 127 48.03149606
138
- 128 47.65625000
139
- 129 47.28682171
140
- 130 46.92307692
141
- 131 46.56488550
142
- 132 46.96969697
143
- 133 46.61654135
144
- 134 46.26865672
145
- 135 45.92592593
146
- 136 45.58823529
147
- 137 45.25547445
148
- 138 44.92753623
149
- 139 45.32374101
150
- 140 45.71428571
151
- 141 46.09929078
152
- 142 46.47887324
153
- 143 46.85314685
154
- 144 47.22222222
155
- 145 47.58620690
156
- 146 47.94520548
157
- 147 47.61904762
158
- 148 47.97297297
159
- 149 47.65100671
160
- 150 47.33333333
161
- 151 47.01986755
162
- 152 47.36842105
163
- 153 47.71241830
164
- 154 47.40259740
165
- 155 47.09677419
166
- 156 46.79487179
167
- 157 46.49681529
168
- 158 46.83544304
169
- 159 46.54088050
170
- 160 46.87500000
171
- 161 46.58385093
172
- 162 46.29629630
173
- 163 46.01226994
174
- 164 45.73170732
175
- 165 46.06060606
176
- 166 45.78313253
177
- 167 46.10778443
178
- 168 45.83333333
179
- 169 46.15384615
180
- 170 45.88235294
181
- 171 46.19883041
182
- 172 45.93023256
183
- 173 45.66473988
184
- 174 45.40229885
185
- 175 45.14285714
186
- 176 44.88636364
187
- 177 44.63276836
188
- 178 44.38202247
189
- 179 44.69273743
190
- 180 44.44444444
191
- 181 44.19889503
192
- 182 44.50549451
193
- 183 44.80874317
194
- 184 44.56521739
195
- 185 44.86486486
196
- 186 45.16129032
197
- 187 44.91978610
198
- 188 45.21276596
199
- 189 45.50264550
200
- 190 45.26315789
201
- 191 45.54973822
202
- 192 45.31250000
203
- 193 45.07772021
204
- 194 45.36082474
205
- 195 45.64102564
206
- 196 45.91836735
207
- 197 45.68527919
208
- 198 45.95959596
209
- 199 45.72864322
210
- 200 45.50000000
211
- 201 45.27363184
212
- 202 45.04950495
213
- 203 44.82758621
214
- 204 44.60784314
215
- 205 44.87804878
216
- 206 44.66019417
217
- 207 44.92753623
218
- 208 44.71153846
219
- 209 44.49760766
220
- 210 44.28571429
221
- 211 44.54976303
222
- 212 44.33962264
223
- 213 44.13145540
224
- 214 43.92523364
225
- 215 44.18604651
226
- 216 44.44444444
227
- 217 44.23963134
228
- 218 44.49541284
229
- 219 44.74885845
230
- 220 45.00000000
231
- 221 44.79638009
232
- 222 44.59459459
233
- 223 44.84304933
234
- 224 45.08928571
235
- 225 45.33333333
236
- 226 45.13274336
237
- 227 44.93392070
238
- 228 45.17543860
239
- 229 44.97816594
240
- 230 44.78260870
241
- 231 45.02164502
242
- 232 45.25862069
243
- 233 45.49356223
244
- 234 45.72649573
245
- 235 45.53191489
246
- 236 45.33898305
247
- 237 45.14767932
248
- 238 44.95798319
249
- 239 44.76987448
250
- 240 45.00000000
251
- 241 44.81327801
252
- 242 45.04132231
253
- 243 45.26748971
254
- 244 45.08196721
255
- 245 45.30612245
256
- 246 45.12195122
257
- 247 44.93927126
258
- 248 45.16129032
259
- 249 44.97991968
260
- 250 44.80000000
261
- 251 44.62151394
262
- 252 44.44444444
263
- 253 44.26877470
264
- 254 44.09448819
265
- 255 43.92156863
266
- 256 44.14062500
267
- 257 44.35797665
268
- 258 44.18604651
269
- 259 44.01544402
270
- 260 44.23076923
271
- 261 44.06130268
272
- 262 43.89312977
273
- 263 43.72623574
274
- 264 43.56060606
275
- 265 43.39622642
276
- 266 43.60902256
277
- 267 43.44569288
278
- 268 43.28358209
279
- 269 43.49442379
280
- 270 43.70370370
281
- 271 43.54243542
282
- 272 43.75000000
283
- 273 43.58974359
284
- 274 43.43065693
285
- 275 43.27272727
286
- 276 43.47826087
287
- 277 43.68231047
288
- 278 43.52517986
289
- 279 43.36917563
290
- 280 43.21428571
291
- 281 43.06049822
292
- 282 42.90780142
293
- 283 42.75618375
294
- 284 42.60563380
295
- 285 42.45614035
296
- 286 42.30769231
297
- 287 42.50871080
298
- 288 42.36111111
299
- 289 42.21453287
300
- 290 42.41379310
301
- 291 42.61168385
302
- 292 42.80821918
303
- 293 43.00341297
304
- 294 42.85714286
305
- 295 42.71186441
306
- 296 42.56756757
307
- 297 42.42424242
308
- 298 42.61744966
309
- 299 42.80936455
310
- 300 43.00000000
311
- 301 42.85714286
312
- 302 43.04635762
313
- 303 42.90429043
314
- 304 43.09210526
315
- 305 43.27868852
316
- 306 43.13725490
317
- 307 43.32247557
318
- 308 43.18181818
319
- 309 43.04207120
320
- 310 42.90322581
321
- 311 43.08681672
322
- 312 43.26923077
323
- 313 43.13099042
324
- 314 43.31210191
325
- 315 43.17460317
326
- 316 43.03797468
327
- 317 43.21766562
328
- 318 43.08176101
329
- 319 43.26018809
330
- 320 43.12500000
331
- 321 42.99065421
332
- 322 43.16770186
333
- 323 43.03405573
334
- 324 42.90123457
335
- 325 43.07692308
336
- 326 42.94478528
337
- 327 43.11926606
338
- 328 42.98780488
339
- 329 42.85714286
340
- 330 42.72727273
341
- 331 42.59818731
342
- 332 42.46987952
343
- 333 42.64264264
344
- 334 42.51497006
345
- 335 42.68656716
346
- 336 42.85714286
347
- 337 42.72997033
348
- 338 42.60355030
349
- 339 42.47787611
350
- 340 42.64705882
351
- 341 42.81524927
352
- 342 42.98245614
353
- 343 42.85714286
354
- 344 42.73255814
355
- 345 42.60869565
356
- 346 42.77456647
357
- 347 42.93948127
358
- 348 42.81609195
359
- 349 42.69340974
360
- 350 42.85714286
361
- 351 42.73504274
362
- 352 42.61363636
363
- 353 42.77620397
364
- 354 42.65536723
365
- 355 42.53521127
366
- 356 42.69662921
367
- 357 42.57703081
368
- 358 42.45810056
369
- 359 42.33983287
370
- 360 42.50000000
371
- 361 42.65927978
372
- 362 42.81767956
373
- 363 42.69972452
374
- 364 42.58241758
375
- 365 42.46575342
376
- 366 42.62295082
377
- 367 42.50681199
378
- 368 42.39130435
379
- 369 42.27642276
380
- 370 42.43243243
381
- 371 42.31805930
382
- 372 42.20430108
383
- 373 42.09115282
384
- 374 41.97860963
385
- 375 42.13333333
386
- 376 42.02127660
387
- 377 42.17506631
388
- 378 42.32804233
389
- 379 42.21635884
390
- 380 42.10526316
391
- 381 42.25721785
392
- 382 42.14659686
393
- 383 42.29765013
394
- 384 42.18750000
395
- 385 42.33766234
396
- 386 42.22797927
397
- 387 42.11886305
398
- 388 42.01030928
399
- 389 42.15938303
400
- 390 42.30769231
401
- 391 42.45524297
402
- 392 42.60204082
403
- 393 42.49363868
404
- 394 42.38578680
405
- 395 42.53164557
406
- 396 42.67676768
407
- 397 42.82115869
408
- 398 42.71356784
409
- 399 42.85714286
410
- 400 42.75000000
411
- 401 42.89276808
412
- 402 43.03482587
413
- 403 42.92803970
414
- 404 43.06930693
415
- 405 43.20987654
416
- 406 43.10344828
417
- 407 42.99754300
418
- 408 42.89215686
419
- 409 43.03178484
420
- 410 42.92682927
421
- 411 42.82238443
422
- 412 42.71844660
423
- 413 42.61501211
424
- 414 42.51207729
425
- 415 42.40963855
426
- 416 42.54807692
427
- 417 42.44604317
428
- 418 42.34449761
429
- 419 42.48210024
430
- 420 42.38095238
431
- 421 42.51781473
432
- 422 42.41706161
433
- 423 42.31678487
434
- 424 42.45283019
435
- 425 42.58823529
436
- 426 42.72300469
437
- 427 42.85714286
438
- 428 42.75700935
439
- 429 42.89044289
440
- 430 42.79069767
441
- 431 42.92343387
442
- 432 42.82407407
443
- 433 42.72517321
444
- 434 42.62672811
445
- 435 42.52873563
446
- 436 42.43119266
447
- 437 42.33409611
448
- 438 42.46575342
449
- 439 42.59681093
450
- 440 42.50000000
451
- 441 42.40362812
452
- 442 42.30769231
453
- 443 42.21218962
454
- 444 42.11711712
455
- 445 42.02247191
456
- 446 41.92825112
457
- 447 41.83445190
458
- 448 41.74107143
459
- 449 41.64810690
460
- 450 41.77777778
461
- 451 41.90687361
462
- 452 41.81415929
463
- 453 41.94260486
464
- 454 41.85022026
465
- 455 41.75824176
466
- 456 41.88596491
467
- 457 42.01312910
468
- 458 41.92139738
469
- 459 41.83006536
470
- 460 41.95652174
471
- 461 41.86550976
472
- 462 41.77489177
473
- 463 41.90064795
474
- 464 42.02586207
475
- 465 41.93548387
476
- 466 41.84549356
477
- 467 41.97002141
478
- 468 42.09401709
479
- 469 42.21748401
480
- 470 42.12765957
481
- 471 42.03821656
482
- 472 41.94915254
483
- 473 42.07188161
484
- 474 41.98312236
485
- 475 41.89473684
486
- 476 42.01680672
487
- 477 41.92872117
488
- 478 42.05020921
489
- 479 41.96242171
490
- 480 42.08333333
491
- 481 41.99584200
492
- 482 41.90871369
493
- 483 41.82194617
494
- 484 41.73553719
495
- 485 41.64948454
496
- 486 41.76954733
497
- 487 41.88911704
498
- 488 41.80327869
499
- 489 41.71779141
500
- 490 41.83673469
501
- 491 41.75152749
502
- 492 41.66666667
503
- 493 41.58215010
504
- 494 41.70040486
505
- 495 41.81818182
506
- 496 41.93548387
507
- 497 41.85110664
508
- 498 41.76706827
509
- 499 41.68336673
510
- 500 41.80000000
511
- 501 41.71656687
512
- 502 41.83266932
513
- 503 41.74950298
514
- 504 41.66666667
515
- 505 41.58415842
516
- 506 41.69960474
517
- 507 41.61735700
518
- 508 41.73228346
519
- 509 41.65029470
520
- 510 41.56862745
521
- 511 41.48727984
522
- 512 41.40625000
523
- 513 41.32553606
524
- 514 41.24513619
525
- 515 41.16504854
526
- 516 41.27906977
527
- 517 41.39264990
528
- 518 41.50579151
529
- 519 41.42581888
530
- 520 41.34615385
531
- 521 41.26679463
532
- 522 41.37931034
533
- 523 41.30019120
534
- 524 41.41221374
535
- 525 41.33333333
536
- 526 41.25475285
537
- 527 41.17647059
538
- 528 41.09848485
539
- 529 41.02079395
540
- 530 41.13207547
541
- 531 41.05461394
542
- 532 40.97744361
543
- 533 40.90056285
544
- 534 41.01123596
545
- 535 41.12149533
546
- 536 41.04477612
547
- 537 41.15456238
548
- 538 41.26394052
549
- 539 41.18738404
550
- 540 41.11111111
551
- 541 41.03512015
552
- 542 41.14391144
553
- 543 41.06813996
554
- 544 41.17647059
555
- 545 41.10091743
556
- 546 41.02564103
557
- 547 40.95063985
558
- 548 41.05839416
559
- 549 40.98360656
560
- 550 40.90909091
561
- 551 41.01633394
562
- 552 40.94202899
563
- 553 40.86799277
564
- 554 40.97472924
565
- 555 40.90090090
566
- 556 41.00719424
567
- 557 41.11310592
568
- 558 41.21863799
569
- 559 41.14490161
570
- 560 41.07142857
571
- 561 40.99821747
572
- 562 41.10320285
573
- 563 41.20781528
574
- 564 41.13475177
575
- 565 41.23893805
576
- 566 41.34275618
577
- 567 41.26984127
578
- 568 41.19718310
579
- 569 41.12478032
580
- 570 41.05263158
581
- 571 40.98073555
582
- 572 41.08391608
583
- 573 41.01221640
584
- 574 40.94076655
585
- 575 40.86956522
586
- 576 40.97222222
587
- 577 41.07452340
588
- 578 41.17647059
589
- 579 41.10535406
590
- 580 41.20689655
591
- 581 41.30808950
592
- 582 41.23711340
593
- 583 41.33790738
594
- 584 41.43835616
595
- 585 41.36752137
596
- 586 41.46757679
597
- 587 41.39693356
598
- 588 41.32653061
599
- 589 41.25636672
600
- 590 41.18644068
601
- 591 41.11675127
602
- 592 41.04729730
603
- 593 41.14671164
604
- 594 41.24579125
605
- 595 41.17647059
606
- 596 41.27516779
607
- 597 41.20603015
608
- 598 41.13712375
609
- 599 41.23539232
610
- 600 41.16666667
611
- 601 41.09816972
612
- 602 41.02990033
613
- 603 41.12769486
614
- 604 41.05960265
615
- 605 40.99173554
616
- 606 40.92409241
617
- 607 41.02141680
618
- 608 40.95394737
619
- 609 40.88669951
620
- 610 40.81967213
621
- 611 40.91653028
622
- 612 40.84967320
623
- 613 40.94616639
624
- 614 41.04234528
625
- 615 40.97560976
626
- 616 41.07142857
627
- 617 41.16693679
628
- 618 41.10032362
629
- 619 41.03392569
630
- 620 40.96774194
631
- 621 40.90177134
632
- 622 40.99678457
633
- 623 40.93097913
634
- 624 40.86538462
635
- 625 40.80000000
636
- 626 40.89456869
637
- 627 40.82934609
638
- 628 40.76433121
639
- 629 40.85850556
640
- 630 40.95238095
641
- 631 40.88748019
642
- 632 40.98101266
643
- 633 40.91627172
644
- 634 40.85173502
645
- 635 40.78740157
646
- 636 40.72327044
647
- 637 40.65934066
648
- 638 40.59561129
649
- 639 40.68857590
650
- 640 40.78125000
651
- 641 40.71762871
652
- 642 40.65420561
653
- 643 40.59097978
654
- 644 40.68322981
655
- 645 40.62015504
656
- 646 40.55727554
657
- 647 40.49459042
658
- 648 40.43209877
659
- 649 40.36979969
660
- 650 40.30769231
661
- 651 40.24577573
662
- 652 40.18404908
663
- 653 40.12251149
664
- 654 40.21406728
665
- 655 40.15267176
666
- 656 40.24390244
667
- 657 40.18264840
668
- 658 40.27355623
669
- 659 40.21244310
670
- 660 40.15151515
671
- 661 40.09077156
672
- 662 40.03021148
673
- 663 39.96983409
674
- 664 39.90963855
675
- 665 40.00000000
676
- 666 39.93993994
677
- 667 40.02998501
678
- 668 40.11976048
679
- 669 40.20926756
680
- 670 40.29850746
681
- 671 40.23845007
682
- 672 40.17857143
683
- 673 40.11887073
684
- 674 40.05934718
685
- 675 40.00000000
686
- 676 39.94082840
687
- 677 39.88183161
688
- 678 39.97050147
689
- 679 39.91163476
690
- 680 40.00000000
691
- 681 40.08810573
692
- 682 40.02932551
693
- 683 39.97071742
694
- 684 40.05847953
695
- 685 40.00000000
696
- 686 40.08746356
697
- 687 40.17467249
698
- 688 40.11627907
699
- 689 40.05805515
700
- 690 40.00000000
701
- 691 39.94211288
702
- 692 39.88439306
703
- 693 39.82683983
704
- 694 39.91354467
705
- 695 39.85611511
706
- 696 39.79885057
707
- 697 39.74175036
708
- 698 39.82808023
709
- 699 39.91416309
710
- 700 39.85714286
711
- 701 39.94293866
712
- 702 40.02849003
713
- 703 40.11379801
714
- 704 40.05681818
715
- 705 40.00000000
716
- 706 39.94334278
717
- 707 39.88684583
718
- 708 39.83050847
719
- 709 39.77433004
720
- 710 39.71830986
721
- 711 39.80309423
722
- 712 39.88764045
723
- 713 39.97194951
724
- 714 40.05602241
725
- 715 40.00000000
726
- 716 39.94413408
727
- 717 39.88842399
728
- 718 39.83286908
729
- 719 39.77746871
730
- 720 39.72222222
731
- 721 39.66712899
732
- 722 39.61218837
733
- 723 39.69571231
734
- 724 39.64088398
735
- 725 39.72413793
736
- 726 39.66942149
737
- 727 39.61485557
738
- 728 39.69780220
739
- 729 39.64334705
740
- 730 39.58904110
741
- 731 39.53488372
742
- 732 39.48087432
743
- 733 39.56343793
744
- 734 39.64577657
745
- 735 39.72789116
746
- 736 39.67391304
747
- 737 39.75576662
748
- 738 39.83739837
749
- 739 39.91880920
750
- 740 39.86486486
751
- 741 39.81106613
752
- 742 39.89218329
753
- 743 39.97308210
754
- 744 40.05376344
755
- 745 40.00000000
756
- 746 39.94638070
757
- 747 40.02677376
758
- 748 39.97326203
759
- 749 39.91989319
760
- 750 40.00000000
761
 
762
- Final result: 40.0000 +/- 1.7900
763
- Random chance: 25.0000 +/- 1.5822
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-IQ4_NL.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 39.7333 +/- 1.7880
6
+ Random chance: 25.0000 +/- 1.5822
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 283.49 ms
10
+ llama_perf_context_print: prompt eval time = 260134.32 ms / 72070 tokens ( 3.61 ms per token, 277.05 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 262954.84 ms / 72071 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-iq4_nl.ppx CHANGED
@@ -1,37 +1,37 @@
1
  ====== Perplexity statistics ======
2
- Mean PPL(Q) : 7.990955 ± 0.051527
3
- Mean PPL(base) : 7.534124 ± 0.048206
4
- Cor(ln(PPL(Q)), ln(PPL(base))): 99.03%
5
- Mean ln(PPL(Q)/PPL(base)) : 0.058868 ± 0.000897
6
- Mean PPL(Q)/PPL(base) : 1.060635 ± 0.000951
7
- Mean PPL(Q)-PPL(base) : 0.456831 ± 0.007702
8
 
9
  ====== KL divergence statistics ======
10
- Mean KLD: 0.051341 ± 0.000211
11
- Maximum KLD: 6.532913
12
- 99.9% KLD: 0.841224
13
- 99.0% KLD: 0.299998
14
- 99.0% KLD: 0.299998
15
- Median KLD: 0.038058
16
- 10.0% KLD: 0.002190
17
- 5.0% KLD: 0.000690
18
- 1.0% KLD: 0.000095
19
- Minimum KLD: -0.000027
20
 
21
  ====== Token probability statistics ======
22
- Mean Δp: -1.276 ± 0.017 %
23
- Maximum Δp: 97.255%
24
- 99.9% Δp: 27.515%
25
- 99.0% Δp: 15.140%
26
- 95.0% Δp: 7.440%
27
- 90.0% Δp: 4.054%
28
- 75.0% Δp: 0.520%
29
- Median Δp: -0.129%
30
- 25.0% Δp: -2.686%
31
- 10.0% Δp: -8.215%
32
- 5.0% Δp: -12.358%
33
- 1.0% Δp: -22.757%
34
- 0.1% Δp: -44.994%
35
- Minimum Δp: -85.147%
36
- RMS Δp : 6.512 ± 0.032 %
37
- Same top p: 87.756 ± 0.086 %
 
1
  ====== Perplexity statistics ======
2
+ Mean PPL(Q) : 7.935917 ± 0.053744
3
+ Mean PPL(base) : 7.237090 ± 0.045539
4
+ Cor(ln(PPL(Q)), ln(PPL(base))): 98.36%
5
+ Mean ln(PPL(Q)/PPL(base)) : 0.092180 ± 0.001277
6
+ Mean PPL(Q)/PPL(base) : 1.096562 ± 0.001400
7
+ Mean PPL(Q)-PPL(base) : 0.698827 ± 0.012156
8
 
9
  ====== KL divergence statistics ======
10
+ Mean KLD: 0.096510 ± 0.000325
11
+ Maximum KLD: 5.312946
12
+ 99.9% KLD: 1.205405
13
+ 99.0% KLD: 0.590747
14
+ 99.0% KLD: 0.590747
15
+ Median KLD: 0.069663
16
+ 10.0% KLD: 0.003776
17
+ 5.0% KLD: 0.001002
18
+ 1.0% KLD: 0.000098
19
+ Minimum KLD: -0.000140
20
 
21
  ====== Token probability statistics ======
22
+ Mean Δp: 0.914 ± 0.023 %
23
+ Maximum Δp: 68.385%
24
+ 99.9% Δp: 48.877%
25
+ 99.0% Δp: 30.041%
26
+ 95.0% Δp: 15.934%
27
+ 90.0% Δp: 10.078%
28
+ 75.0% Δp: 2.824%
29
+ Median Δp: 0.026%
30
+ 25.0% Δp: -1.244%
31
+ 10.0% Δp: -6.629%
32
+ 5.0% Δp: -11.886%
33
+ 1.0% Δp: -25.558%
34
+ 0.1% Δp: -49.037%
35
+ Minimum Δp: -91.804%
36
+ RMS Δp : 8.908 ± 0.037 %
37
+ Same top p: 85.211 ± 0.094 %
scores/Watt-Tool-8B-iq4_nl.tqa CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 817 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 817 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 50.00000000
13
- 3 33.33333333
14
- 4 25.00000000
15
- 5 20.00000000
16
- 6 16.66666667
17
- 7 14.28571429
18
- 8 12.50000000
19
- 9 11.11111111
20
- 10 10.00000000
21
- 11 9.09090909
22
- 12 16.66666667
23
- 13 23.07692308
24
- 14 21.42857143
25
- 15 26.66666667
26
- 16 31.25000000
27
- 17 29.41176471
28
- 18 27.77777778
29
- 19 31.57894737
30
- 20 30.00000000
31
- 21 28.57142857
32
- 22 27.27272727
33
- 23 26.08695652
34
- 24 29.16666667
35
- 25 28.00000000
36
- 26 26.92307692
37
- 27 25.92592593
38
- 28 25.00000000
39
- 29 27.58620690
40
- 30 26.66666667
41
- 31 25.80645161
42
- 32 28.12500000
43
- 33 30.30303030
44
- 34 29.41176471
45
- 35 28.57142857
46
- 36 27.77777778
47
- 37 29.72972973
48
- 38 28.94736842
49
- 39 30.76923077
50
- 40 30.00000000
51
- 41 29.26829268
52
- 42 28.57142857
53
- 43 30.23255814
54
- 44 29.54545455
55
- 45 31.11111111
56
- 46 30.43478261
57
- 47 31.91489362
58
- 48 33.33333333
59
- 49 32.65306122
60
- 50 34.00000000
61
- 51 33.33333333
62
- 52 32.69230769
63
- 53 32.07547170
64
- 54 31.48148148
65
- 55 32.72727273
66
- 56 32.14285714
67
- 57 31.57894737
68
- 58 31.03448276
69
- 59 32.20338983
70
- 60 33.33333333
71
- 61 32.78688525
72
- 62 33.87096774
73
- 63 33.33333333
74
- 64 34.37500000
75
- 65 33.84615385
76
- 66 34.84848485
77
- 67 34.32835821
78
- 68 33.82352941
79
- 69 34.78260870
80
- 70 34.28571429
81
- 71 33.80281690
82
- 72 34.72222222
83
- 73 35.61643836
84
- 74 35.13513514
85
- 75 34.66666667
86
- 76 34.21052632
87
- 77 33.76623377
88
- 78 33.33333333
89
- 79 32.91139241
90
- 80 33.75000000
91
- 81 33.33333333
92
- 82 34.14634146
93
- 83 33.73493976
94
- 84 33.33333333
95
- 85 32.94117647
96
- 86 32.55813953
97
- 87 32.18390805
98
- 88 31.81818182
99
- 89 32.58426966
100
- 90 32.22222222
101
- 91 31.86813187
102
- 92 31.52173913
103
- 93 31.18279570
104
- 94 30.85106383
105
- 95 30.52631579
106
- 96 31.25000000
107
- 97 31.95876289
108
- 98 31.63265306
109
- 99 32.32323232
110
- 100 33.00000000
111
- 101 33.66336634
112
- 102 33.33333333
113
- 103 33.00970874
114
- 104 32.69230769
115
- 105 32.38095238
116
- 106 32.07547170
117
- 107 32.71028037
118
- 108 33.33333333
119
- 109 33.02752294
120
- 110 33.63636364
121
- 111 33.33333333
122
- 112 33.92857143
123
- 113 33.62831858
124
- 114 34.21052632
125
- 115 33.91304348
126
- 116 34.48275862
127
- 117 34.18803419
128
- 118 33.89830508
129
- 119 33.61344538
130
- 120 33.33333333
131
- 121 33.88429752
132
- 122 33.60655738
133
- 123 33.33333333
134
- 124 33.87096774
135
- 125 34.40000000
136
- 126 34.12698413
137
- 127 33.85826772
138
- 128 33.59375000
139
- 129 33.33333333
140
- 130 33.84615385
141
- 131 33.58778626
142
- 132 33.33333333
143
- 133 33.08270677
144
- 134 32.83582090
145
- 135 33.33333333
146
- 136 33.08823529
147
- 137 33.57664234
148
- 138 34.05797101
149
- 139 34.53237410
150
- 140 34.28571429
151
- 141 34.04255319
152
- 142 33.80281690
153
- 143 33.56643357
154
- 144 34.02777778
155
- 145 33.79310345
156
- 146 33.56164384
157
- 147 33.33333333
158
- 148 33.10810811
159
- 149 32.88590604
160
- 150 32.66666667
161
- 151 33.11258278
162
- 152 32.89473684
163
- 153 32.67973856
164
- 154 32.46753247
165
- 155 32.25806452
166
- 156 32.69230769
167
- 157 32.48407643
168
- 158 32.91139241
169
- 159 33.33333333
170
- 160 33.12500000
171
- 161 33.54037267
172
- 162 33.95061728
173
- 163 33.74233129
174
- 164 33.53658537
175
- 165 33.93939394
176
- 166 34.33734940
177
- 167 34.13173653
178
- 168 33.92857143
179
- 169 33.72781065
180
- 170 33.52941176
181
- 171 33.33333333
182
- 172 33.13953488
183
- 173 32.94797688
184
- 174 33.33333333
185
- 175 33.71428571
186
- 176 34.09090909
187
- 177 33.89830508
188
- 178 33.70786517
189
- 179 34.07821229
190
- 180 34.44444444
191
- 181 34.25414365
192
- 182 34.06593407
193
- 183 33.87978142
194
- 184 33.69565217
195
- 185 33.51351351
196
- 186 33.33333333
197
- 187 33.15508021
198
- 188 32.97872340
199
- 189 33.33333333
200
- 190 33.68421053
201
- 191 33.50785340
202
- 192 33.33333333
203
- 193 33.16062176
204
- 194 33.50515464
205
- 195 33.33333333
206
- 196 33.16326531
207
- 197 33.50253807
208
- 198 33.83838384
209
- 199 33.66834171
210
- 200 34.00000000
211
- 201 33.83084577
212
- 202 34.15841584
213
- 203 33.99014778
214
- 204 33.82352941
215
- 205 33.65853659
216
- 206 33.49514563
217
- 207 33.81642512
218
- 208 33.65384615
219
- 209 33.49282297
220
- 210 33.33333333
221
- 211 33.17535545
222
- 212 33.01886792
223
- 213 32.86384977
224
- 214 32.71028037
225
- 215 32.55813953
226
- 216 32.87037037
227
- 217 33.17972350
228
- 218 33.48623853
229
- 219 33.33333333
230
- 220 33.18181818
231
- 221 33.03167421
232
- 222 32.88288288
233
- 223 33.18385650
234
- 224 33.48214286
235
- 225 33.33333333
236
- 226 33.18584071
237
- 227 33.48017621
238
- 228 33.33333333
239
- 229 33.18777293
240
- 230 33.04347826
241
- 231 32.90043290
242
- 232 32.75862069
243
- 233 32.61802575
244
- 234 32.90598291
245
- 235 32.76595745
246
- 236 33.05084746
247
- 237 33.33333333
248
- 238 33.61344538
249
- 239 33.47280335
250
- 240 33.33333333
251
- 241 33.19502075
252
- 242 33.05785124
253
- 243 32.92181070
254
- 244 32.78688525
255
- 245 32.65306122
256
- 246 32.52032520
257
- 247 32.79352227
258
- 248 32.66129032
259
- 249 32.93172691
260
- 250 32.80000000
261
- 251 33.06772908
262
- 252 33.33333333
263
- 253 33.20158103
264
- 254 33.07086614
265
- 255 32.94117647
266
- 256 32.81250000
267
- 257 32.68482490
268
- 258 32.55813953
269
- 259 32.43243243
270
- 260 32.30769231
271
- 261 32.56704981
272
- 262 32.44274809
273
- 263 32.31939163
274
- 264 32.57575758
275
- 265 32.83018868
276
- 266 32.70676692
277
- 267 32.58426966
278
- 268 32.46268657
279
- 269 32.34200743
280
- 270 32.22222222
281
- 271 32.10332103
282
- 272 31.98529412
283
- 273 32.23443223
284
- 274 32.11678832
285
- 275 32.00000000
286
- 276 31.88405797
287
- 277 31.76895307
288
- 278 31.65467626
289
- 279 31.54121864
290
- 280 31.78571429
291
- 281 32.02846975
292
- 282 31.91489362
293
- 283 32.15547703
294
- 284 32.39436620
295
- 285 32.28070175
296
- 286 32.51748252
297
- 287 32.75261324
298
- 288 32.98611111
299
- 289 32.87197232
300
- 290 32.75862069
301
- 291 32.98969072
302
- 292 32.87671233
303
- 293 33.10580205
304
- 294 32.99319728
305
- 295 33.22033898
306
- 296 33.44594595
307
- 297 33.33333333
308
- 298 33.22147651
309
- 299 33.11036789
310
- 300 33.00000000
311
- 301 32.89036545
312
- 302 32.78145695
313
- 303 32.67326733
314
- 304 32.56578947
315
- 305 32.45901639
316
- 306 32.35294118
317
- 307 32.24755700
318
- 308 32.46753247
319
- 309 32.36245955
320
- 310 32.25806452
321
- 311 32.15434084
322
- 312 32.37179487
323
- 313 32.26837061
324
- 314 32.16560510
325
- 315 32.06349206
326
- 316 31.96202532
327
- 317 31.86119874
328
- 318 31.76100629
329
- 319 31.66144201
330
- 320 31.56250000
331
- 321 31.46417445
332
- 322 31.67701863
333
- 323 31.57894737
334
- 324 31.48148148
335
- 325 31.38461538
336
- 326 31.28834356
337
- 327 31.49847095
338
- 328 31.70731707
339
- 329 31.61094225
340
- 330 31.81818182
341
- 331 31.72205438
342
- 332 31.92771084
343
- 333 31.83183183
344
- 334 31.73652695
345
- 335 31.94029851
346
- 336 32.14285714
347
- 337 32.04747774
348
- 338 31.95266272
349
- 339 32.15339233
350
- 340 32.05882353
351
- 341 32.25806452
352
- 342 32.45614035
353
- 343 32.36151603
354
- 344 32.55813953
355
- 345 32.75362319
356
- 346 32.65895954
357
- 347 32.85302594
358
- 348 32.75862069
359
- 349 32.66475645
360
- 350 32.57142857
361
- 351 32.47863248
362
- 352 32.67045455
363
- 353 32.57790368
364
- 354 32.48587571
365
- 355 32.67605634
366
- 356 32.58426966
367
- 357 32.77310924
368
- 358 32.96089385
369
- 359 32.86908078
370
- 360 32.77777778
371
- 361 32.96398892
372
- 362 32.87292818
373
- 363 33.05785124
374
- 364 32.96703297
375
- 365 32.87671233
376
- 366 32.78688525
377
- 367 32.97002725
378
- 368 32.88043478
379
- 369 32.79132791
380
- 370 32.70270270
381
- 371 32.61455526
382
- 372 32.79569892
383
- 373 32.70777480
384
- 374 32.88770053
385
- 375 32.80000000
386
- 376 32.71276596
387
- 377 32.89124668
388
- 378 32.80423280
389
- 379 32.71767810
390
- 380 32.89473684
391
- 381 32.80839895
392
- 382 32.72251309
393
- 383 32.63707572
394
- 384 32.55208333
395
- 385 32.46753247
396
- 386 32.38341969
397
- 387 32.29974160
398
- 388 32.47422680
399
- 389 32.39074550
400
- 390 32.30769231
401
- 391 32.48081841
402
- 392 32.39795918
403
- 393 32.56997455
404
- 394 32.74111675
405
- 395 32.65822785
406
- 396 32.82828283
407
- 397 32.99748111
408
- 398 32.91457286
409
- 399 32.83208020
410
- 400 32.75000000
411
- 401 32.91770574
412
- 402 33.08457711
413
- 403 33.25062035
414
- 404 33.16831683
415
- 405 33.08641975
416
- 406 33.00492611
417
- 407 32.92383292
418
- 408 32.84313725
419
- 409 32.76283619
420
- 410 32.68292683
421
- 411 32.60340633
422
- 412 32.76699029
423
- 413 32.68765133
424
- 414 32.85024155
425
- 415 33.01204819
426
- 416 32.93269231
427
- 417 32.85371703
428
- 418 32.77511962
429
- 419 32.69689737
430
- 420 32.85714286
431
- 421 33.01662708
432
- 422 32.93838863
433
- 423 32.86052009
434
- 424 32.78301887
435
- 425 32.70588235
436
- 426 32.62910798
437
- 427 32.55269321
438
- 428 32.47663551
439
- 429 32.40093240
440
- 430 32.32558140
441
- 431 32.25058005
442
- 432 32.40740741
443
- 433 32.56351039
444
- 434 32.71889401
445
- 435 32.64367816
446
- 436 32.56880734
447
- 437 32.49427918
448
- 438 32.64840183
449
- 439 32.80182232
450
- 440 32.72727273
451
- 441 32.65306122
452
- 442 32.57918552
453
- 443 32.50564334
454
- 444 32.65765766
455
- 445 32.58426966
456
- 446 32.51121076
457
- 447 32.43847875
458
- 448 32.36607143
459
- 449 32.51670379
460
- 450 32.66666667
461
- 451 32.59423503
462
- 452 32.52212389
463
- 453 32.67108168
464
- 454 32.59911894
465
- 455 32.52747253
466
- 456 32.45614035
467
- 457 32.38512035
468
- 458 32.31441048
469
- 459 32.24400871
470
- 460 32.17391304
471
- 461 32.10412148
472
- 462 32.25108225
473
- 463 32.39740821
474
- 464 32.54310345
475
- 465 32.68817204
476
- 466 32.61802575
477
- 467 32.54817987
478
- 468 32.69230769
479
- 469 32.62260128
480
- 470 32.55319149
481
- 471 32.69639066
482
- 472 32.83898305
483
- 473 32.76955603
484
- 474 32.70042194
485
- 475 32.84210526
486
- 476 32.77310924
487
- 477 32.70440252
488
- 478 32.63598326
489
- 479 32.56784969
490
- 480 32.50000000
491
- 481 32.43243243
492
- 482 32.57261411
493
- 483 32.50517598
494
- 484 32.64462810
495
- 485 32.78350515
496
- 486 32.71604938
497
- 487 32.85420945
498
- 488 32.78688525
499
- 489 32.92433538
500
- 490 33.06122449
501
- 491 33.19755601
502
- 492 33.33333333
503
- 493 33.26572008
504
- 494 33.40080972
505
- 495 33.33333333
506
- 496 33.26612903
507
- 497 33.19919517
508
- 498 33.13253012
509
- 499 33.06613226
510
- 500 33.00000000
511
- 501 32.93413174
512
- 502 32.86852590
513
- 503 32.80318091
514
- 504 32.93650794
515
- 505 32.87128713
516
- 506 32.80632411
517
- 507 32.93885602
518
- 508 33.07086614
519
- 509 33.20235756
520
- 510 33.33333333
521
- 511 33.26810176
522
- 512 33.20312500
523
- 513 33.13840156
524
- 514 33.07392996
525
- 515 33.20388350
526
- 516 33.13953488
527
- 517 33.26885880
528
- 518 33.20463320
529
- 519 33.14065511
530
- 520 33.07692308
531
- 521 33.01343570
532
- 522 32.95019157
533
- 523 32.88718929
534
- 524 32.82442748
535
- 525 32.76190476
536
- 526 32.69961977
537
- 527 32.63757116
538
- 528 32.57575758
539
- 529 32.70321361
540
- 530 32.64150943
541
- 531 32.58003766
542
- 532 32.51879699
543
- 533 32.64540338
544
- 534 32.58426966
545
- 535 32.52336449
546
- 536 32.46268657
547
- 537 32.58845438
548
- 538 32.52788104
549
- 539 32.46753247
550
- 540 32.40740741
551
- 541 32.34750462
552
- 542 32.28782288
553
- 543 32.22836096
554
- 544 32.16911765
555
- 545 32.11009174
556
- 546 32.05128205
557
- 547 32.17550274
558
- 548 32.29927007
559
- 549 32.24043716
560
- 550 32.18181818
561
- 551 32.12341198
562
- 552 32.24637681
563
- 553 32.36889693
564
- 554 32.31046931
565
- 555 32.25225225
566
- 556 32.19424460
567
- 557 32.13644524
568
- 558 32.07885305
569
- 559 32.02146691
570
- 560 32.14285714
571
- 561 32.26381462
572
- 562 32.20640569
573
- 563 32.14920071
574
- 564 32.09219858
575
- 565 32.21238938
576
- 566 32.33215548
577
- 567 32.27513228
578
- 568 32.39436620
579
- 569 32.51318102
580
- 570 32.45614035
581
- 571 32.57443082
582
- 572 32.51748252
583
- 573 32.46073298
584
- 574 32.57839721
585
- 575 32.69565217
586
- 576 32.81250000
587
- 577 32.92894281
588
- 578 33.04498270
589
- 579 33.16062176
590
- 580 33.27586207
591
- 581 33.21858864
592
- 582 33.33333333
593
- 583 33.27615780
594
- 584 33.39041096
595
- 585 33.33333333
596
- 586 33.44709898
597
- 587 33.39011925
598
- 588 33.33333333
599
- 589 33.27674024
600
- 590 33.22033898
601
- 591 33.16412860
602
- 592 33.10810811
603
- 593 33.22091062
604
- 594 33.33333333
605
- 595 33.27731092
606
- 596 33.22147651
607
- 597 33.16582915
608
- 598 33.11036789
609
- 599 33.05509182
610
- 600 33.00000000
611
- 601 33.11148087
612
- 602 33.22259136
613
- 603 33.33333333
614
- 604 33.27814570
615
- 605 33.22314050
616
- 606 33.33333333
617
- 607 33.27841845
618
- 608 33.22368421
619
- 609 33.16912972
620
- 610 33.11475410
621
- 611 33.06055646
622
- 612 33.16993464
623
- 613 33.11582382
624
- 614 33.06188925
625
- 615 33.00813008
626
- 616 33.11688312
627
- 617 33.22528363
628
- 618 33.17152104
629
- 619 33.27948304
630
- 620 33.22580645
631
- 621 33.17230274
632
- 622 33.27974277
633
- 623 33.22632424
634
- 624 33.17307692
635
- 625 33.12000000
636
- 626 33.06709265
637
- 627 33.17384370
638
- 628 33.28025478
639
- 629 33.22734499
640
- 630 33.17460317
641
- 631 33.28050713
642
- 632 33.22784810
643
- 633 33.17535545
644
- 634 33.28075710
645
- 635 33.22834646
646
- 636 33.17610063
647
- 637 33.12401884
648
- 638 33.07210031
649
- 639 33.02034429
650
- 640 33.12500000
651
- 641 33.07332293
652
- 642 33.17757009
653
- 643 33.12597201
654
- 644 33.07453416
655
- 645 33.17829457
656
- 646 33.12693498
657
- 647 33.23029366
658
- 648 33.33333333
659
- 649 33.28197227
660
- 650 33.38461538
661
- 651 33.48694316
662
- 652 33.58895706
663
- 653 33.69065850
664
- 654 33.63914373
665
- 655 33.58778626
666
- 656 33.68902439
667
- 657 33.78995434
668
- 658 33.73860182
669
- 659 33.83915023
670
- 660 33.78787879
671
- 661 33.73676248
672
- 662 33.68580060
673
- 663 33.63499246
674
- 664 33.58433735
675
- 665 33.68421053
676
- 666 33.78378378
677
- 667 33.88305847
678
- 668 33.83233533
679
- 669 33.78176383
680
- 670 33.88059701
681
- 671 33.83010432
682
- 672 33.77976190
683
- 673 33.72956909
684
- 674 33.67952522
685
- 675 33.77777778
686
- 676 33.87573964
687
- 677 33.82570162
688
- 678 33.77581121
689
- 679 33.72606775
690
- 680 33.67647059
691
- 681 33.62701909
692
- 682 33.72434018
693
- 683 33.67496340
694
- 684 33.77192982
695
- 685 33.72262774
696
- 686 33.67346939
697
- 687 33.62445415
698
- 688 33.57558140
699
- 689 33.67198839
700
- 690 33.62318841
701
- 691 33.71924747
702
- 692 33.81502890
703
- 693 33.76623377
704
- 694 33.71757925
705
- 695 33.66906475
706
- 696 33.76436782
707
- 697 33.71592539
708
- 698 33.66762178
709
- 699 33.61945637
710
- 700 33.71428571
711
- 701 33.66619116
712
- 702 33.61823362
713
- 703 33.57041252
714
- 704 33.52272727
715
- 705 33.47517730
716
- 706 33.56940510
717
- 707 33.66336634
718
- 708 33.61581921
719
- 709 33.56840621
720
- 710 33.52112676
721
- 711 33.47398031
722
- 712 33.56741573
723
- 713 33.52033661
724
- 714 33.47338936
725
- 715 33.56643357
726
- 716 33.51955307
727
- 717 33.61227336
728
- 718 33.56545961
729
- 719 33.65785814
730
- 720 33.61111111
731
- 721 33.56449376
732
- 722 33.51800554
733
- 723 33.47164592
734
- 724 33.42541436
735
- 725 33.37931034
736
- 726 33.47107438
737
- 727 33.42503439
738
- 728 33.37912088
739
- 729 33.47050754
740
- 730 33.56164384
741
- 731 33.51573187
742
- 732 33.60655738
743
- 733 33.56070941
744
- 734 33.51498638
745
- 735 33.46938776
746
- 736 33.42391304
747
- 737 33.51424695
748
- 738 33.46883469
749
- 739 33.42354533
750
- 740 33.37837838
751
- 741 33.46828610
752
- 742 33.42318059
753
- 743 33.37819650
754
- 744 33.46774194
755
- 745 33.42281879
756
- 746 33.51206434
757
- 747 33.60107095
758
- 748 33.55614973
759
- 749 33.64485981
760
- 750 33.73333333
761
 
762
- Final result: 33.7333 +/- 1.7276
763
- Random chance: 19.8992 +/- 1.4588
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-IQ4_NL.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 33.9564 +/- 2.6473
6
+ Random chance: 28.5587 +/- 2.5250
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 300.67 ms
10
+ llama_perf_context_print: prompt eval time = 74415.31 ms / 17418 tokens ( 4.27 ms per token, 234.06 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 75640.51 ms / 17419 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-iq4_nl.wng CHANGED
@@ -1,760 +1,11 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- winogrande_score : loaded 1267 tasks from prompt.
6
- winogrande_score : selecting 750 random tasks
7
- winogrande_score : tokenizing selected tasks
8
- winogrande_score : calculating winogrande score over selected tasks.
9
- 1 100.0000 -2.830675 -3.941673 1 1
10
- 2 100.0000 -2.061954 -2.199474 1 1
11
- 3 100.0000 -3.355775 -2.817899 2 2
12
- 4 75.0000 -4.900475 -4.824972 2 1
13
- 5 60.0000 -2.649025 -3.052794 1 2
14
- 6 50.0000 -4.911099 -4.846340 2 1
15
- 7 57.1429 -4.170405 -2.531311 2 2
16
- 8 62.5000 -2.570491 -2.074745 2 2
17
- 9 66.6667 -3.870417 -5.373734 1 1
18
- 10 70.0000 -1.878735 -1.723521 2 2
19
- 11 72.7273 -9.629203 -5.279615 2 2
20
- 12 75.0000 -2.899102 -1.629077 2 2
21
- 13 76.9231 -3.097030 -5.123330 1 1
22
- 14 78.5714 -3.195134 -3.805247 1 1
23
- 15 73.3333 -3.628571 -5.807333 1 2
24
- 16 75.0000 -3.420616 -3.066670 2 2
25
- 17 70.5882 -6.901192 -7.089924 1 2
26
- 18 72.2222 -1.088278 -3.144958 1 1
27
- 19 73.6842 -3.187990 -5.313015 1 1
28
- 20 75.0000 -2.941244 -3.193235 1 1
29
- 21 76.1905 -1.705717 -3.269566 1 1
30
- 22 77.2727 -2.432213 -2.420577 2 2
31
- 23 78.2609 -4.514483 -4.653703 1 1
32
- 24 75.0000 -2.501765 -2.284458 2 1
33
- 25 72.0000 -2.089951 -2.208999 1 2
34
- 26 73.0769 -3.467419 -4.280454 1 1
35
- 27 70.3704 -4.713809 -4.975879 1 2
36
- 28 71.4286 -4.453796 -4.689594 1 1
37
- 29 68.9655 -3.799340 -3.777359 2 1
38
- 30 70.0000 -2.097255 -2.055520 2 2
39
- 31 70.9677 -4.817040 -5.108164 1 1
40
- 32 71.8750 -4.144757 -5.895689 1 1
41
- 33 72.7273 -4.289439 -5.541212 1 1
42
- 34 70.5882 -2.332751 -2.311998 2 1
43
- 35 68.5714 -4.745343 -4.773489 1 2
44
- 36 69.4444 -3.715028 -3.614710 2 2
45
- 37 70.2703 -3.206895 -3.413583 1 1
46
- 38 68.4211 -3.699581 -3.664534 2 1
47
- 39 69.2308 -5.775852 -4.918131 2 2
48
- 40 67.5000 -4.586945 -5.560170 1 2
49
- 41 68.2927 -2.026350 -2.513564 1 1
50
- 42 69.0476 -1.929358 -3.277385 1 1
51
- 43 67.4419 -3.215272 -3.266553 1 2
52
- 44 68.1818 -2.139297 -1.526763 2 2
53
- 45 68.8889 -0.101506 -12.453684 1 1
54
- 46 67.3913 -2.618230 -2.599678 2 1
55
- 47 65.9574 -3.237711 -3.317803 1 2
56
- 48 66.6667 -4.026316 -3.787686 2 2
57
- 49 65.3061 -3.022687 -1.339004 2 1
58
- 50 64.0000 -2.376312 -2.409117 1 2
59
- 51 64.7059 -2.298282 -3.112837 1 1
60
- 52 65.3846 -3.164985 -1.853093 2 2
61
- 53 66.0377 -3.204421 -3.846365 1 1
62
- 54 66.6667 -2.636727 -1.971077 2 2
63
- 55 67.2727 -2.819647 -2.413808 2 2
64
- 56 66.0714 -3.811858 -4.836819 1 2
65
- 57 66.6667 -3.217517 -2.828481 2 2
66
- 58 67.2414 -3.378184 -4.266493 1 1
67
- 59 67.7966 -3.248554 -3.206690 2 2
68
- 60 68.3333 -3.973295 -3.636571 2 2
69
- 61 68.8525 -3.330601 -3.728400 1 1
70
- 62 69.3548 -5.697174 -6.115126 1 1
71
- 63 68.2540 -4.325087 -4.479603 1 2
72
- 64 67.1875 -3.944867 -3.780811 2 1
73
- 65 67.6923 -5.231625 -4.854895 2 2
74
- 66 68.1818 -2.751302 -3.115241 1 1
75
- 67 68.6567 -3.227085 -2.699860 2 2
76
- 68 67.6471 -5.270287 -5.510540 1 2
77
- 69 68.1159 -5.648033 -7.037736 1 1
78
- 70 68.5714 -3.567181 -3.310087 2 2
79
- 71 67.6056 -3.013945 -2.327962 2 1
80
- 72 66.6667 -3.626065 -3.645537 1 2
81
- 73 67.1233 -2.680456 -2.835059 1 1
82
- 74 67.5676 -2.402092 -2.938369 1 1
83
- 75 68.0000 -1.984865 -2.685462 1 1
84
- 76 68.4211 -2.075246 -4.534829 1 1
85
- 77 68.8312 -3.517473 -2.739412 2 2
86
- 78 69.2308 -0.789226 -1.482932 1 1
87
- 79 68.3544 -2.095829 -2.484662 1 2
88
- 80 68.7500 -2.301425 -3.562987 1 1
89
- 81 69.1358 -5.233383 -4.112867 2 2
90
- 82 68.2927 -3.105363 -3.359093 1 2
91
- 83 68.6747 -5.866801 -4.701226 2 2
92
- 84 69.0476 -2.364064 -2.189471 2 2
93
- 85 69.4118 -5.109601 -5.513272 1 1
94
- 86 68.6047 -5.974204 -6.207721 1 2
95
- 87 68.9655 -1.622011 -1.634683 1 1
96
- 88 68.1818 -4.541931 -4.083561 2 1
97
- 89 68.5393 -3.347734 -3.575997 1 1
98
- 90 68.8889 -4.715583 -3.537085 2 2
99
- 91 68.1319 -4.785750 -4.462862 2 1
100
- 92 68.4783 -5.027726 -4.237718 2 2
101
- 93 68.8172 -2.334241 -2.195620 2 2
102
- 94 69.1489 -4.634887 -4.152183 2 2
103
- 95 69.4737 -3.915092 -4.378292 1 1
104
- 96 69.7917 -1.470951 -2.625960 1 1
105
- 97 69.0722 -5.362595 -5.124524 2 1
106
- 98 69.3878 -3.980508 -3.569432 2 2
107
- 99 68.6869 -2.914405 -2.840211 2 1
108
- 100 69.0000 -5.663173 -5.037541 2 2
109
- 101 68.3168 -4.616508 -4.744431 1 2
110
- 102 68.6275 -3.635138 -4.095443 1 1
111
- 103 68.9320 -3.103400 -2.361967 2 2
112
- 104 69.2308 -7.303843 -7.256544 2 2
113
- 105 69.5238 -1.100734 -2.208578 1 1
114
- 106 69.8113 -2.849782 -2.301137 2 2
115
- 107 70.0935 -2.890924 -2.768275 2 2
116
- 108 70.3704 -4.105050 -3.097357 2 2
117
- 109 69.7248 -4.773708 -5.323071 1 2
118
- 110 70.0000 -4.424457 -3.702542 2 2
119
- 111 70.2703 -3.029452 -2.669752 2 2
120
- 112 69.6429 -3.506168 -3.167754 2 1
121
- 113 69.0265 -4.294383 -5.275711 1 2
122
- 114 69.2982 -3.965883 -4.895136 1 1
123
- 115 69.5652 -1.399476 -2.227950 1 1
124
- 116 69.8276 -2.967876 -2.277220 2 2
125
- 117 69.2308 -4.866026 -5.200023 1 2
126
- 118 69.4915 -1.458950 -2.414563 1 1
127
- 119 69.7479 -1.799379 -2.834760 1 1
128
- 120 69.1667 -3.254340 -2.834539 2 1
129
- 121 69.4215 -1.101832 -0.865706 2 2
130
- 122 69.6721 -6.971323 -5.891404 2 2
131
- 123 69.9187 -3.843635 -4.282731 1 1
132
- 124 70.1613 -2.703189 -2.544167 2 2
133
- 125 70.4000 -2.529926 -2.318540 2 2
134
- 126 70.6349 -6.684674 -6.540388 2 2
135
- 127 70.8661 -2.556489 -1.519692 2 2
136
- 128 71.0938 -2.034021 -2.213632 1 1
137
- 129 71.3178 -2.410480 -1.742737 2 2
138
- 130 71.5385 -5.585640 -4.729264 2 2
139
- 131 71.7557 -3.771631 -3.535952 2 2
140
- 132 71.9697 -2.754056 -2.259092 2 2
141
- 133 72.1805 -1.572946 -3.781993 1 1
142
- 134 72.3881 -5.235231 -4.469508 2 2
143
- 135 72.5926 -4.606221 -5.446578 1 1
144
- 136 72.7941 -4.026054 -2.897658 2 2
145
- 137 72.9927 -5.342050 -5.673006 1 1
146
- 138 73.1884 -4.294590 -4.157501 2 2
147
- 139 73.3813 -3.808771 -4.586473 1 1
148
- 140 73.5714 -1.304468 -1.071936 2 2
149
- 141 73.7589 -5.825905 -4.008801 2 2
150
- 142 73.9437 -3.158239 -2.207273 2 2
151
- 143 74.1259 -3.549173 -6.603521 1 1
152
- 144 74.3056 -2.036648 -2.058387 1 1
153
- 145 73.7931 -1.549567 -14.311398 1 2
154
- 146 73.2877 -3.728300 -3.809523 1 2
155
- 147 73.4694 -4.847789 -4.020986 2 2
156
- 148 73.6486 -2.073982 -2.680964 1 1
157
- 149 73.1544 -3.699376 -3.860786 1 2
158
- 150 73.3333 -4.327263 -3.412598 2 2
159
- 151 72.8477 -1.995331 -1.965114 2 1
160
- 152 72.3684 -3.815183 -4.094164 1 2
161
- 153 72.5490 -2.398588 -3.206105 1 1
162
- 154 72.7273 -3.287839 -3.632639 1 1
163
- 155 72.2581 -2.569896 -3.071654 1 2
164
- 156 72.4359 -2.103709 -2.408102 1 1
165
- 157 72.6115 -2.978404 -4.283286 1 1
166
- 158 72.7848 -4.713658 -4.332416 2 2
167
- 159 72.9560 -3.379546 -4.356255 1 1
168
- 160 73.1250 -3.363397 -3.084490 2 2
169
- 161 73.2919 -2.174735 -2.576679 1 1
170
- 162 72.8395 -0.053693 -11.131796 1 2
171
- 163 73.0061 -1.627976 -1.914218 1 1
172
- 164 73.1707 -5.729863 -4.484262 2 2
173
- 165 72.7273 -4.105697 -4.125713 1 2
174
- 166 72.8916 -4.622510 -3.988583 2 2
175
- 167 72.4551 -4.337939 -3.966908 2 1
176
- 168 72.6190 -5.226591 -4.097583 2 2
177
- 169 72.1893 -3.535910 -4.237757 1 2
178
- 170 72.3529 -2.389850 -2.882176 1 1
179
- 171 72.5146 -3.748956 -4.847082 1 1
180
- 172 72.6744 -3.235588 -4.247825 1 1
181
- 173 72.8324 -2.846079 -2.727115 2 2
182
- 174 72.9885 -5.360032 -5.160922 2 2
183
- 175 73.1429 -4.019013 -4.186170 1 1
184
- 176 73.2955 -2.158002 -3.736483 1 1
185
- 177 73.4463 -5.575706 -5.545458 2 2
186
- 178 73.0337 -2.919030 -2.765315 2 1
187
- 179 73.1844 -2.447016 -2.852178 1 1
188
- 180 73.3333 -5.242766 -4.906011 2 2
189
- 181 73.4807 -3.561336 -2.525043 2 2
190
- 182 73.6264 -3.921040 -2.676193 2 2
191
- 183 73.7705 -5.426070 -5.349180 2 2
192
- 184 73.9130 -1.569631 -2.806191 1 1
193
- 185 73.5135 -4.446148 -4.644056 1 2
194
- 186 73.6559 -2.733779 -3.407808 1 1
195
- 187 73.7968 -5.352100 -4.513143 2 2
196
- 188 73.4043 -4.584832 -3.330668 2 1
197
- 189 73.0159 -4.272583 -4.745075 1 2
198
- 190 73.1579 -4.606952 -4.574039 2 2
199
- 191 73.2984 -2.603557 -3.043170 1 1
200
- 192 73.4375 -4.968874 -4.377175 2 2
201
- 193 73.5751 -3.978384 -3.926218 2 2
202
- 194 73.7113 -1.784765 -1.537702 2 2
203
- 195 73.3333 -3.194351 -3.067529 2 1
204
- 196 72.9592 -3.286875 -3.615645 1 2
205
- 197 73.0964 -1.005818 -3.764204 1 1
206
- 198 73.2323 -4.334623 -7.305194 1 1
207
- 199 73.3668 -4.998895 -4.159181 2 2
208
- 200 73.5000 -3.063681 -2.838114 2 2
209
- 201 73.6318 -3.016357 -3.283758 1 1
210
- 202 73.7624 -8.223533 -7.811516 2 2
211
- 203 73.8916 -3.262570 -3.018777 2 2
212
- 204 74.0196 -2.416231 -3.229268 1 1
213
- 205 74.1463 -3.489873 -2.743218 2 2
214
- 206 73.7864 -4.474365 -4.951780 1 2
215
- 207 73.9130 -3.080745 -3.385201 1 1
216
- 208 74.0385 -3.484128 -2.531397 2 2
217
- 209 74.1627 -3.946151 -3.357706 2 2
218
- 210 74.2857 -6.750890 -6.733737 2 2
219
- 211 73.9336 -5.533973 -5.357141 2 1
220
- 212 74.0566 -5.054101 -4.205606 2 2
221
- 213 73.7089 -3.672145 -5.080173 1 2
222
- 214 73.3645 -2.903055 -2.794860 2 1
223
- 215 73.4884 -4.591434 -4.420226 2 2
224
- 216 73.1481 -3.371099 -3.254742 2 1
225
- 217 73.2719 -1.073941 -1.692958 1 1
226
- 218 73.3945 -3.189541 -2.721104 2 2
227
- 219 73.0594 -3.152003 -2.971987 2 1
228
- 220 72.7273 -3.258331 -3.560826 1 2
229
- 221 72.8507 -5.293450 -9.400375 1 1
230
- 222 72.5225 -0.250929 -8.695187 1 2
231
- 223 72.6457 -3.840676 -3.264467 2 2
232
- 224 72.7679 -6.301306 -4.687080 2 2
233
- 225 72.4444 -2.922806 -3.071722 1 2
234
- 226 72.5664 -2.325580 -3.364507 1 1
235
- 227 72.6872 -3.062913 -3.306427 1 1
236
- 228 72.8070 -1.443471 -2.491328 1 1
237
- 229 72.9258 -3.468754 -3.288135 2 2
238
- 230 73.0435 -3.164336 -3.546242 1 1
239
- 231 73.1602 -2.344953 -2.808439 1 1
240
- 232 73.2759 -2.981562 -2.349851 2 2
241
- 233 73.3906 -1.642088 -2.253488 1 1
242
- 234 73.5043 -3.357635 -3.018413 2 2
243
- 235 73.6170 -3.165408 -2.974583 2 2
244
- 236 73.3051 -2.143413 -1.810812 2 1
245
- 237 73.4177 -2.374904 -2.428030 1 1
246
- 238 73.5294 -3.134026 -2.812371 2 2
247
- 239 73.2218 -2.688086 -2.767568 1 2
248
- 240 72.9167 -6.815941 -6.860892 1 2
249
- 241 72.6141 -2.818256 -3.504025 1 2
250
- 242 72.7273 -5.916381 -5.489482 2 2
251
- 243 72.4280 -4.595194 -4.220147 2 1
252
- 244 72.5410 -2.636791 -4.285039 1 1
253
- 245 72.6531 -4.134146 -4.946243 1 1
254
- 246 72.7642 -2.637845 -3.140551 1 1
255
- 247 72.8745 -3.363490 -3.181943 2 2
256
- 248 72.9839 -2.869668 -2.531408 2 2
257
- 249 73.0924 -3.147039 -2.158013 2 2
258
- 250 73.2000 -2.448039 -3.037428 1 1
259
- 251 73.3068 -1.455766 -1.911787 1 1
260
- 252 73.4127 -6.394641 -5.796336 2 2
261
- 253 73.1225 -5.582470 -5.196656 2 1
262
- 254 73.2283 -4.837812 -4.211230 2 2
263
- 255 73.3333 -3.360677 -4.497277 1 1
264
- 256 73.0469 -3.485765 -3.150474 2 1
265
- 257 72.7626 -2.495196 -1.885140 2 1
266
- 258 72.4806 -3.529618 -3.740626 1 2
267
- 259 72.5869 -5.133690 -5.492273 1 1
268
- 260 72.3077 -5.980809 -5.907661 2 1
269
- 261 72.0307 -3.617036 -3.738911 1 2
270
- 262 72.1374 -5.143147 -4.679624 2 2
271
- 263 72.2433 -3.540560 -3.998374 1 1
272
- 264 72.3485 -4.275608 -3.900524 2 2
273
- 265 72.4528 -1.143584 -2.335167 1 1
274
- 266 72.5564 -4.786168 -5.787732 1 1
275
- 267 72.2846 -3.320943 -2.825491 2 1
276
- 268 72.0149 -2.959198 -2.950714 2 1
277
- 269 72.1190 -2.864025 -2.705706 2 2
278
- 270 72.2222 -2.367183 -3.389299 1 1
279
- 271 72.3247 -2.736540 -2.631493 2 2
280
- 272 72.4265 -3.874883 -3.490861 2 2
281
- 273 72.5275 -3.526004 -3.427820 2 2
282
- 274 72.6277 -3.511436 -3.226444 2 2
283
- 275 72.7273 -2.506452 -2.752770 1 1
284
- 276 72.4638 -2.590999 -2.461828 2 1
285
- 277 72.5632 -3.177670 -3.984281 1 1
286
- 278 72.3022 -2.762772 -2.666787 2 1
287
- 279 72.4014 -5.010150 -3.540094 2 2
288
- 280 72.5000 -3.781436 -3.311634 2 2
289
- 281 72.5979 -2.393726 -1.515618 2 2
290
- 282 72.6950 -4.073411 -3.528929 2 2
291
- 283 72.4382 -3.330749 -3.418035 1 2
292
- 284 72.5352 -2.351225 -3.265712 1 1
293
- 285 72.6316 -3.259357 -4.762096 1 1
294
- 286 72.7273 -3.694923 -3.737571 1 1
295
- 287 72.8223 -2.837430 -4.627598 1 1
296
- 288 72.9167 -3.108994 -2.549959 2 2
297
- 289 73.0104 -1.794721 -2.399608 1 1
298
- 290 73.1034 -3.960924 -4.768822 1 1
299
- 291 73.1959 -1.976306 -2.720532 1 1
300
- 292 73.2877 -2.826163 -1.883800 2 2
301
- 293 73.3788 -3.336426 -2.831367 2 2
302
- 294 73.1293 -4.287539 -3.606570 2 1
303
- 295 73.2203 -1.679554 -1.776792 1 1
304
- 296 73.3108 -5.058667 -4.845234 2 2
305
- 297 73.0640 -1.657021 -1.951675 1 2
306
- 298 72.8188 -4.584625 -4.804875 1 2
307
- 299 72.9097 -2.263467 -2.064225 2 2
308
- 300 73.0000 -2.375790 -2.919557 1 1
309
- 301 73.0897 -3.973177 -4.922337 1 1
310
- 302 73.1788 -4.209826 -3.520937 2 2
311
- 303 72.9373 -8.694237 -8.242511 2 1
312
- 304 73.0263 -4.811926 -3.531962 2 2
313
- 305 73.1148 -3.388959 -3.073548 2 2
314
- 306 73.2026 -2.166226 -1.795775 2 2
315
- 307 73.2899 -2.695967 -2.948281 1 1
316
- 308 73.3766 -5.458758 -4.541926 2 2
317
- 309 73.4628 -3.397056 -4.131337 1 1
318
- 310 73.5484 -6.323372 -5.942644 2 2
319
- 311 73.6334 -0.934344 -4.568596 1 1
320
- 312 73.3974 -4.551960 -4.470311 2 1
321
- 313 73.1629 -4.282782 -4.598550 1 2
322
- 314 73.2484 -4.578860 -4.165161 2 2
323
- 315 73.3333 -2.225363 -3.965967 1 1
324
- 316 73.4177 -7.707310 -5.422570 2 2
325
- 317 73.5016 -4.890440 -4.909233 1 1
326
- 318 73.5849 -4.690365 -4.410842 2 2
327
- 319 73.6677 -1.230813 -1.507771 1 1
328
- 320 73.7500 -4.070480 -4.915927 1 1
329
- 321 73.5202 -2.750127 -2.858993 1 2
330
- 322 73.6025 -2.717104 -2.448582 2 2
331
- 323 73.6842 -3.839714 -4.346527 1 1
332
- 324 73.7654 -4.147870 -4.255856 1 1
333
- 325 73.8462 -5.214693 -5.009925 2 2
334
- 326 73.9264 -2.449900 -1.732465 2 2
335
- 327 74.0061 -2.362751 -2.973465 1 1
336
- 328 73.7805 -3.962049 -3.908539 2 1
337
- 329 73.8602 -2.993814 -6.936022 1 1
338
- 330 73.9394 -1.514976 -2.235572 1 1
339
- 331 73.7160 -4.374049 -3.290600 2 1
340
- 332 73.7952 -1.659311 -2.493210 1 1
341
- 333 73.8739 -4.188179 -1.989266 2 2
342
- 334 73.9521 -3.516273 -5.407811 1 1
343
- 335 73.7313 -5.021869 -4.748017 2 1
344
- 336 73.5119 -3.119081 -3.590930 1 2
345
- 337 73.5905 -2.037523 -1.515154 2 2
346
- 338 73.3728 -3.952789 -3.156059 2 1
347
- 339 73.1563 -5.243426 -5.577666 1 2
348
- 340 73.2353 -4.484188 -5.150270 1 1
349
- 341 73.3138 -3.090800 -4.180591 1 1
350
- 342 73.3918 -5.497242 -4.868430 2 2
351
- 343 73.4694 -3.147635 -3.688682 1 1
352
- 344 73.5465 -2.385859 -3.338260 1 1
353
- 345 73.6232 -0.810811 -2.772375 1 1
354
- 346 73.4104 -5.682144 -6.513924 1 2
355
- 347 73.4870 -2.810752 -2.609541 2 2
356
- 348 73.5632 -0.785951 -1.843531 1 1
357
- 349 73.3524 -3.475440 -4.368287 1 2
358
- 350 73.1429 -1.234152 -3.090410 1 2
359
- 351 73.2194 -3.037217 -3.506358 1 1
360
- 352 73.2955 -3.108251 -2.686356 2 2
361
- 353 73.3711 -2.257773 -2.288059 1 1
362
- 354 73.4463 -6.254824 -5.436137 2 2
363
- 355 73.5211 -2.645957 -2.225766 2 2
364
- 356 73.3146 -3.438194 -3.550509 1 2
365
- 357 73.3894 -4.618271 -4.291102 2 2
366
- 358 73.4637 -2.184197 -1.941622 2 2
367
- 359 73.5376 -5.734339 -3.728736 2 2
368
- 360 73.3333 -3.324123 -3.575033 1 2
369
- 361 73.4072 -2.816282 -1.437798 2 2
370
- 362 73.4807 -5.932158 -5.593666 2 2
371
- 363 73.2782 -3.038556 -2.536505 2 1
372
- 364 73.3516 -5.532372 -5.938004 1 1
373
- 365 73.4247 -5.031021 -5.223843 1 1
374
- 366 73.2240 -2.408573 -6.826363 1 2
375
- 367 73.2970 -5.154603 -5.419285 1 1
376
- 368 73.3696 -4.562939 -3.962709 2 2
377
- 369 73.1707 -3.333404 -3.696216 1 2
378
- 370 72.9730 -5.286895 -4.714594 2 1
379
- 371 73.0458 -2.906307 -3.932457 1 1
380
- 372 73.1183 -2.171890 -2.415550 1 1
381
- 373 72.9223 -4.602514 -3.997835 2 1
382
- 374 72.9947 -1.830723 -2.820654 1 1
383
- 375 73.0667 -3.934113 -2.949951 2 2
384
- 376 73.1383 -5.358327 -3.775577 2 2
385
- 377 73.2095 -4.241274 -3.713278 2 2
386
- 378 73.0159 -3.097253 -3.481090 1 2
387
- 379 72.8232 -0.994626 -2.314632 1 2
388
- 380 72.8947 -5.163737 -5.900650 1 1
389
- 381 72.9659 -10.230210 -8.880025 2 2
390
- 382 73.0366 -2.270624 -2.264003 2 2
391
- 383 73.1070 -2.642549 -1.623435 2 2
392
- 384 72.9167 -3.573603 -2.777339 2 1
393
- 385 72.9870 -1.067923 -1.136301 1 1
394
- 386 73.0570 -2.521335 -14.256692 1 1
395
- 387 73.1266 -1.779838 -2.495657 1 1
396
- 388 73.1959 -5.986874 -5.907176 2 2
397
- 389 73.2648 -2.542588 -2.188824 2 2
398
- 390 73.3333 -2.561356 -2.664809 1 1
399
- 391 73.1458 -3.881993 -4.576675 1 2
400
- 392 72.9592 -4.128429 -3.250164 2 1
401
- 393 72.7735 -5.823662 -5.886276 1 2
402
- 394 72.8426 -1.848247 -3.165315 1 1
403
- 395 72.9114 -3.353259 -3.285776 2 2
404
- 396 72.9798 -2.386097 -3.402795 1 1
405
- 397 73.0479 -6.037673 -7.013412 1 1
406
- 398 73.1156 -2.584986 -5.259095 1 1
407
- 399 73.1830 -3.142169 -3.833313 1 1
408
- 400 73.2500 -1.405006 -6.800112 1 1
409
- 401 73.3167 -3.443726 -3.959369 1 1
410
- 402 73.3831 -3.865415 -3.863189 2 2
411
- 403 73.2010 -6.230977 -6.550858 1 2
412
- 404 73.2673 -4.134281 -2.938529 2 2
413
- 405 73.3333 -3.080359 -4.215906 1 1
414
- 406 73.3990 -7.681197 -8.099322 1 1
415
- 407 73.2187 -4.885221 -4.909389 1 2
416
- 408 73.0392 -3.143947 -3.231467 1 2
417
- 409 73.1051 -3.617351 -4.429572 1 1
418
- 410 72.9268 -3.792608 -3.726537 2 1
419
- 411 72.9927 -3.788900 -5.049955 1 1
420
- 412 73.0583 -3.914684 -3.150494 2 2
421
- 413 73.1235 -3.294648 -2.351789 2 2
422
- 414 72.9469 -3.939450 -4.322599 1 2
423
- 415 73.0120 -3.567327 -3.080024 2 2
424
- 416 73.0769 -4.139552 -2.548860 2 2
425
- 417 73.1415 -3.695684 -4.694050 1 1
426
- 418 73.2057 -5.755840 -5.943969 1 1
427
- 419 73.2697 -3.990274 -3.736888 2 2
428
- 420 73.0952 -3.072586 -3.195949 1 2
429
- 421 73.1591 -4.083356 -3.827005 2 2
430
- 422 73.2227 -1.618295 -1.736628 1 1
431
- 423 73.2861 -3.424792 -3.528804 1 1
432
- 424 73.3491 -4.694325 -5.251293 1 1
433
- 425 73.4118 -4.952539 -2.866773 2 2
434
- 426 73.4742 -1.861789 -1.390831 2 2
435
- 427 73.5363 -3.441283 -4.382609 1 1
436
- 428 73.3645 -3.731501 -3.308204 2 1
437
- 429 73.4266 -3.226462 -3.003190 2 2
438
- 430 73.4884 -5.517939 -5.677110 1 1
439
- 431 73.5499 -3.923601 -5.575803 1 1
440
- 432 73.3796 -3.539736 -4.182064 1 2
441
- 433 73.4411 -2.497999 -2.061543 2 2
442
- 434 73.5023 -6.574561 -7.298841 1 1
443
- 435 73.5632 -3.259751 -3.475883 1 1
444
- 436 73.3945 -2.308008 -2.150135 2 1
445
- 437 73.2265 -3.612182 -3.663677 1 2
446
- 438 73.2877 -4.026180 -3.831503 2 2
447
- 439 73.3485 -2.731662 -3.557570 1 1
448
- 440 73.4091 -3.102165 -3.589588 1 1
449
- 441 73.4694 -3.302856 -5.137486 1 1
450
- 442 73.5294 -4.415608 -4.436215 1 1
451
- 443 73.5892 -1.728067 -3.727095 1 1
452
- 444 73.4234 -5.851809 -5.980094 1 2
453
- 445 73.4831 -1.698410 -2.564716 1 1
454
- 446 73.5426 -3.837349 -2.667679 2 2
455
- 447 73.3781 -2.733521 -2.282870 2 1
456
- 448 73.4375 -4.318438 -4.138916 2 2
457
- 449 73.4967 -2.806792 -2.257827 2 2
458
- 450 73.5556 -5.417691 -4.337748 2 2
459
- 451 73.6142 -4.321202 -3.915606 2 2
460
- 452 73.6726 -3.486775 -4.096069 1 1
461
- 453 73.7307 -4.175915 -3.356729 2 2
462
- 454 73.7885 -3.520458 -3.942152 1 1
463
- 455 73.6264 -1.284805 -10.062407 1 2
464
- 456 73.6842 -3.730599 -3.576404 2 2
465
- 457 73.5230 -8.789034 -8.146083 2 1
466
- 458 73.5808 -3.292186 -2.640854 2 2
467
- 459 73.6383 -4.225148 -4.120842 2 2
468
- 460 73.6957 -0.654129 -9.116646 1 1
469
- 461 73.7527 -1.821052 -2.788685 1 1
470
- 462 73.8095 -5.097099 -5.393923 1 1
471
- 463 73.8661 -1.185778 -1.492629 1 1
472
- 464 73.9224 -1.638469 -2.165240 1 1
473
- 465 73.9785 -3.493961 -2.574859 2 2
474
- 466 74.0343 -5.257381 -5.195807 2 2
475
- 467 73.8758 -4.715145 -4.634686 2 1
476
- 468 73.9316 -4.829343 -3.201736 2 2
477
- 469 73.7740 -1.958525 -1.545654 2 1
478
- 470 73.8298 -1.760127 -1.274052 2 2
479
- 471 73.8854 -2.029215 -2.401083 1 1
480
- 472 73.9407 -3.512060 -3.425378 2 2
481
- 473 73.9958 -3.826608 -3.140702 2 2
482
- 474 74.0506 -3.486912 -5.117774 1 1
483
- 475 74.1053 -2.091841 -1.951240 2 2
484
- 476 74.1597 -4.665457 -4.210580 2 2
485
- 477 74.0042 -4.397153 -5.032299 1 2
486
- 478 74.0586 -2.508753 -2.581853 1 1
487
- 479 73.9040 -3.075969 -3.060396 2 1
488
- 480 73.9583 -5.071450 -4.374039 2 2
489
- 481 74.0125 -1.944461 -2.029138 1 1
490
- 482 73.8589 -4.465379 -5.614033 1 2
491
- 483 73.9130 -3.882707 -4.483698 1 1
492
- 484 73.7603 -2.891012 -2.444814 2 1
493
- 485 73.8144 -1.392002 -2.068040 1 1
494
- 486 73.8683 -1.944494 -2.578127 1 1
495
- 487 73.9220 -3.356415 -3.016878 2 2
496
- 488 73.9754 -3.975504 -3.233751 2 2
497
- 489 74.0286 -2.434635 -2.922386 1 1
498
- 490 74.0816 -2.570170 -2.949575 1 1
499
- 491 74.1344 -5.108028 -4.193113 2 2
500
- 492 74.1870 -5.094119 -4.238470 2 2
501
- 493 74.2394 -1.852984 -2.680790 1 1
502
- 494 74.2915 -3.689822 -3.033239 2 2
503
- 495 74.3434 -3.631328 -3.830582 1 1
504
- 496 74.3952 -3.110355 -3.048936 2 2
505
- 497 74.4467 -4.626122 -4.811220 1 1
506
- 498 74.2972 -2.099021 -2.348722 1 2
507
- 499 74.3487 -2.618156 -3.153173 1 1
508
- 500 74.4000 -5.324596 -4.629584 2 2
509
- 501 74.4511 -2.406194 -2.726190 1 1
510
- 502 74.5020 -2.080312 -2.963145 1 1
511
- 503 74.5527 -3.259957 -3.027171 2 2
512
- 504 74.6032 -2.804955 -3.007889 1 1
513
- 505 74.6535 -7.001708 -6.583428 2 2
514
- 506 74.7036 -4.390716 -5.185130 1 1
515
- 507 74.7535 -3.175409 -3.397121 1 1
516
- 508 74.8031 -5.308577 -2.626710 2 2
517
- 509 74.8527 -5.696872 -8.711957 1 1
518
- 510 74.7059 -4.174140 -4.057052 2 1
519
- 511 74.7554 -5.580155 -4.234007 2 2
520
- 512 74.8047 -2.779043 -3.065385 1 1
521
- 513 74.8538 -3.212018 -2.944828 2 2
522
- 514 74.7082 -4.233897 -3.544245 2 1
523
- 515 74.7573 -6.809156 -9.698454 1 1
524
- 516 74.8062 -4.286294 -4.318830 1 1
525
- 517 74.8549 -1.655329 -2.021208 1 1
526
- 518 74.7104 -2.770018 -2.836296 1 2
527
- 519 74.7592 -7.090511 -6.796824 2 2
528
- 520 74.8077 -2.580245 -2.430886 2 2
529
- 521 74.6641 -1.726790 -1.684430 2 1
530
- 522 74.7126 -3.722552 -2.149171 2 2
531
- 523 74.7610 -3.789690 -3.620882 2 2
532
- 524 74.8092 -4.282014 -5.018980 1 1
533
- 525 74.8571 -3.539413 -2.132561 2 2
534
- 526 74.9049 -0.995498 -14.069503 1 1
535
- 527 74.9526 -5.350708 -3.626143 2 2
536
- 528 75.0000 -5.855553 -5.515306 2 2
537
- 529 75.0473 -5.048443 -5.629791 1 1
538
- 530 75.0943 -4.722565 -2.451430 2 2
539
- 531 75.1412 -3.351028 -3.547198 1 1
540
- 532 75.1880 -2.205390 -3.303182 1 1
541
- 533 75.2345 -3.769376 -2.297494 2 2
542
- 534 75.2809 -2.636830 -3.033358 1 1
543
- 535 75.3271 -3.539464 -3.432133 2 2
544
- 536 75.1866 -2.550410 -2.263073 2 1
545
- 537 75.2328 -4.815022 -6.180703 1 1
546
- 538 75.2788 -2.446483 -4.452942 1 1
547
- 539 75.3247 -4.030140 -1.743078 2 2
548
- 540 75.3704 -2.773297 -2.731184 2 2
549
- 541 75.4159 -2.568646 -3.105608 1 1
550
- 542 75.2768 -4.103900 -5.061793 1 2
551
- 543 75.3223 -5.570627 -4.134392 2 2
552
- 544 75.3676 -2.834203 -3.906796 1 1
553
- 545 75.2294 -2.429997 -3.548324 1 2
554
- 546 75.2747 -2.981934 -4.702179 1 1
555
- 547 75.3199 -5.949451 -5.345825 2 2
556
- 548 75.1825 -4.500102 -4.352159 2 1
557
- 549 75.2277 -5.333830 -5.478172 1 1
558
- 550 75.2727 -2.564449 -2.528668 2 2
559
- 551 75.1361 -0.888034 -13.595346 1 2
560
- 552 75.0000 -2.166970 -2.042335 2 1
561
- 553 74.8644 -5.136461 -5.207839 1 2
562
- 554 74.9097 -4.687700 -4.544545 2 2
563
- 555 74.9550 -1.860352 -1.456827 2 2
564
- 556 75.0000 -5.687045 -5.186214 2 2
565
- 557 75.0449 -4.224676 -4.680477 1 1
566
- 558 75.0896 -1.709875 -3.103723 1 1
567
- 559 75.1342 -2.285528 -2.243107 2 2
568
- 560 75.0000 -2.907297 -2.776588 2 1
569
- 561 75.0446 -1.842168 -2.370754 1 1
570
- 562 75.0890 -0.823465 -2.263441 1 1
571
- 563 74.9556 -1.163865 -2.288908 1 2
572
- 564 75.0000 -2.531640 -2.914842 1 1
573
- 565 75.0442 -4.606893 -4.350835 2 2
574
- 566 75.0883 -4.020670 -4.728627 1 1
575
- 567 75.1323 -2.102988 -1.725632 2 2
576
- 568 75.1761 -7.223089 -2.992253 2 2
577
- 569 75.0439 -5.049522 -3.567580 2 1
578
- 570 75.0877 -4.219952 -4.999730 1 1
579
- 571 75.1313 -4.062715 -3.070554 2 2
580
- 572 75.1748 -2.608581 -2.370102 2 2
581
- 573 75.0436 -3.689412 -3.811342 1 2
582
- 574 75.0871 -3.355217 -3.333362 2 2
583
- 575 75.1304 -5.215151 -5.066965 2 2
584
- 576 75.1736 -2.918771 -3.688029 1 1
585
- 577 75.2166 -4.372516 -3.802884 2 2
586
- 578 75.2595 -3.088501 -2.942692 2 2
587
- 579 75.1295 -4.124115 -4.721822 1 2
588
- 580 75.1724 -1.900870 -2.894287 1 1
589
- 581 75.2151 -6.386947 -7.242309 1 1
590
- 582 75.0859 -4.975612 -5.155877 1 2
591
- 583 75.1286 -4.618403 -4.301981 2 2
592
- 584 75.0000 -4.218491 -4.195620 2 1
593
- 585 75.0427 -0.569117 -1.620439 1 1
594
- 586 75.0853 -3.731462 -3.952114 1 1
595
- 587 75.1278 -4.606149 -4.751362 1 1
596
- 588 75.1701 -3.399364 -2.243202 2 2
597
- 589 75.2122 -3.900348 -3.896920 2 2
598
- 590 75.2542 -3.686379 -2.857552 2 2
599
- 591 75.2961 -3.985135 -3.431690 2 2
600
- 592 75.3378 -3.223303 -4.073713 1 1
601
- 593 75.3794 -3.924019 -3.357842 2 2
602
- 594 75.2525 -4.580763 -4.323801 2 1
603
- 595 75.1261 -1.675664 -1.742366 1 2
604
- 596 75.1678 -4.949000 -4.855286 2 2
605
- 597 75.2094 -1.608619 -1.935477 1 1
606
- 598 75.2508 -3.038946 -3.862474 1 1
607
- 599 75.2922 -2.741371 -3.033667 1 1
608
- 600 75.3333 -4.085466 -3.829785 2 2
609
- 601 75.3744 -3.490582 -3.430320 2 2
610
- 602 75.4153 -2.642563 -4.215326 1 1
611
- 603 75.4561 -5.807222 -5.438059 2 2
612
- 604 75.3311 -2.655333 -2.777631 1 2
613
- 605 75.2066 -2.721641 -2.185980 2 1
614
- 606 75.2475 -3.530785 -2.587831 2 2
615
- 607 75.1236 -2.319178 -1.709419 2 1
616
- 608 75.1645 -2.711894 -2.712034 1 1
617
- 609 75.2053 -5.625054 -6.239610 1 1
618
- 610 75.2459 -2.615806 -2.639396 1 1
619
- 611 75.2864 -4.691599 -4.850064 1 1
620
- 612 75.3268 -3.560272 -5.490305 1 1
621
- 613 75.3670 -2.609219 -2.287420 2 2
622
- 614 75.4072 -3.279566 -2.140824 2 2
623
- 615 75.4472 -2.191392 -1.942126 2 2
624
- 616 75.3247 -3.643527 -3.743683 1 2
625
- 617 75.3647 -10.214173 -2.761441 2 2
626
- 618 75.2427 -3.872384 -3.945694 1 2
627
- 619 75.2827 -3.433329 -2.999075 2 2
628
- 620 75.3226 -4.132913 -4.338573 1 1
629
- 621 75.3623 -3.566050 -3.200900 2 2
630
- 622 75.4019 -2.636543 -2.698215 1 1
631
- 623 75.2809 -2.727010 -2.564171 2 1
632
- 624 75.3205 -4.114738 -3.438207 2 2
633
- 625 75.3600 -3.238032 -5.256586 1 1
634
- 626 75.2396 -2.514247 -2.221473 2 1
635
- 627 75.2791 -3.715107 -3.817087 1 1
636
- 628 75.3185 -2.734435 -2.097336 2 2
637
- 629 75.3577 -1.744548 -2.163700 1 1
638
- 630 75.3968 -1.160430 -1.715122 1 1
639
- 631 75.4358 -1.441543 -2.342646 1 1
640
- 632 75.3165 -1.889578 -1.895899 1 2
641
- 633 75.3555 -3.248693 -2.493080 2 2
642
- 634 75.3943 -2.230629 -3.057205 1 1
643
- 635 75.2756 -3.922247 -4.720690 1 2
644
- 636 75.3145 -1.995444 -4.003211 1 1
645
- 637 75.3532 -3.527036 -5.823039 1 1
646
- 638 75.3918 -4.985180 -5.841948 1 1
647
- 639 75.4304 -1.102773 -1.440566 1 1
648
- 640 75.3125 -1.555686 -2.092469 1 2
649
- 641 75.3510 -4.522011 -5.131280 1 1
650
- 642 75.3894 -4.735169 -2.789386 2 2
651
- 643 75.4277 -1.420842 -0.734960 2 2
652
- 644 75.4658 -4.375111 -4.987925 1 1
653
- 645 75.5039 -1.436872 -1.550294 1 1
654
- 646 75.3870 -3.930212 -3.822849 2 1
655
- 647 75.2705 -3.536510 -3.020024 2 1
656
- 648 75.3086 -3.258257 -3.115051 2 2
657
- 649 75.3467 -2.887537 -3.175544 1 1
658
- 650 75.3846 -3.452408 -4.302853 1 1
659
- 651 75.4224 -6.121977 -5.764140 2 2
660
- 652 75.4601 -5.067463 -4.483727 2 2
661
- 653 75.4977 -6.886916 -5.914289 2 2
662
- 654 75.3823 -3.183117 -3.485486 1 2
663
- 655 75.4198 -2.839803 -2.673842 2 2
664
- 656 75.4573 -4.561643 -2.480217 2 2
665
- 657 75.3425 -5.425309 -5.908160 1 2
666
- 658 75.3799 -5.133717 -3.009473 2 2
667
- 659 75.4173 -2.130508 -1.779564 2 2
668
- 660 75.4545 -3.497257 -2.555586 2 2
669
- 661 75.4917 -2.697710 -3.166183 1 1
670
- 662 75.5287 -3.263794 -3.426273 1 1
671
- 663 75.5656 -5.478396 -4.219425 2 2
672
- 664 75.6024 -3.338886 -3.406757 1 1
673
- 665 75.6391 -3.757879 -3.522321 2 2
674
- 666 75.6757 -4.568055 -4.993385 1 1
675
- 667 75.5622 -6.302129 -9.167962 1 2
676
- 668 75.5988 -2.909007 -3.055274 1 1
677
- 669 75.6353 -2.620282 -2.997510 1 1
678
- 670 75.5224 -3.366896 -2.866287 2 1
679
- 671 75.5589 -3.490412 -4.959549 1 1
680
- 672 75.5952 -2.778010 -2.755957 2 2
681
- 673 75.6315 -1.912511 -1.858993 2 2
682
- 674 75.5193 -5.674745 -5.498894 2 1
683
- 675 75.4074 -4.424205 -3.906595 2 1
684
- 676 75.2959 -2.078282 -2.462609 1 2
685
- 677 75.3323 -3.703651 -3.967614 1 1
686
- 678 75.2212 -5.275679 -5.336710 1 2
687
- 679 75.2577 -5.533029 -5.280270 2 2
688
- 680 75.2941 -3.373024 -5.041277 1 1
689
- 681 75.1836 -3.257920 -2.072685 2 1
690
- 682 75.0733 -1.815852 -1.998923 1 2
691
- 683 75.1098 -1.834555 -2.673700 1 1
692
- 684 75.1462 -5.081506 -7.915822 1 1
693
- 685 75.1825 -4.004911 -1.744561 2 2
694
- 686 75.2187 -5.449542 -4.860441 2 2
695
- 687 75.2547 -2.906272 -2.059308 2 2
696
- 688 75.2907 -6.893285 -7.515503 1 1
697
- 689 75.3266 -2.481367 -1.900985 2 2
698
- 690 75.3623 -2.694639 -2.564559 2 2
699
- 691 75.2533 -4.029155 -3.327098 2 1
700
- 692 75.2890 -1.574014 -2.901084 1 1
701
- 693 75.3247 -4.079288 -4.474146 1 1
702
- 694 75.3602 -3.588190 -3.199070 2 2
703
- 695 75.3957 -4.381227 -5.051682 1 1
704
- 696 75.2874 -4.977376 -4.616941 2 1
705
- 697 75.3228 -2.669790 -2.762102 1 1
706
- 698 75.3582 -4.203280 -3.645899 2 2
707
- 699 75.3934 -3.922485 -5.649545 1 1
708
- 700 75.4286 -3.317356 -2.430110 2 2
709
- 701 75.3210 -2.092158 -1.582822 2 1
710
- 702 75.2137 -4.218829 -4.832745 1 2
711
- 703 75.1067 -4.455004 -3.947652 2 1
712
- 704 75.1420 -1.586174 -1.680350 1 1
713
- 705 75.1773 -2.481956 -2.053693 2 2
714
- 706 75.2125 -2.960253 -2.930925 2 2
715
- 707 75.2475 -4.665153 -4.712732 1 1
716
- 708 75.2825 -3.923853 -3.060729 2 2
717
- 709 75.3173 -3.567501 -2.983513 2 2
718
- 710 75.3521 -2.669982 -3.798574 1 1
719
- 711 75.3868 -2.331685 -1.867720 2 2
720
- 712 75.2809 -2.888279 -3.754961 1 2
721
- 713 75.1753 -3.281252 -3.311838 1 2
722
- 714 75.2101 -2.883123 -2.554054 2 2
723
- 715 75.2448 -5.265210 -4.185415 2 2
724
- 716 75.2793 -4.399998 -3.622588 2 2
725
- 717 75.3138 -2.000395 -0.594467 2 2
726
- 718 75.3482 -3.547055 -3.970223 1 1
727
- 719 75.2434 -5.027711 -4.721919 2 1
728
- 720 75.1389 -1.542250 -4.583506 1 2
729
- 721 75.0347 -4.491573 -4.974329 1 2
730
- 722 75.0693 -2.411076 -2.479419 1 1
731
- 723 74.9654 -3.104930 -3.051952 2 1
732
- 724 74.8619 -3.041220 -3.472128 1 2
733
- 725 74.7586 -2.875374 -2.950399 1 2
734
- 726 74.6556 -4.252460 -4.399210 1 2
735
- 727 74.6905 -2.957076 -2.272223 2 2
736
- 728 74.7253 -4.350930 -4.415499 1 1
737
- 729 74.7599 -1.942379 -2.823197 1 1
738
- 730 74.6575 -3.467944 -3.207817 2 1
739
- 731 74.6922 -3.477276 -4.058329 1 1
740
- 732 74.7268 -3.166644 -2.310535 2 2
741
- 733 74.6248 -3.511018 -3.574278 1 2
742
- 734 74.6594 -0.147358 -7.923507 1 1
743
- 735 74.5578 -2.626792 -2.341353 2 1
744
- 736 74.5924 -3.212721 -3.863147 1 1
745
- 737 74.6269 -3.893528 -4.005075 1 1
746
- 738 74.6612 -4.432023 -4.231154 2 2
747
- 739 74.6955 -0.203277 -8.689461 1 1
748
- 740 74.7297 -4.053092 -3.088568 2 2
749
- 741 74.6289 -4.185293 -3.907790 2 1
750
- 742 74.6631 -2.096561 -1.536294 2 2
751
- 743 74.6972 -3.600339 -3.325960 2 2
752
- 744 74.7312 -5.124127 -4.164810 2 2
753
- 745 74.7651 -3.547146 -4.600223 1 1
754
- 746 74.7989 -5.833979 -6.161608 1 1
755
- 747 74.8327 -4.853059 -5.007121 1 1
756
- 748 74.8663 -3.404722 -3.128401 2 2
757
- 749 74.8999 -3.197073 -3.737285 1 1
758
- 750 74.9333 -1.698206 -7.912483 1 1
759
 
760
- Final Winogrande score(750 tasks): 74.9333 +/- 1.5836
 
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-IQ4_NL.gguf (version GGUF V3 (latest))
4
 
5
+ Final Winogrande score(750 tasks): 71.8667 +/- 1.6430
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
 
7
+ llama_perf_context_print: load time = 302.27 ms
8
+ llama_perf_context_print: prompt eval time = 88132.28 ms / 22378 tokens ( 3.94 ms per token, 253.91 tokens per second)
9
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
10
+ llama_perf_context_print: total time = 89331.17 ms / 22379 tokens
11
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-q3_k_l.arc CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 869 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 869 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 100.00000000
13
- 3 100.00000000
14
- 4 75.00000000
15
- 5 80.00000000
16
- 6 83.33333333
17
- 7 85.71428571
18
- 8 87.50000000
19
- 9 77.77777778
20
- 10 80.00000000
21
- 11 72.72727273
22
- 12 75.00000000
23
- 13 69.23076923
24
- 14 71.42857143
25
- 15 73.33333333
26
- 16 75.00000000
27
- 17 76.47058824
28
- 18 72.22222222
29
- 19 73.68421053
30
- 20 70.00000000
31
- 21 71.42857143
32
- 22 72.72727273
33
- 23 69.56521739
34
- 24 70.83333333
35
- 25 68.00000000
36
- 26 65.38461538
37
- 27 66.66666667
38
- 28 64.28571429
39
- 29 65.51724138
40
- 30 66.66666667
41
- 31 64.51612903
42
- 32 65.62500000
43
- 33 63.63636364
44
- 34 61.76470588
45
- 35 62.85714286
46
- 36 61.11111111
47
- 37 62.16216216
48
- 38 63.15789474
49
- 39 64.10256410
50
- 40 65.00000000
51
- 41 65.85365854
52
- 42 64.28571429
53
- 43 62.79069767
54
- 44 61.36363636
55
- 45 62.22222222
56
- 46 63.04347826
57
- 47 63.82978723
58
- 48 64.58333333
59
- 49 65.30612245
60
- 50 66.00000000
61
- 51 64.70588235
62
- 52 65.38461538
63
- 53 64.15094340
64
- 54 62.96296296
65
- 55 61.81818182
66
- 56 62.50000000
67
- 57 63.15789474
68
- 58 63.79310345
69
- 59 64.40677966
70
- 60 65.00000000
71
- 61 65.57377049
72
- 62 64.51612903
73
- 63 65.07936508
74
- 64 65.62500000
75
- 65 64.61538462
76
- 66 65.15151515
77
- 67 65.67164179
78
- 68 66.17647059
79
- 69 66.66666667
80
- 70 65.71428571
81
- 71 64.78873239
82
- 72 63.88888889
83
- 73 64.38356164
84
- 74 64.86486486
85
- 75 65.33333333
86
- 76 65.78947368
87
- 77 66.23376623
88
- 78 66.66666667
89
- 79 65.82278481
90
- 80 66.25000000
91
- 81 65.43209877
92
- 82 65.85365854
93
- 83 65.06024096
94
- 84 65.47619048
95
- 85 65.88235294
96
- 86 66.27906977
97
- 87 65.51724138
98
- 88 65.90909091
99
- 89 66.29213483
100
- 90 66.66666667
101
- 91 67.03296703
102
- 92 67.39130435
103
- 93 66.66666667
104
- 94 65.95744681
105
- 95 66.31578947
106
- 96 66.66666667
107
- 97 65.97938144
108
- 98 65.30612245
109
- 99 65.65656566
110
- 100 66.00000000
111
- 101 65.34653465
112
- 102 65.68627451
113
- 103 65.04854369
114
- 104 65.38461538
115
- 105 65.71428571
116
- 106 65.09433962
117
- 107 65.42056075
118
- 108 64.81481481
119
- 109 64.22018349
120
- 110 64.54545455
121
- 111 64.86486486
122
- 112 65.17857143
123
- 113 64.60176991
124
- 114 64.91228070
125
- 115 65.21739130
126
- 116 65.51724138
127
- 117 64.95726496
128
- 118 65.25423729
129
- 119 64.70588235
130
- 120 65.00000000
131
- 121 64.46280992
132
- 122 64.75409836
133
- 123 65.04065041
134
- 124 64.51612903
135
- 125 64.80000000
136
- 126 65.07936508
137
- 127 65.35433071
138
- 128 65.62500000
139
- 129 65.11627907
140
- 130 64.61538462
141
- 131 64.88549618
142
- 132 65.15151515
143
- 133 65.41353383
144
- 134 64.92537313
145
- 135 64.44444444
146
- 136 63.97058824
147
- 137 64.23357664
148
- 138 64.49275362
149
- 139 64.74820144
150
- 140 65.00000000
151
- 141 64.53900709
152
- 142 64.78873239
153
- 143 65.03496503
154
- 144 64.58333333
155
- 145 64.82758621
156
- 146 64.38356164
157
- 147 64.62585034
158
- 148 64.86486486
159
- 149 65.10067114
160
- 150 65.33333333
161
- 151 65.56291391
162
- 152 65.78947368
163
- 153 66.01307190
164
- 154 65.58441558
165
- 155 65.80645161
166
- 156 66.02564103
167
- 157 66.24203822
168
- 158 65.82278481
169
- 159 66.03773585
170
- 160 65.62500000
171
- 161 65.83850932
172
- 162 66.04938272
173
- 163 65.64417178
174
- 164 65.24390244
175
- 165 65.45454545
176
- 166 65.66265060
177
- 167 65.26946108
178
- 168 65.47619048
179
- 169 65.08875740
180
- 170 65.29411765
181
- 171 65.49707602
182
- 172 65.11627907
183
- 173 65.31791908
184
- 174 65.51724138
185
- 175 65.14285714
186
- 176 65.34090909
187
- 177 65.53672316
188
- 178 65.73033708
189
- 179 65.92178771
190
- 180 66.11111111
191
- 181 65.74585635
192
- 182 65.93406593
193
- 183 65.57377049
194
- 184 65.76086957
195
- 185 65.94594595
196
- 186 66.12903226
197
- 187 66.31016043
198
- 188 65.95744681
199
- 189 65.60846561
200
- 190 65.26315789
201
- 191 65.44502618
202
- 192 65.62500000
203
- 193 65.80310881
204
- 194 65.97938144
205
- 195 66.15384615
206
- 196 66.32653061
207
- 197 66.49746193
208
- 198 66.16161616
209
- 199 65.82914573
210
- 200 66.00000000
211
- 201 66.16915423
212
- 202 65.84158416
213
- 203 66.00985222
214
- 204 66.17647059
215
- 205 65.85365854
216
- 206 65.53398058
217
- 207 65.70048309
218
- 208 65.86538462
219
- 209 66.02870813
220
- 210 66.19047619
221
- 211 66.35071090
222
- 212 66.03773585
223
- 213 66.19718310
224
- 214 66.35514019
225
- 215 66.51162791
226
- 216 66.20370370
227
- 217 65.89861751
228
- 218 65.59633028
229
- 219 65.75342466
230
- 220 65.45454545
231
- 221 65.15837104
232
- 222 64.86486486
233
- 223 64.57399103
234
- 224 64.73214286
235
- 225 64.44444444
236
- 226 64.60176991
237
- 227 64.75770925
238
- 228 64.91228070
239
- 229 65.06550218
240
- 230 65.21739130
241
- 231 64.93506494
242
- 232 64.65517241
243
- 233 64.80686695
244
- 234 64.95726496
245
- 235 64.68085106
246
- 236 64.40677966
247
- 237 64.13502110
248
- 238 64.28571429
249
- 239 64.43514644
250
- 240 64.58333333
251
- 241 64.73029046
252
- 242 64.87603306
253
- 243 64.60905350
254
- 244 64.34426230
255
- 245 64.48979592
256
- 246 64.63414634
257
- 247 64.77732794
258
- 248 64.91935484
259
- 249 64.65863454
260
- 250 64.80000000
261
- 251 64.94023904
262
- 252 65.07936508
263
- 253 65.21739130
264
- 254 64.96062992
265
- 255 65.09803922
266
- 256 64.84375000
267
- 257 64.59143969
268
- 258 64.72868217
269
- 259 64.47876448
270
- 260 64.61538462
271
- 261 64.75095785
272
- 262 64.88549618
273
- 263 65.01901141
274
- 264 65.15151515
275
- 265 64.90566038
276
- 266 64.66165414
277
- 267 64.79400749
278
- 268 64.92537313
279
- 269 65.05576208
280
- 270 65.18518519
281
- 271 65.31365314
282
- 272 65.07352941
283
- 273 64.83516484
284
- 274 64.59854015
285
- 275 64.36363636
286
- 276 64.49275362
287
- 277 64.62093863
288
- 278 64.38848921
289
- 279 64.51612903
290
- 280 64.64285714
291
- 281 64.76868327
292
- 282 64.89361702
293
- 283 64.66431095
294
- 284 64.78873239
295
- 285 64.91228070
296
- 286 65.03496503
297
- 287 65.15679443
298
- 288 65.27777778
299
- 289 65.05190311
300
- 290 65.17241379
301
- 291 65.29209622
302
- 292 65.41095890
303
- 293 65.52901024
304
- 294 65.30612245
305
- 295 65.42372881
306
- 296 65.54054054
307
- 297 65.65656566
308
- 298 65.77181208
309
- 299 65.88628763
310
- 300 65.66666667
311
- 301 65.44850498
312
- 302 65.56291391
313
- 303 65.67656766
314
- 304 65.78947368
315
- 305 65.90163934
316
- 306 65.68627451
317
- 307 65.79804560
318
- 308 65.90909091
319
- 309 66.01941748
320
- 310 66.12903226
321
- 311 66.23794212
322
- 312 66.34615385
323
- 313 66.45367412
324
- 314 66.24203822
325
- 315 66.34920635
326
- 316 66.45569620
327
- 317 66.56151420
328
- 318 66.35220126
329
- 319 66.14420063
330
- 320 65.93750000
331
- 321 66.04361371
332
- 322 66.14906832
333
- 323 66.25386997
334
- 324 66.35802469
335
- 325 66.46153846
336
- 326 66.56441718
337
- 327 66.36085627
338
- 328 66.46341463
339
- 329 66.56534954
340
- 330 66.36363636
341
- 331 66.46525680
342
- 332 66.56626506
343
- 333 66.66666667
344
- 334 66.76646707
345
- 335 66.56716418
346
- 336 66.66666667
347
- 337 66.76557864
348
- 338 66.56804734
349
- 339 66.66666667
350
- 340 66.76470588
351
- 341 66.86217009
352
- 342 66.95906433
353
- 343 67.05539359
354
- 344 67.15116279
355
- 345 66.95652174
356
- 346 67.05202312
357
- 347 66.85878963
358
- 348 66.66666667
359
- 349 66.76217765
360
- 350 66.57142857
361
- 351 66.66666667
362
- 352 66.76136364
363
- 353 66.85552408
364
- 354 66.66666667
365
- 355 66.47887324
366
- 356 66.57303371
367
- 357 66.66666667
368
- 358 66.75977654
369
- 359 66.57381616
370
- 360 66.66666667
371
- 361 66.75900277
372
- 362 66.85082873
373
- 363 66.66666667
374
- 364 66.48351648
375
- 365 66.57534247
376
- 366 66.39344262
377
- 367 66.21253406
378
- 368 66.30434783
379
- 369 66.39566396
380
- 370 66.48648649
381
- 371 66.57681941
382
- 372 66.66666667
383
- 373 66.75603217
384
- 374 66.84491979
385
- 375 66.66666667
386
- 376 66.48936170
387
- 377 66.31299735
388
- 378 66.40211640
389
- 379 66.22691293
390
- 380 66.31578947
391
- 381 66.14173228
392
- 382 65.96858639
393
- 383 65.79634465
394
- 384 65.88541667
395
- 385 65.97402597
396
- 386 66.06217617
397
- 387 66.14987080
398
- 388 65.97938144
399
- 389 65.80976864
400
- 390 65.64102564
401
- 391 65.47314578
402
- 392 65.56122449
403
- 393 65.64885496
404
- 394 65.73604061
405
- 395 65.82278481
406
- 396 65.65656566
407
- 397 65.74307305
408
- 398 65.57788945
409
- 399 65.66416040
410
- 400 65.75000000
411
- 401 65.58603491
412
- 402 65.67164179
413
- 403 65.75682382
414
- 404 65.84158416
415
- 405 65.92592593
416
- 406 66.00985222
417
- 407 66.09336609
418
- 408 66.17647059
419
- 409 66.01466993
420
- 410 66.09756098
421
- 411 66.18004866
422
- 412 66.26213592
423
- 413 66.10169492
424
- 414 66.18357488
425
- 415 66.02409639
426
- 416 66.10576923
427
- 417 65.94724221
428
- 418 66.02870813
429
- 419 65.87112172
430
- 420 65.95238095
431
- 421 65.79572447
432
- 422 65.87677725
433
- 423 65.95744681
434
- 424 65.80188679
435
- 425 65.88235294
436
- 426 65.72769953
437
- 427 65.80796253
438
- 428 65.65420561
439
- 429 65.73426573
440
- 430 65.58139535
441
- 431 65.66125290
442
- 432 65.74074074
443
- 433 65.58891455
444
- 434 65.66820276
445
- 435 65.74712644
446
- 436 65.82568807
447
- 437 65.90389016
448
- 438 65.75342466
449
- 439 65.83143508
450
- 440 65.68181818
451
- 441 65.53287982
452
- 442 65.38461538
453
- 443 65.23702032
454
- 444 65.09009009
455
- 445 65.16853933
456
- 446 65.24663677
457
- 447 65.10067114
458
- 448 65.17857143
459
- 449 65.25612472
460
- 450 65.11111111
461
- 451 65.18847007
462
- 452 65.26548673
463
- 453 65.12141280
464
- 454 65.19823789
465
- 455 65.05494505
466
- 456 65.13157895
467
- 457 65.20787746
468
- 458 65.28384279
469
- 459 65.35947712
470
- 460 65.43478261
471
- 461 65.50976139
472
- 462 65.58441558
473
- 463 65.44276458
474
- 464 65.30172414
475
- 465 65.37634409
476
- 466 65.45064378
477
- 467 65.52462527
478
- 468 65.59829060
479
- 469 65.45842217
480
- 470 65.53191489
481
- 471 65.60509554
482
- 472 65.46610169
483
- 473 65.32769556
484
- 474 65.40084388
485
- 475 65.47368421
486
- 476 65.54621849
487
- 477 65.61844864
488
- 478 65.69037657
489
- 479 65.76200418
490
- 480 65.83333333
491
- 481 65.90436590
492
- 482 65.97510373
493
- 483 66.04554865
494
- 484 66.11570248
495
- 485 66.18556701
496
- 486 66.25514403
497
- 487 66.11909651
498
- 488 66.18852459
499
- 489 66.25766871
500
- 490 66.12244898
501
- 491 65.98778004
502
- 492 66.05691057
503
- 493 65.92292089
504
- 494 65.99190283
505
- 495 65.85858586
506
- 496 65.92741935
507
- 497 65.79476861
508
- 498 65.86345382
509
- 499 65.93186373
510
- 500 66.00000000
511
- 501 65.86826347
512
- 502 65.93625498
513
- 503 66.00397614
514
- 504 66.07142857
515
- 505 66.13861386
516
- 506 66.00790514
517
- 507 65.87771203
518
- 508 65.74803150
519
- 509 65.61886051
520
- 510 65.49019608
521
- 511 65.36203523
522
- 512 65.42968750
523
- 513 65.30214425
524
- 514 65.36964981
525
- 515 65.43689320
526
- 516 65.50387597
527
- 517 65.57059961
528
- 518 65.63706564
529
- 519 65.70327553
530
- 520 65.57692308
531
- 521 65.64299424
532
- 522 65.70881226
533
- 523 65.58317400
534
- 524 65.45801527
535
- 525 65.33333333
536
- 526 65.20912548
537
- 527 65.08538899
538
- 528 64.96212121
539
- 529 64.83931947
540
- 530 64.90566038
541
- 531 64.78342750
542
- 532 64.84962406
543
- 533 64.91557223
544
- 534 64.79400749
545
- 535 64.67289720
546
- 536 64.73880597
547
- 537 64.80446927
548
- 538 64.86988848
549
- 539 64.93506494
550
- 540 64.81481481
551
- 541 64.69500924
552
- 542 64.76014760
553
- 543 64.64088398
554
- 544 64.70588235
555
- 545 64.77064220
556
- 546 64.65201465
557
- 547 64.53382084
558
- 548 64.59854015
559
- 549 64.66302368
560
- 550 64.72727273
561
- 551 64.60980036
562
- 552 64.49275362
563
- 553 64.55696203
564
- 554 64.62093863
565
- 555 64.68468468
566
- 556 64.56834532
567
- 557 64.63195691
568
- 558 64.51612903
569
- 559 64.57960644
570
- 560 64.64285714
571
- 561 64.70588235
572
- 562 64.76868327
573
- 563 64.83126110
574
- 564 64.89361702
575
- 565 64.77876106
576
- 566 64.84098940
577
- 567 64.72663139
578
- 568 64.78873239
579
- 569 64.67486819
580
- 570 64.56140351
581
- 571 64.62346760
582
- 572 64.51048951
583
- 573 64.57242583
584
- 574 64.63414634
585
- 575 64.52173913
586
- 576 64.58333333
587
- 577 64.64471404
588
- 578 64.70588235
589
- 579 64.76683938
590
- 580 64.82758621
591
- 581 64.71600688
592
- 582 64.60481100
593
- 583 64.66552316
594
- 584 64.72602740
595
- 585 64.78632479
596
- 586 64.67576792
597
- 587 64.73594549
598
- 588 64.79591837
599
- 589 64.85568761
600
- 590 64.91525424
601
- 591 64.97461929
602
- 592 64.86486486
603
- 593 64.75548061
604
- 594 64.81481481
605
- 595 64.70588235
606
- 596 64.59731544
607
- 597 64.48911223
608
- 598 64.54849498
609
- 599 64.60767947
610
- 600 64.50000000
611
- 601 64.55906822
612
- 602 64.45182724
613
- 603 64.34494196
614
- 604 64.40397351
615
- 605 64.46280992
616
- 606 64.52145215
617
- 607 64.57990115
618
- 608 64.63815789
619
- 609 64.53201970
620
- 610 64.59016393
621
- 611 64.48445172
622
- 612 64.54248366
623
- 613 64.43719413
624
- 614 64.49511401
625
- 615 64.39024390
626
- 616 64.28571429
627
- 617 64.34359806
628
- 618 64.40129450
629
- 619 64.29725363
630
- 620 64.35483871
631
- 621 64.41223833
632
- 622 64.30868167
633
- 623 64.20545746
634
- 624 64.10256410
635
- 625 64.00000000
636
- 626 63.89776358
637
- 627 63.95534290
638
- 628 64.01273885
639
- 629 63.91096979
640
- 630 63.96825397
641
- 631 64.02535658
642
- 632 63.92405063
643
- 633 63.82306477
644
- 634 63.88012618
645
- 635 63.93700787
646
- 636 63.99371069
647
- 637 63.89324961
648
- 638 63.94984326
649
- 639 64.00625978
650
- 640 63.90625000
651
- 641 63.96255850
652
- 642 63.86292835
653
- 643 63.76360809
654
- 644 63.81987578
655
- 645 63.72093023
656
- 646 63.77708978
657
- 647 63.83307573
658
- 648 63.73456790
659
- 649 63.63636364
660
- 650 63.69230769
661
- 651 63.74807988
662
- 652 63.65030675
663
- 653 63.55283308
664
- 654 63.60856269
665
- 655 63.51145038
666
- 656 63.41463415
667
- 657 63.31811263
668
- 658 63.22188450
669
- 659 63.12594841
670
- 660 63.18181818
671
- 661 63.08623298
672
- 662 63.14199396
673
- 663 63.19758673
674
- 664 63.25301205
675
- 665 63.30827068
676
- 666 63.21321321
677
- 667 63.26836582
678
- 668 63.17365269
679
- 669 63.07922272
680
- 670 63.13432836
681
- 671 63.04023845
682
- 672 63.09523810
683
- 673 63.00148588
684
- 674 62.90801187
685
- 675 62.81481481
686
- 676 62.72189349
687
- 677 62.77695716
688
- 678 62.83185841
689
- 679 62.88659794
690
- 680 62.94117647
691
- 681 62.84875184
692
- 682 62.75659824
693
- 683 62.66471449
694
- 684 62.71929825
695
- 685 62.77372263
696
- 686 62.68221574
697
- 687 62.59097525
698
- 688 62.64534884
699
- 689 62.55442671
700
- 690 62.60869565
701
- 691 62.66280753
702
- 692 62.71676301
703
- 693 62.77056277
704
- 694 62.82420749
705
- 695 62.73381295
706
- 696 62.64367816
707
- 697 62.55380201
708
- 698 62.46418338
709
- 699 62.51788269
710
- 700 62.57142857
711
- 701 62.48216833
712
- 702 62.53561254
713
- 703 62.58890469
714
- 704 62.64204545
715
- 705 62.69503546
716
- 706 62.74787535
717
- 707 62.80056577
718
- 708 62.85310734
719
- 709 62.90550071
720
- 710 62.95774648
721
- 711 63.00984529
722
- 712 63.06179775
723
- 713 63.11360449
724
- 714 63.02521008
725
- 715 62.93706294
726
- 716 62.98882682
727
- 717 62.90097629
728
- 718 62.81337047
729
- 719 62.72600834
730
- 720 62.77777778
731
- 721 62.82940361
732
- 722 62.88088643
733
- 723 62.93222683
734
- 724 62.84530387
735
- 725 62.89655172
736
- 726 62.80991736
737
- 727 62.72352132
738
- 728 62.77472527
739
- 729 62.82578875
740
- 730 62.87671233
741
- 731 62.92749658
742
- 732 62.97814208
743
- 733 63.02864939
744
- 734 63.07901907
745
- 735 63.12925170
746
- 736 63.17934783
747
- 737 63.09362280
748
- 738 63.14363144
749
- 739 63.05818674
750
- 740 62.97297297
751
- 741 63.02294197
752
- 742 62.93800539
753
- 743 62.98788694
754
- 744 62.90322581
755
- 745 62.95302013
756
- 746 63.00268097
757
- 747 62.91834003
758
- 748 62.83422460
759
- 749 62.75033378
760
- 750 62.80000000
761
 
762
- Final result: 62.8000 +/- 1.7661
763
- Random chance: 25.0083 +/- 1.5824
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q3_K_L.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 62.6506 +/- 1.7711
6
+ Random chance: 25.0251 +/- 1.5859
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 1624.57 ms
10
+ llama_perf_context_print: prompt eval time = 171694.83 ms / 36304 tokens ( 4.73 ms per token, 211.44 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 173922.19 ms / 36305 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-q3_k_l.hsw CHANGED
@@ -1,760 +1,12 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- hellaswag_score : loaded 10042 tasks from prompt.
6
- ================================= is_spm = 0
7
- hellaswag_score : selecting 750 randomized tasks.
8
- hellaswag_score : calculating hellaswag score over selected tasks.
9
 
10
- task acc_norm
11
- 1 0.00000000
12
- 2 50.00000000
13
- 3 66.66666667
14
- 4 75.00000000
15
- 5 80.00000000
16
- 6 83.33333333
17
- 7 71.42857143
18
- 8 75.00000000
19
- 9 66.66666667
20
- 10 70.00000000
21
- 11 72.72727273
22
- 12 66.66666667
23
- 13 69.23076923
24
- 14 71.42857143
25
- 15 73.33333333
26
- 16 75.00000000
27
- 17 70.58823529
28
- 18 72.22222222
29
- 19 68.42105263
30
- 20 70.00000000
31
- 21 71.42857143
32
- 22 72.72727273
33
- 23 73.91304348
34
- 24 70.83333333
35
- 25 68.00000000
36
- 26 65.38461538
37
- 27 62.96296296
38
- 28 64.28571429
39
- 29 65.51724138
40
- 30 63.33333333
41
- 31 64.51612903
42
- 32 62.50000000
43
- 33 63.63636364
44
- 34 64.70588235
45
- 35 65.71428571
46
- 36 63.88888889
47
- 37 62.16216216
48
- 38 63.15789474
49
- 39 64.10256410
50
- 40 62.50000000
51
- 41 63.41463415
52
- 42 61.90476190
53
- 43 62.79069767
54
- 44 63.63636364
55
- 45 62.22222222
56
- 46 63.04347826
57
- 47 63.82978723
58
- 48 64.58333333
59
- 49 65.30612245
60
- 50 64.00000000
61
- 51 64.70588235
62
- 52 63.46153846
63
- 53 62.26415094
64
- 54 61.11111111
65
- 55 60.00000000
66
- 56 58.92857143
67
- 57 59.64912281
68
- 58 60.34482759
69
- 59 61.01694915
70
- 60 61.66666667
71
- 61 60.65573770
72
- 62 61.29032258
73
- 63 61.90476190
74
- 64 62.50000000
75
- 65 63.07692308
76
- 66 62.12121212
77
- 67 62.68656716
78
- 68 63.23529412
79
- 69 63.76811594
80
- 70 64.28571429
81
- 71 63.38028169
82
- 72 63.88888889
83
- 73 64.38356164
84
- 74 64.86486486
85
- 75 64.00000000
86
- 76 64.47368421
87
- 77 64.93506494
88
- 78 65.38461538
89
- 79 65.82278481
90
- 80 66.25000000
91
- 81 66.66666667
92
- 82 67.07317073
93
- 83 67.46987952
94
- 84 67.85714286
95
- 85 68.23529412
96
- 86 68.60465116
97
- 87 68.96551724
98
- 88 69.31818182
99
- 89 68.53932584
100
- 90 68.88888889
101
- 91 69.23076923
102
- 92 68.47826087
103
- 93 68.81720430
104
- 94 69.14893617
105
- 95 69.47368421
106
- 96 69.79166667
107
- 97 70.10309278
108
- 98 70.40816327
109
- 99 70.70707071
110
- 100 70.00000000
111
- 101 70.29702970
112
- 102 69.60784314
113
- 103 68.93203883
114
- 104 69.23076923
115
- 105 69.52380952
116
- 106 69.81132075
117
- 107 70.09345794
118
- 108 70.37037037
119
- 109 69.72477064
120
- 110 70.00000000
121
- 111 70.27027027
122
- 112 70.53571429
123
- 113 70.79646018
124
- 114 71.05263158
125
- 115 71.30434783
126
- 116 70.68965517
127
- 117 70.08547009
128
- 118 70.33898305
129
- 119 70.58823529
130
- 120 70.00000000
131
- 121 70.24793388
132
- 122 70.49180328
133
- 123 70.73170732
134
- 124 70.96774194
135
- 125 71.20000000
136
- 126 71.42857143
137
- 127 71.65354331
138
- 128 71.09375000
139
- 129 71.31782946
140
- 130 70.76923077
141
- 131 70.99236641
142
- 132 71.21212121
143
- 133 71.42857143
144
- 134 71.64179104
145
- 135 71.85185185
146
- 136 72.05882353
147
- 137 72.26277372
148
- 138 72.46376812
149
- 139 71.94244604
150
- 140 72.14285714
151
- 141 72.34042553
152
- 142 72.53521127
153
- 143 72.72727273
154
- 144 72.91666667
155
- 145 73.10344828
156
- 146 73.28767123
157
- 147 73.46938776
158
- 148 73.64864865
159
- 149 73.82550336
160
- 150 74.00000000
161
- 151 73.50993377
162
- 152 73.02631579
163
- 153 73.20261438
164
- 154 73.37662338
165
- 155 73.54838710
166
- 156 73.71794872
167
- 157 73.24840764
168
- 158 73.41772152
169
- 159 72.95597484
170
- 160 73.12500000
171
- 161 73.29192547
172
- 162 73.45679012
173
- 163 73.61963190
174
- 164 73.78048780
175
- 165 73.93939394
176
- 166 74.09638554
177
- 167 74.25149701
178
- 168 74.40476190
179
- 169 73.96449704
180
- 170 74.11764706
181
- 171 74.26900585
182
- 172 74.41860465
183
- 173 73.98843931
184
- 174 74.13793103
185
- 175 74.28571429
186
- 176 74.43181818
187
- 177 74.57627119
188
- 178 74.71910112
189
- 179 74.30167598
190
- 180 74.44444444
191
- 181 74.03314917
192
- 182 74.17582418
193
- 183 74.31693989
194
- 184 73.91304348
195
- 185 74.05405405
196
- 186 73.65591398
197
- 187 73.26203209
198
- 188 73.40425532
199
- 189 73.01587302
200
- 190 73.15789474
201
- 191 73.29842932
202
- 192 73.43750000
203
- 193 73.05699482
204
- 194 73.19587629
205
- 195 72.82051282
206
- 196 72.95918367
207
- 197 72.58883249
208
- 198 72.22222222
209
- 199 72.36180905
210
- 200 72.50000000
211
- 201 72.63681592
212
- 202 72.77227723
213
- 203 72.41379310
214
- 204 72.54901961
215
- 205 72.68292683
216
- 206 72.81553398
217
- 207 72.94685990
218
- 208 72.59615385
219
- 209 72.72727273
220
- 210 72.85714286
221
- 211 72.98578199
222
- 212 73.11320755
223
- 213 73.23943662
224
- 214 73.36448598
225
- 215 73.02325581
226
- 216 73.14814815
227
- 217 72.81105991
228
- 218 72.93577982
229
- 219 73.05936073
230
- 220 73.18181818
231
- 221 72.85067873
232
- 222 72.97297297
233
- 223 73.09417040
234
- 224 73.21428571
235
- 225 73.33333333
236
- 226 73.45132743
237
- 227 73.12775330
238
- 228 73.24561404
239
- 229 73.36244541
240
- 230 73.47826087
241
- 231 73.16017316
242
- 232 72.84482759
243
- 233 72.96137339
244
- 234 73.07692308
245
- 235 72.76595745
246
- 236 72.45762712
247
- 237 72.57383966
248
- 238 72.68907563
249
- 239 72.80334728
250
- 240 72.91666667
251
- 241 73.02904564
252
- 242 73.14049587
253
- 243 73.25102881
254
- 244 72.95081967
255
- 245 72.65306122
256
- 246 72.35772358
257
- 247 72.46963563
258
- 248 72.58064516
259
- 249 72.69076305
260
- 250 72.80000000
261
- 251 72.90836653
262
- 252 73.01587302
263
- 253 72.72727273
264
- 254 72.83464567
265
- 255 72.94117647
266
- 256 73.04687500
267
- 257 72.76264591
268
- 258 72.86821705
269
- 259 72.97297297
270
- 260 73.07692308
271
- 261 73.18007663
272
- 262 72.90076336
273
- 263 73.00380228
274
- 264 73.10606061
275
- 265 73.20754717
276
- 266 73.30827068
277
- 267 73.40823970
278
- 268 73.50746269
279
- 269 73.60594796
280
- 270 73.70370370
281
- 271 73.43173432
282
- 272 73.52941176
283
- 273 73.26007326
284
- 274 73.35766423
285
- 275 73.09090909
286
- 276 73.18840580
287
- 277 73.28519856
288
- 278 73.38129496
289
- 279 73.47670251
290
- 280 73.57142857
291
- 281 73.66548043
292
- 282 73.75886525
293
- 283 73.85159011
294
- 284 73.94366197
295
- 285 73.68421053
296
- 286 73.77622378
297
- 287 73.86759582
298
- 288 73.95833333
299
- 289 74.04844291
300
- 290 74.13793103
301
- 291 74.22680412
302
- 292 73.97260274
303
- 293 74.06143345
304
- 294 74.14965986
305
- 295 74.23728814
306
- 296 74.32432432
307
- 297 74.41077441
308
- 298 74.49664430
309
- 299 74.24749164
310
- 300 74.00000000
311
- 301 74.08637874
312
- 302 74.17218543
313
- 303 73.92739274
314
- 304 73.68421053
315
- 305 73.77049180
316
- 306 73.85620915
317
- 307 73.61563518
318
- 308 73.70129870
319
- 309 73.78640777
320
- 310 73.87096774
321
- 311 73.95498392
322
- 312 74.03846154
323
- 313 74.12140575
324
- 314 74.20382166
325
- 315 74.28571429
326
- 316 74.36708861
327
- 317 74.44794953
328
- 318 74.21383648
329
- 319 73.98119122
330
- 320 74.06250000
331
- 321 73.83177570
332
- 322 73.60248447
333
- 323 73.68421053
334
- 324 73.76543210
335
- 325 73.84615385
336
- 326 73.92638037
337
- 327 73.70030581
338
- 328 73.78048780
339
- 329 73.86018237
340
- 330 73.93939394
341
- 331 74.01812689
342
- 332 74.09638554
343
- 333 74.17417417
344
- 334 73.95209581
345
- 335 74.02985075
346
- 336 74.10714286
347
- 337 73.88724036
348
- 338 73.66863905
349
- 339 73.74631268
350
- 340 73.82352941
351
- 341 73.90029326
352
- 342 73.97660819
353
- 343 74.05247813
354
- 344 73.83720930
355
- 345 73.91304348
356
- 346 73.69942197
357
- 347 73.77521614
358
- 348 73.56321839
359
- 349 73.35243553
360
- 350 73.42857143
361
- 351 73.50427350
362
- 352 73.57954545
363
- 353 73.65439093
364
- 354 73.72881356
365
- 355 73.80281690
366
- 356 73.87640449
367
- 357 73.94957983
368
- 358 74.02234637
369
- 359 74.09470752
370
- 360 73.88888889
371
- 361 73.96121884
372
- 362 74.03314917
373
- 363 74.10468320
374
- 364 74.17582418
375
- 365 74.24657534
376
- 366 74.31693989
377
- 367 74.38692098
378
- 368 74.45652174
379
- 369 74.52574526
380
- 370 74.59459459
381
- 371 74.66307278
382
- 372 74.73118280
383
- 373 74.79892761
384
- 374 74.86631016
385
- 375 74.93333333
386
- 376 75.00000000
387
- 377 74.80106101
388
- 378 74.86772487
389
- 379 74.93403694
390
- 380 75.00000000
391
- 381 75.06561680
392
- 382 75.13089005
393
- 383 75.19582245
394
- 384 75.26041667
395
- 385 75.32467532
396
- 386 75.38860104
397
- 387 75.45219638
398
- 388 75.25773196
399
- 389 75.32133676
400
- 390 75.38461538
401
- 391 75.19181586
402
- 392 75.25510204
403
- 393 75.31806616
404
- 394 75.38071066
405
- 395 75.44303797
406
- 396 75.50505051
407
- 397 75.56675063
408
- 398 75.62814070
409
- 399 75.68922306
410
- 400 75.75000000
411
- 401 75.81047382
412
- 402 75.87064677
413
- 403 75.68238213
414
- 404 75.74257426
415
- 405 75.55555556
416
- 406 75.61576355
417
- 407 75.67567568
418
- 408 75.73529412
419
- 409 75.79462103
420
- 410 75.85365854
421
- 411 75.66909976
422
- 412 75.72815534
423
- 413 75.54479419
424
- 414 75.60386473
425
- 415 75.42168675
426
- 416 75.48076923
427
- 417 75.53956835
428
- 418 75.59808612
429
- 419 75.65632458
430
- 420 75.71428571
431
- 421 75.53444181
432
- 422 75.59241706
433
- 423 75.65011820
434
- 424 75.70754717
435
- 425 75.76470588
436
- 426 75.82159624
437
- 427 75.64402810
438
- 428 75.70093458
439
- 429 75.75757576
440
- 430 75.81395349
441
- 431 75.63805104
442
- 432 75.46296296
443
- 433 75.28868360
444
- 434 75.34562212
445
- 435 75.17241379
446
- 436 75.22935780
447
- 437 75.05720824
448
- 438 74.88584475
449
- 439 74.94305239
450
- 440 74.77272727
451
- 441 74.82993197
452
- 442 74.88687783
453
- 443 74.94356659
454
- 444 75.00000000
455
- 445 75.05617978
456
- 446 75.11210762
457
- 447 75.16778523
458
- 448 75.22321429
459
- 449 75.27839644
460
- 450 75.33333333
461
- 451 75.16629712
462
- 452 75.00000000
463
- 453 75.05518764
464
- 454 75.11013216
465
- 455 75.16483516
466
- 456 75.21929825
467
- 457 75.27352298
468
- 458 75.32751092
469
- 459 75.38126362
470
- 460 75.43478261
471
- 461 75.48806941
472
- 462 75.54112554
473
- 463 75.59395248
474
- 464 75.64655172
475
- 465 75.48387097
476
- 466 75.53648069
477
- 467 75.58886510
478
- 468 75.42735043
479
- 469 75.47974414
480
- 470 75.31914894
481
- 471 75.37154989
482
- 472 75.42372881
483
- 473 75.47568710
484
- 474 75.52742616
485
- 475 75.57894737
486
- 476 75.63025210
487
- 477 75.47169811
488
- 478 75.52301255
489
- 479 75.57411273
490
- 480 75.62500000
491
- 481 75.67567568
492
- 482 75.51867220
493
- 483 75.56935818
494
- 484 75.41322314
495
- 485 75.25773196
496
- 486 75.30864198
497
- 487 75.35934292
498
- 488 75.20491803
499
- 489 75.25562372
500
- 490 75.30612245
501
- 491 75.35641548
502
- 492 75.40650407
503
- 493 75.45638945
504
- 494 75.50607287
505
- 495 75.55555556
506
- 496 75.60483871
507
- 497 75.65392354
508
- 498 75.70281124
509
- 499 75.55110220
510
- 500 75.60000000
511
- 501 75.44910180
512
- 502 75.49800797
513
- 503 75.54671968
514
- 504 75.59523810
515
- 505 75.64356436
516
- 506 75.69169960
517
- 507 75.73964497
518
- 508 75.78740157
519
- 509 75.83497053
520
- 510 75.88235294
521
- 511 75.92954990
522
- 512 75.78125000
523
- 513 75.63352827
524
- 514 75.68093385
525
- 515 75.72815534
526
- 516 75.58139535
527
- 517 75.43520309
528
- 518 75.48262548
529
- 519 75.52986513
530
- 520 75.57692308
531
- 521 75.43186180
532
- 522 75.28735632
533
- 523 75.33460803
534
- 524 75.38167939
535
- 525 75.42857143
536
- 526 75.28517110
537
- 527 75.14231499
538
- 528 75.18939394
539
- 529 75.23629490
540
- 530 75.28301887
541
- 531 75.32956685
542
- 532 75.37593985
543
- 533 75.42213884
544
- 534 75.46816479
545
- 535 75.51401869
546
- 536 75.55970149
547
- 537 75.60521415
548
- 538 75.65055762
549
- 539 75.51020408
550
- 540 75.55555556
551
- 541 75.60073937
552
- 542 75.64575646
553
- 543 75.50644567
554
- 544 75.55147059
555
- 545 75.59633028
556
- 546 75.64102564
557
- 547 75.68555759
558
- 548 75.72992701
559
- 549 75.77413479
560
- 550 75.63636364
561
- 551 75.68058076
562
- 552 75.54347826
563
- 553 75.58770344
564
- 554 75.63176895
565
- 555 75.49549550
566
- 556 75.53956835
567
- 557 75.40394973
568
- 558 75.26881720
569
- 559 75.31305903
570
- 560 75.35714286
571
- 561 75.40106952
572
- 562 75.44483986
573
- 563 75.48845471
574
- 564 75.53191489
575
- 565 75.39823009
576
- 566 75.26501767
577
- 567 75.30864198
578
- 568 75.35211268
579
- 569 75.21968366
580
- 570 75.26315789
581
- 571 75.30647986
582
- 572 75.34965035
583
- 573 75.39267016
584
- 574 75.43554007
585
- 575 75.47826087
586
- 576 75.52083333
587
- 577 75.56325823
588
- 578 75.60553633
589
- 579 75.64766839
590
- 580 75.68965517
591
- 581 75.55938038
592
- 582 75.60137457
593
- 583 75.64322470
594
- 584 75.68493151
595
- 585 75.72649573
596
- 586 75.76791809
597
- 587 75.80919932
598
- 588 75.68027211
599
- 589 75.55178268
600
- 590 75.59322034
601
- 591 75.63451777
602
- 592 75.67567568
603
- 593 75.54806071
604
- 594 75.42087542
605
- 595 75.46218487
606
- 596 75.50335570
607
- 597 75.54438861
608
- 598 75.58528428
609
- 599 75.45909850
610
- 600 75.50000000
611
- 601 75.54076539
612
- 602 75.58139535
613
- 603 75.62189055
614
- 604 75.66225166
615
- 605 75.70247934
616
- 606 75.57755776
617
- 607 75.61779242
618
- 608 75.49342105
619
- 609 75.53366174
620
- 610 75.57377049
621
- 611 75.61374795
622
- 612 75.49019608
623
- 613 75.36704731
624
- 614 75.40716612
625
- 615 75.44715447
626
- 616 75.32467532
627
- 617 75.36466775
628
- 618 75.40453074
629
- 619 75.44426494
630
- 620 75.48387097
631
- 621 75.52334944
632
- 622 75.56270096
633
- 623 75.60192616
634
- 624 75.64102564
635
- 625 75.68000000
636
- 626 75.71884984
637
- 627 75.75757576
638
- 628 75.79617834
639
- 629 75.67567568
640
- 630 75.55555556
641
- 631 75.59429477
642
- 632 75.47468354
643
- 633 75.35545024
644
- 634 75.39432177
645
- 635 75.43307087
646
- 636 75.47169811
647
- 637 75.51020408
648
- 638 75.54858934
649
- 639 75.58685446
650
- 640 75.62500000
651
- 641 75.66302652
652
- 642 75.70093458
653
- 643 75.73872473
654
- 644 75.77639752
655
- 645 75.65891473
656
- 646 75.69659443
657
- 647 75.73415765
658
- 648 75.77160494
659
- 649 75.65485362
660
- 650 75.69230769
661
- 651 75.72964670
662
- 652 75.76687117
663
- 653 75.80398162
664
- 654 75.84097859
665
- 655 75.87786260
666
- 656 75.91463415
667
- 657 75.95129376
668
- 658 75.98784195
669
- 659 75.87253414
670
- 660 75.75757576
671
- 661 75.79425113
672
- 662 75.83081571
673
- 663 75.86726998
674
- 664 75.90361446
675
- 665 75.93984962
676
- 666 75.97597598
677
- 667 76.01199400
678
- 668 76.04790419
679
- 669 76.08370703
680
- 670 76.11940299
681
- 671 76.15499255
682
- 672 76.19047619
683
- 673 76.22585438
684
- 674 76.26112760
685
- 675 76.29629630
686
- 676 76.33136095
687
- 677 76.36632201
688
- 678 76.40117994
689
- 679 76.43593520
690
- 680 76.47058824
691
- 681 76.35829662
692
- 682 76.39296188
693
- 683 76.42752562
694
- 684 76.46198830
695
- 685 76.49635036
696
- 686 76.53061224
697
- 687 76.56477438
698
- 688 76.59883721
699
- 689 76.63280116
700
- 690 76.66666667
701
- 691 76.70043415
702
- 692 76.73410405
703
- 693 76.62337662
704
- 694 76.65706052
705
- 695 76.69064748
706
- 696 76.72413793
707
- 697 76.75753228
708
- 698 76.79083095
709
- 699 76.68097282
710
- 700 76.71428571
711
- 701 76.74750357
712
- 702 76.78062678
713
- 703 76.81365576
714
- 704 76.84659091
715
- 705 76.87943262
716
- 706 76.91218130
717
- 707 76.94483734
718
- 708 76.97740113
719
- 709 77.00987306
720
- 710 77.04225352
721
- 711 76.93389592
722
- 712 76.96629213
723
- 713 76.99859748
724
- 714 77.03081232
725
- 715 76.92307692
726
- 716 76.95530726
727
- 717 76.98744770
728
- 718 77.01949861
729
- 719 77.05146036
730
- 720 77.08333333
731
- 721 77.11511789
732
- 722 77.14681440
733
- 723 77.17842324
734
- 724 77.20994475
735
- 725 77.24137931
736
- 726 77.27272727
737
- 727 77.30398900
738
- 728 77.33516484
739
- 729 77.22908093
740
- 730 77.26027397
741
- 731 77.29138167
742
- 732 77.18579235
743
- 733 77.21691678
744
- 734 77.24795640
745
- 735 77.14285714
746
- 736 77.17391304
747
- 737 77.06919946
748
- 738 76.96476965
749
- 739 76.86062246
750
- 740 76.89189189
751
- 741 76.92307692
752
- 742 76.95417790
753
- 743 76.98519515
754
- 744 77.01612903
755
- 745 77.04697987
756
- 746 77.07774799
757
- 747 76.97456493
758
- 748 77.00534759
759
- 749 77.03604806
760
- 750 77.06666667
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q3_K_L.gguf (version GGUF V3 (latest))
4
 
5
+ 750 74.66666667
 
 
 
 
6
 
7
+
8
+ llama_perf_context_print: load time = 308.34 ms
9
+ llama_perf_context_print: prompt eval time = 469602.68 ms / 125000 tokens ( 3.76 ms per token, 266.18 tokens per second)
10
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
11
+ llama_perf_context_print: total time = 475338.97 ms / 125001 tokens
12
+ ggml_metal_free: deallocating
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scores/Watt-Tool-8B-q3_k_l.mmlu CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 1548 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 1548 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 50.00000000
13
- 3 33.33333333
14
- 4 50.00000000
15
- 5 40.00000000
16
- 6 33.33333333
17
- 7 42.85714286
18
- 8 50.00000000
19
- 9 44.44444444
20
- 10 50.00000000
21
- 11 45.45454545
22
- 12 50.00000000
23
- 13 46.15384615
24
- 14 42.85714286
25
- 15 46.66666667
26
- 16 50.00000000
27
- 17 52.94117647
28
- 18 50.00000000
29
- 19 52.63157895
30
- 20 55.00000000
31
- 21 57.14285714
32
- 22 54.54545455
33
- 23 52.17391304
34
- 24 54.16666667
35
- 25 56.00000000
36
- 26 53.84615385
37
- 27 55.55555556
38
- 28 57.14285714
39
- 29 58.62068966
40
- 30 60.00000000
41
- 31 61.29032258
42
- 32 62.50000000
43
- 33 60.60606061
44
- 34 61.76470588
45
- 35 62.85714286
46
- 36 63.88888889
47
- 37 62.16216216
48
- 38 60.52631579
49
- 39 58.97435897
50
- 40 57.50000000
51
- 41 56.09756098
52
- 42 54.76190476
53
- 43 55.81395349
54
- 44 54.54545455
55
- 45 55.55555556
56
- 46 54.34782609
57
- 47 53.19148936
58
- 48 54.16666667
59
- 49 53.06122449
60
- 50 52.00000000
61
- 51 50.98039216
62
- 52 50.00000000
63
- 53 49.05660377
64
- 54 48.14814815
65
- 55 49.09090909
66
- 56 50.00000000
67
- 57 50.87719298
68
- 58 51.72413793
69
- 59 52.54237288
70
- 60 51.66666667
71
- 61 52.45901639
72
- 62 51.61290323
73
- 63 50.79365079
74
- 64 50.00000000
75
- 65 49.23076923
76
- 66 48.48484848
77
- 67 47.76119403
78
- 68 47.05882353
79
- 69 46.37681159
80
- 70 47.14285714
81
- 71 47.88732394
82
- 72 47.22222222
83
- 73 46.57534247
84
- 74 47.29729730
85
- 75 48.00000000
86
- 76 48.68421053
87
- 77 49.35064935
88
- 78 50.00000000
89
- 79 49.36708861
90
- 80 50.00000000
91
- 81 50.61728395
92
- 82 50.00000000
93
- 83 50.60240964
94
- 84 51.19047619
95
- 85 50.58823529
96
- 86 50.00000000
97
- 87 50.57471264
98
- 88 51.13636364
99
- 89 50.56179775
100
- 90 50.00000000
101
- 91 49.45054945
102
- 92 48.91304348
103
- 93 48.38709677
104
- 94 47.87234043
105
- 95 47.36842105
106
- 96 47.91666667
107
- 97 48.45360825
108
- 98 47.95918367
109
- 99 48.48484848
110
- 100 48.00000000
111
- 101 48.51485149
112
- 102 48.03921569
113
- 103 47.57281553
114
- 104 48.07692308
115
- 105 48.57142857
116
- 106 49.05660377
117
- 107 48.59813084
118
- 108 48.14814815
119
- 109 47.70642202
120
- 110 48.18181818
121
- 111 48.64864865
122
- 112 49.10714286
123
- 113 48.67256637
124
- 114 49.12280702
125
- 115 49.56521739
126
- 116 50.00000000
127
- 117 50.42735043
128
- 118 50.00000000
129
- 119 50.42016807
130
- 120 50.00000000
131
- 121 50.41322314
132
- 122 50.00000000
133
- 123 49.59349593
134
- 124 50.00000000
135
- 125 50.40000000
136
- 126 50.00000000
137
- 127 49.60629921
138
- 128 49.21875000
139
- 129 48.83720930
140
- 130 48.46153846
141
- 131 48.09160305
142
- 132 48.48484848
143
- 133 48.12030075
144
- 134 47.76119403
145
- 135 47.40740741
146
- 136 47.05882353
147
- 137 46.71532847
148
- 138 46.37681159
149
- 139 46.76258993
150
- 140 47.14285714
151
- 141 47.51773050
152
- 142 47.88732394
153
- 143 48.25174825
154
- 144 47.91666667
155
- 145 47.58620690
156
- 146 47.94520548
157
- 147 47.61904762
158
- 148 47.97297297
159
- 149 47.65100671
160
- 150 47.33333333
161
- 151 47.01986755
162
- 152 47.36842105
163
- 153 47.71241830
164
- 154 47.40259740
165
- 155 47.09677419
166
- 156 46.79487179
167
- 157 47.13375796
168
- 158 47.46835443
169
- 159 47.16981132
170
- 160 47.50000000
171
- 161 47.20496894
172
- 162 46.91358025
173
- 163 46.62576687
174
- 164 46.34146341
175
- 165 46.66666667
176
- 166 46.38554217
177
- 167 46.70658683
178
- 168 46.42857143
179
- 169 46.15384615
180
- 170 45.88235294
181
- 171 46.19883041
182
- 172 45.93023256
183
- 173 45.66473988
184
- 174 45.40229885
185
- 175 45.14285714
186
- 176 45.45454545
187
- 177 45.19774011
188
- 178 44.94382022
189
- 179 45.25139665
190
- 180 45.00000000
191
- 181 44.75138122
192
- 182 44.50549451
193
- 183 44.80874317
194
- 184 44.56521739
195
- 185 44.86486486
196
- 186 45.16129032
197
- 187 44.91978610
198
- 188 44.68085106
199
- 189 44.97354497
200
- 190 44.73684211
201
- 191 45.02617801
202
- 192 45.31250000
203
- 193 45.07772021
204
- 194 45.36082474
205
- 195 45.64102564
206
- 196 45.91836735
207
- 197 45.68527919
208
- 198 45.95959596
209
- 199 45.72864322
210
- 200 45.50000000
211
- 201 45.27363184
212
- 202 45.04950495
213
- 203 44.82758621
214
- 204 44.60784314
215
- 205 44.87804878
216
- 206 44.66019417
217
- 207 44.92753623
218
- 208 44.71153846
219
- 209 44.49760766
220
- 210 44.28571429
221
- 211 44.54976303
222
- 212 44.33962264
223
- 213 44.13145540
224
- 214 43.92523364
225
- 215 44.18604651
226
- 216 44.44444444
227
- 217 44.23963134
228
- 218 44.49541284
229
- 219 44.74885845
230
- 220 45.00000000
231
- 221 44.79638009
232
- 222 44.59459459
233
- 223 44.84304933
234
- 224 45.08928571
235
- 225 45.33333333
236
- 226 45.13274336
237
- 227 44.93392070
238
- 228 45.17543860
239
- 229 44.97816594
240
- 230 44.78260870
241
- 231 44.58874459
242
- 232 44.39655172
243
- 233 44.20600858
244
- 234 44.44444444
245
- 235 44.25531915
246
- 236 44.06779661
247
- 237 43.88185654
248
- 238 43.69747899
249
- 239 43.51464435
250
- 240 43.75000000
251
- 241 43.56846473
252
- 242 43.80165289
253
- 243 44.03292181
254
- 244 43.85245902
255
- 245 44.08163265
256
- 246 43.90243902
257
- 247 43.72469636
258
- 248 43.54838710
259
- 249 43.37349398
260
- 250 43.20000000
261
- 251 43.02788845
262
- 252 42.85714286
263
- 253 42.68774704
264
- 254 42.51968504
265
- 255 42.35294118
266
- 256 42.57812500
267
- 257 42.80155642
268
- 258 42.63565891
269
- 259 42.47104247
270
- 260 42.69230769
271
- 261 42.52873563
272
- 262 42.36641221
273
- 263 42.20532319
274
- 264 42.04545455
275
- 265 42.26415094
276
- 266 42.48120301
277
- 267 42.32209738
278
- 268 42.16417910
279
- 269 42.37918216
280
- 270 42.59259259
281
- 271 42.43542435
282
- 272 42.64705882
283
- 273 42.49084249
284
- 274 42.33576642
285
- 275 42.18181818
286
- 276 42.39130435
287
- 277 42.59927798
288
- 278 42.44604317
289
- 279 42.29390681
290
- 280 42.14285714
291
- 281 42.34875445
292
- 282 42.19858156
293
- 283 42.04946996
294
- 284 41.90140845
295
- 285 41.75438596
296
- 286 41.60839161
297
- 287 41.81184669
298
- 288 41.66666667
299
- 289 41.52249135
300
- 290 41.72413793
301
- 291 41.92439863
302
- 292 42.12328767
303
- 293 42.32081911
304
- 294 42.17687075
305
- 295 42.03389831
306
- 296 41.89189189
307
- 297 41.75084175
308
- 298 41.94630872
309
- 299 42.14046823
310
- 300 42.33333333
311
- 301 42.19269103
312
- 302 42.38410596
313
- 303 42.24422442
314
- 304 42.10526316
315
- 305 42.29508197
316
- 306 42.15686275
317
- 307 42.34527687
318
- 308 42.20779221
319
- 309 42.07119741
320
- 310 41.93548387
321
- 311 42.12218650
322
- 312 42.30769231
323
- 313 42.49201278
324
- 314 42.67515924
325
- 315 42.53968254
326
- 316 42.40506329
327
- 317 42.58675079
328
- 318 42.45283019
329
- 319 42.63322884
330
- 320 42.50000000
331
- 321 42.36760125
332
- 322 42.54658385
333
- 323 42.41486068
334
- 324 42.28395062
335
- 325 42.46153846
336
- 326 42.33128834
337
- 327 42.50764526
338
- 328 42.37804878
339
- 329 42.24924012
340
- 330 42.12121212
341
- 331 41.99395770
342
- 332 41.86746988
343
- 333 42.04204204
344
- 334 41.91616766
345
- 335 42.08955224
346
- 336 41.96428571
347
- 337 41.83976261
348
- 338 41.71597633
349
- 339 41.59292035
350
- 340 41.76470588
351
- 341 41.64222874
352
- 342 41.81286550
353
- 343 41.69096210
354
- 344 41.56976744
355
- 345 41.44927536
356
- 346 41.61849711
357
- 347 41.78674352
358
- 348 41.66666667
359
- 349 41.54727794
360
- 350 41.71428571
361
- 351 41.59544160
362
- 352 41.47727273
363
- 353 41.64305949
364
- 354 41.52542373
365
- 355 41.40845070
366
- 356 41.57303371
367
- 357 41.45658263
368
- 358 41.34078212
369
- 359 41.22562674
370
- 360 41.38888889
371
- 361 41.55124654
372
- 362 41.71270718
373
- 363 41.59779614
374
- 364 41.48351648
375
- 365 41.36986301
376
- 366 41.53005464
377
- 367 41.41689373
378
- 368 41.30434783
379
- 369 41.19241192
380
- 370 41.35135135
381
- 371 41.23989218
382
- 372 41.12903226
383
- 373 41.01876676
384
- 374 40.90909091
385
- 375 41.06666667
386
- 376 40.95744681
387
- 377 41.11405836
388
- 378 41.26984127
389
- 379 41.16094987
390
- 380 41.05263158
391
- 381 41.20734908
392
- 382 41.36125654
393
- 383 41.51436031
394
- 384 41.40625000
395
- 385 41.55844156
396
- 386 41.45077720
397
- 387 41.34366925
398
- 388 41.23711340
399
- 389 41.38817481
400
- 390 41.53846154
401
- 391 41.68797954
402
- 392 41.83673469
403
- 393 41.73027990
404
- 394 41.62436548
405
- 395 41.77215190
406
- 396 41.91919192
407
- 397 42.06549118
408
- 398 41.95979899
409
- 399 42.10526316
410
- 400 42.00000000
411
- 401 42.14463840
412
- 402 42.03980100
413
- 403 41.93548387
414
- 404 42.07920792
415
- 405 41.97530864
416
- 406 41.87192118
417
- 407 41.76904177
418
- 408 41.66666667
419
- 409 41.80929095
420
- 410 41.70731707
421
- 411 41.60583942
422
- 412 41.50485437
423
- 413 41.40435835
424
- 414 41.30434783
425
- 415 41.20481928
426
- 416 41.34615385
427
- 417 41.48681055
428
- 418 41.38755981
429
- 419 41.52744630
430
- 420 41.42857143
431
- 421 41.56769596
432
- 422 41.46919431
433
- 423 41.37115839
434
- 424 41.50943396
435
- 425 41.41176471
436
- 426 41.31455399
437
- 427 41.21779859
438
- 428 41.12149533
439
- 429 41.25874126
440
- 430 41.16279070
441
- 431 41.29930394
442
- 432 41.43518519
443
- 433 41.33949192
444
- 434 41.47465438
445
- 435 41.37931034
446
- 436 41.28440367
447
- 437 41.18993135
448
- 438 41.32420091
449
- 439 41.45785877
450
- 440 41.36363636
451
- 441 41.49659864
452
- 442 41.40271493
453
- 443 41.30925508
454
- 444 41.21621622
455
- 445 41.12359551
456
- 446 41.25560538
457
- 447 41.16331096
458
- 448 41.07142857
459
- 449 40.97995546
460
- 450 41.11111111
461
- 451 41.01995565
462
- 452 40.92920354
463
- 453 41.05960265
464
- 454 40.96916300
465
- 455 40.87912088
466
- 456 41.00877193
467
- 457 41.13785558
468
- 458 41.04803493
469
- 459 40.95860566
470
- 460 41.08695652
471
- 461 40.99783080
472
- 462 40.90909091
473
- 463 41.03671706
474
- 464 41.16379310
475
- 465 41.07526882
476
- 466 40.98712446
477
- 467 41.11349036
478
- 468 41.02564103
479
- 469 41.15138593
480
- 470 41.27659574
481
- 471 41.18895966
482
- 472 41.10169492
483
- 473 41.22621564
484
- 474 41.13924051
485
- 475 41.05263158
486
- 476 41.17647059
487
- 477 41.09014675
488
- 478 41.21338912
489
- 479 41.33611691
490
- 480 41.45833333
491
- 481 41.37214137
492
- 482 41.28630705
493
- 483 41.20082816
494
- 484 41.11570248
495
- 485 41.03092784
496
- 486 40.94650206
497
- 487 41.06776181
498
- 488 40.98360656
499
- 489 40.89979550
500
- 490 41.02040816
501
- 491 40.93686354
502
- 492 40.85365854
503
- 493 40.77079108
504
- 494 40.89068826
505
- 495 41.01010101
506
- 496 41.12903226
507
- 497 41.04627767
508
- 498 40.96385542
509
- 499 40.88176353
510
- 500 41.00000000
511
- 501 40.91816367
512
- 502 40.83665339
513
- 503 40.75546720
514
- 504 40.67460317
515
- 505 40.59405941
516
- 506 40.71146245
517
- 507 40.63116371
518
- 508 40.74803150
519
- 509 40.66797642
520
- 510 40.78431373
521
- 511 40.70450098
522
- 512 40.62500000
523
- 513 40.54580897
524
- 514 40.46692607
525
- 515 40.58252427
526
- 516 40.69767442
527
- 517 40.81237911
528
- 518 40.92664093
529
- 519 40.84778420
530
- 520 40.96153846
531
- 521 40.88291747
532
- 522 40.99616858
533
- 523 40.91778203
534
- 524 40.83969466
535
- 525 40.76190476
536
- 526 40.68441065
537
- 527 40.60721063
538
- 528 40.53030303
539
- 529 40.64272212
540
- 530 40.75471698
541
- 531 40.67796610
542
- 532 40.60150376
543
- 533 40.52532833
544
- 534 40.63670412
545
- 535 40.74766355
546
- 536 40.67164179
547
- 537 40.78212291
548
- 538 40.89219331
549
- 539 40.81632653
550
- 540 40.74074074
551
- 541 40.66543438
552
- 542 40.77490775
553
- 543 40.69981584
554
- 544 40.62500000
555
- 545 40.55045872
556
- 546 40.47619048
557
- 547 40.40219378
558
- 548 40.51094891
559
- 549 40.43715847
560
- 550 40.36363636
561
- 551 40.47186933
562
- 552 40.39855072
563
- 553 40.32549729
564
- 554 40.25270758
565
- 555 40.18018018
566
- 556 40.28776978
567
- 557 40.39497307
568
- 558 40.50179211
569
- 559 40.42933810
570
- 560 40.35714286
571
- 561 40.28520499
572
- 562 40.39145907
573
- 563 40.49733570
574
- 564 40.42553191
575
- 565 40.53097345
576
- 566 40.45936396
577
- 567 40.38800705
578
- 568 40.31690141
579
- 569 40.24604569
580
- 570 40.17543860
581
- 571 40.10507881
582
- 572 40.20979021
583
- 573 40.13961606
584
- 574 40.06968641
585
- 575 40.00000000
586
- 576 40.10416667
587
- 577 40.20797227
588
- 578 40.13840830
589
- 579 40.06908463
590
- 580 40.17241379
591
- 581 40.10327022
592
- 582 40.03436426
593
- 583 40.13722127
594
- 584 40.23972603
595
- 585 40.34188034
596
- 586 40.44368601
597
- 587 40.37478705
598
- 588 40.30612245
599
- 589 40.23769100
600
- 590 40.16949153
601
- 591 40.10152284
602
- 592 40.03378378
603
- 593 40.13490725
604
- 594 40.23569024
605
- 595 40.16806723
606
- 596 40.26845638
607
- 597 40.20100503
608
- 598 40.13377926
609
- 599 40.23372287
610
- 600 40.16666667
611
- 601 40.09983361
612
- 602 40.03322259
613
- 603 40.13266998
614
- 604 40.06622517
615
- 605 40.00000000
616
- 606 39.93399340
617
- 607 40.03294893
618
- 608 39.96710526
619
- 609 39.90147783
620
- 610 39.83606557
621
- 611 39.93453355
622
- 612 39.86928105
623
- 613 39.96737357
624
- 614 39.90228013
625
- 615 39.83739837
626
- 616 39.93506494
627
- 617 40.03241491
628
- 618 39.96763754
629
- 619 39.90306947
630
- 620 39.83870968
631
- 621 39.77455717
632
- 622 39.71061093
633
- 623 39.64686998
634
- 624 39.58333333
635
- 625 39.52000000
636
- 626 39.61661342
637
- 627 39.55342903
638
- 628 39.49044586
639
- 629 39.58664547
640
- 630 39.52380952
641
- 631 39.61965135
642
- 632 39.71518987
643
- 633 39.65244866
644
- 634 39.58990536
645
- 635 39.52755906
646
- 636 39.46540881
647
- 637 39.40345369
648
- 638 39.34169279
649
- 639 39.43661972
650
- 640 39.53125000
651
- 641 39.46957878
652
- 642 39.40809969
653
- 643 39.34681182
654
- 644 39.44099379
655
- 645 39.37984496
656
- 646 39.31888545
657
- 647 39.25811437
658
- 648 39.19753086
659
- 649 39.13713405
660
- 650 39.07692308
661
- 651 39.01689708
662
- 652 38.95705521
663
- 653 38.89739663
664
- 654 38.99082569
665
- 655 38.93129771
666
- 656 38.87195122
667
- 657 38.81278539
668
- 658 38.90577508
669
- 659 38.84673748
670
- 660 38.78787879
671
- 661 38.72919818
672
- 662 38.67069486
673
- 663 38.61236802
674
- 664 38.55421687
675
- 665 38.49624060
676
- 666 38.43843844
677
- 667 38.53073463
678
- 668 38.62275449
679
- 669 38.71449925
680
- 670 38.80597015
681
- 671 38.89716841
682
- 672 38.83928571
683
- 673 38.78157504
684
- 674 38.87240356
685
- 675 38.81481481
686
- 676 38.75739645
687
- 677 38.70014771
688
- 678 38.79056047
689
- 679 38.73343152
690
- 680 38.82352941
691
- 681 38.91336270
692
- 682 38.85630499
693
- 683 38.79941435
694
- 684 38.88888889
695
- 685 38.83211679
696
- 686 38.92128280
697
- 687 38.86462882
698
- 688 38.80813953
699
- 689 38.75181422
700
- 690 38.69565217
701
- 691 38.78437048
702
- 692 38.72832370
703
- 693 38.67243867
704
- 694 38.61671470
705
- 695 38.56115108
706
- 696 38.50574713
707
- 697 38.45050215
708
- 698 38.53868195
709
- 699 38.62660944
710
- 700 38.71428571
711
- 701 38.80171184
712
- 702 38.88888889
713
- 703 38.97581792
714
- 704 39.06250000
715
- 705 39.00709220
716
- 706 38.95184136
717
- 707 38.89674682
718
- 708 38.84180791
719
- 709 38.78702398
720
- 710 38.73239437
721
- 711 38.81856540
722
- 712 38.90449438
723
- 713 38.99018233
724
- 714 39.07563025
725
- 715 39.02097902
726
- 716 38.96648045
727
- 717 38.91213389
728
- 718 38.85793872
729
- 719 38.94297636
730
- 720 38.88888889
731
- 721 38.83495146
732
- 722 38.78116343
733
- 723 38.86583679
734
- 724 38.95027624
735
- 725 39.03448276
736
- 726 38.98071625
737
- 727 38.92709766
738
- 728 39.01098901
739
- 729 38.95747599
740
- 730 38.90410959
741
- 731 38.85088919
742
- 732 38.79781421
743
- 733 38.88130969
744
- 734 38.82833787
745
- 735 38.77551020
746
- 736 38.72282609
747
- 737 38.80597015
748
- 738 38.88888889
749
- 739 38.97158322
750
- 740 38.91891892
751
- 741 38.86639676
752
- 742 38.94878706
753
- 743 39.03095559
754
- 744 39.11290323
755
- 745 39.06040268
756
- 746 39.00804290
757
- 747 39.08969210
758
- 748 39.03743316
759
- 749 38.98531375
760
- 750 39.06666667
761
 
762
- Final result: 39.0667 +/- 1.7827
763
- Random chance: 25.0000 +/- 1.5822
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q3_K_L.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 36.0000 +/- 1.7539
6
+ Random chance: 25.0000 +/- 1.5822
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 330.46 ms
10
+ llama_perf_context_print: prompt eval time = 271968.64 ms / 67808 tokens ( 4.01 ms per token, 249.32 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 274735.21 ms / 67809 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-q3_k_l.ppx CHANGED
@@ -1,37 +1,37 @@
1
  ====== Perplexity statistics ======
2
- Mean PPL(Q) : 8.895382 ± 0.058817
3
- Mean PPL(base) : 7.534124 ± 0.048206
4
- Cor(ln(PPL(Q)), ln(PPL(base))): 97.35%
5
- Mean ln(PPL(Q)/PPL(base)) : 0.166090 ± 0.001513
6
- Mean PPL(Q)/PPL(base) : 1.180679 ± 0.001787
7
- Mean PPL(Q)-PPL(base) : 1.361258 ± 0.016217
8
 
9
  ====== KL divergence statistics ======
10
- Mean KLD: 0.146296 ± 0.000446
11
- Maximum KLD: 8.539277
12
- 99.9% KLD: 1.800523
13
- 99.0% KLD: 0.699970
14
- 99.0% KLD: 0.699970
15
- Median KLD: 0.118646
16
- 10.0% KLD: 0.006209
17
- 5.0% KLD: 0.001891
18
- 1.0% KLD: 0.000254
19
- Minimum KLD: -0.000001
20
 
21
  ====== Token probability statistics ======
22
- Mean Δp: -2.899 ± 0.028 %
23
- Maximum Δp: 77.074%
24
- 99.9% Δp: 39.937%
25
- 99.0% Δp: 24.573%
26
- 95.0% Δp: 11.867%
27
- 90.0% Δp: 6.258%
28
- 75.0% Δp: 0.461%
29
- Median Δp: -0.451%
30
- 25.0% Δp: -5.618%
31
- 10.0% Δp: -15.734%
32
- 5.0% Δp: -23.078%
33
- 1.0% Δp: -39.433%
34
- 0.1% Δp: -67.421%
35
- Minimum Δp: -97.164%
36
- RMS Δp : 11.179 ± 0.043 %
37
- Same top p: 78.943 ± 0.108 %
 
1
  ====== Perplexity statistics ======
2
+ Mean PPL(Q) : 9.923766 ± 0.061734
3
+ Mean PPL(base) : 7.237090 ± 0.045539
4
+ Cor(ln(PPL(Q)), ln(PPL(base))): 94.33%
5
+ Mean ln(PPL(Q)/PPL(base)) : 0.315713 ± 0.002108
6
+ Mean PPL(Q)/PPL(base) : 1.371237 ± 0.002890
7
+ Mean PPL(Q)-PPL(base) : 2.686677 ± 0.024105
8
 
9
  ====== KL divergence statistics ======
10
+ Mean KLD: 0.292497 ± 0.000978
11
+ Maximum KLD: 11.110364
12
+ 99.9% KLD: 3.423501
13
+ 99.0% KLD: 2.016132
14
+ 99.0% KLD: 2.016132
15
+ Median KLD: 0.210275
16
+ 10.0% KLD: 0.019525
17
+ 5.0% KLD: 0.005606
18
+ 1.0% KLD: 0.000617
19
+ Minimum KLD: 0.000004
20
 
21
  ====== Token probability statistics ======
22
+ Mean Δp: -8.373 ± 0.044 %
23
+ Maximum Δp: 78.738%
24
+ 99.9% Δp: 39.788%
25
+ 99.0% Δp: 20.797%
26
+ 95.0% Δp: 7.364%
27
+ 90.0% Δp: 2.736%
28
+ 75.0% Δp: 0.005%
29
+ Median Δp: -2.257%
30
+ 25.0% Δp: -12.828%
31
+ 10.0% Δp: -28.442%
32
+ 5.0% Δp: -41.830%
33
+ 1.0% Δp: -75.747%
34
+ 0.1% Δp: -90.320%
35
+ Minimum Δp: -99.555%
36
+ RMS Δp : 18.605 ± 0.069 %
37
+ Same top p: 74.871 ± 0.114 %
scores/Watt-Tool-8B-q3_k_l.tqa CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 817 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 817 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 50.00000000
13
- 3 33.33333333
14
- 4 25.00000000
15
- 5 20.00000000
16
- 6 16.66666667
17
- 7 14.28571429
18
- 8 25.00000000
19
- 9 22.22222222
20
- 10 20.00000000
21
- 11 18.18181818
22
- 12 25.00000000
23
- 13 30.76923077
24
- 14 28.57142857
25
- 15 26.66666667
26
- 16 31.25000000
27
- 17 29.41176471
28
- 18 27.77777778
29
- 19 26.31578947
30
- 20 25.00000000
31
- 21 23.80952381
32
- 22 22.72727273
33
- 23 21.73913043
34
- 24 25.00000000
35
- 25 24.00000000
36
- 26 23.07692308
37
- 27 22.22222222
38
- 28 21.42857143
39
- 29 24.13793103
40
- 30 23.33333333
41
- 31 22.58064516
42
- 32 25.00000000
43
- 33 27.27272727
44
- 34 26.47058824
45
- 35 25.71428571
46
- 36 25.00000000
47
- 37 27.02702703
48
- 38 26.31578947
49
- 39 28.20512821
50
- 40 27.50000000
51
- 41 26.82926829
52
- 42 26.19047619
53
- 43 27.90697674
54
- 44 27.27272727
55
- 45 28.88888889
56
- 46 28.26086957
57
- 47 29.78723404
58
- 48 31.25000000
59
- 49 30.61224490
60
- 50 32.00000000
61
- 51 31.37254902
62
- 52 30.76923077
63
- 53 30.18867925
64
- 54 29.62962963
65
- 55 30.90909091
66
- 56 30.35714286
67
- 57 29.82456140
68
- 58 29.31034483
69
- 59 30.50847458
70
- 60 31.66666667
71
- 61 31.14754098
72
- 62 32.25806452
73
- 63 31.74603175
74
- 64 31.25000000
75
- 65 30.76923077
76
- 66 31.81818182
77
- 67 31.34328358
78
- 68 30.88235294
79
- 69 31.88405797
80
- 70 31.42857143
81
- 71 30.98591549
82
- 72 31.94444444
83
- 73 31.50684932
84
- 74 31.08108108
85
- 75 30.66666667
86
- 76 31.57894737
87
- 77 31.16883117
88
- 78 30.76923077
89
- 79 30.37974684
90
- 80 31.25000000
91
- 81 30.86419753
92
- 82 31.70731707
93
- 83 31.32530120
94
- 84 30.95238095
95
- 85 30.58823529
96
- 86 30.23255814
97
- 87 29.88505747
98
- 88 29.54545455
99
- 89 30.33707865
100
- 90 30.00000000
101
- 91 29.67032967
102
- 92 29.34782609
103
- 93 29.03225806
104
- 94 28.72340426
105
- 95 29.47368421
106
- 96 30.20833333
107
- 97 30.92783505
108
- 98 30.61224490
109
- 99 31.31313131
110
- 100 32.00000000
111
- 101 32.67326733
112
- 102 32.35294118
113
- 103 32.03883495
114
- 104 31.73076923
115
- 105 31.42857143
116
- 106 31.13207547
117
- 107 31.77570093
118
- 108 32.40740741
119
- 109 32.11009174
120
- 110 32.72727273
121
- 111 32.43243243
122
- 112 33.03571429
123
- 113 33.62831858
124
- 114 34.21052632
125
- 115 33.91304348
126
- 116 33.62068966
127
- 117 33.33333333
128
- 118 33.05084746
129
- 119 32.77310924
130
- 120 33.33333333
131
- 121 33.88429752
132
- 122 33.60655738
133
- 123 33.33333333
134
- 124 33.87096774
135
- 125 34.40000000
136
- 126 34.92063492
137
- 127 34.64566929
138
- 128 34.37500000
139
- 129 34.88372093
140
- 130 35.38461538
141
- 131 35.11450382
142
- 132 34.84848485
143
- 133 34.58646617
144
- 134 34.32835821
145
- 135 34.81481481
146
- 136 34.55882353
147
- 137 35.03649635
148
- 138 35.50724638
149
- 139 35.97122302
150
- 140 35.71428571
151
- 141 35.46099291
152
- 142 35.21126761
153
- 143 34.96503497
154
- 144 35.41666667
155
- 145 35.17241379
156
- 146 34.93150685
157
- 147 35.37414966
158
- 148 35.13513514
159
- 149 34.89932886
160
- 150 34.66666667
161
- 151 35.09933775
162
- 152 34.86842105
163
- 153 34.64052288
164
- 154 34.41558442
165
- 155 34.83870968
166
- 156 35.25641026
167
- 157 35.03184713
168
- 158 35.44303797
169
- 159 35.22012579
170
- 160 35.00000000
171
- 161 34.78260870
172
- 162 35.18518519
173
- 163 34.96932515
174
- 164 34.75609756
175
- 165 35.15151515
176
- 166 35.54216867
177
- 167 35.32934132
178
- 168 35.71428571
179
- 169 35.50295858
180
- 170 35.29411765
181
- 171 35.08771930
182
- 172 34.88372093
183
- 173 34.68208092
184
- 174 35.05747126
185
- 175 34.85714286
186
- 176 35.22727273
187
- 177 35.02824859
188
- 178 34.83146067
189
- 179 35.19553073
190
- 180 35.55555556
191
- 181 35.35911602
192
- 182 35.16483516
193
- 183 34.97267760
194
- 184 34.78260870
195
- 185 34.59459459
196
- 186 34.94623656
197
- 187 34.75935829
198
- 188 34.57446809
199
- 189 34.92063492
200
- 190 34.73684211
201
- 191 34.55497382
202
- 192 34.37500000
203
- 193 34.19689119
204
- 194 34.53608247
205
- 195 34.35897436
206
- 196 34.18367347
207
- 197 34.51776650
208
- 198 34.84848485
209
- 199 34.67336683
210
- 200 35.00000000
211
- 201 34.82587065
212
- 202 35.14851485
213
- 203 34.97536946
214
- 204 34.80392157
215
- 205 34.63414634
216
- 206 34.46601942
217
- 207 34.78260870
218
- 208 34.61538462
219
- 209 34.44976077
220
- 210 34.28571429
221
- 211 34.12322275
222
- 212 33.96226415
223
- 213 33.80281690
224
- 214 33.64485981
225
- 215 33.48837209
226
- 216 33.79629630
227
- 217 34.10138249
228
- 218 34.40366972
229
- 219 34.24657534
230
- 220 34.09090909
231
- 221 33.93665158
232
- 222 33.78378378
233
- 223 33.63228700
234
- 224 33.92857143
235
- 225 33.77777778
236
- 226 34.07079646
237
- 227 34.36123348
238
- 228 34.21052632
239
- 229 34.06113537
240
- 230 33.91304348
241
- 231 33.76623377
242
- 232 33.62068966
243
- 233 33.47639485
244
- 234 33.76068376
245
- 235 33.61702128
246
- 236 33.89830508
247
- 237 33.75527426
248
- 238 34.03361345
249
- 239 33.89121339
250
- 240 33.75000000
251
- 241 33.60995851
252
- 242 33.47107438
253
- 243 33.74485597
254
- 244 33.60655738
255
- 245 33.46938776
256
- 246 33.33333333
257
- 247 33.60323887
258
- 248 33.46774194
259
- 249 33.73493976
260
- 250 33.60000000
261
- 251 33.86454183
262
- 252 34.12698413
263
- 253 33.99209486
264
- 254 33.85826772
265
- 255 33.72549020
266
- 256 33.59375000
267
- 257 33.46303502
268
- 258 33.33333333
269
- 259 33.20463320
270
- 260 33.07692308
271
- 261 33.33333333
272
- 262 33.20610687
273
- 263 33.07984791
274
- 264 33.33333333
275
- 265 33.58490566
276
- 266 33.45864662
277
- 267 33.33333333
278
- 268 33.20895522
279
- 269 33.08550186
280
- 270 32.96296296
281
- 271 32.84132841
282
- 272 32.72058824
283
- 273 32.96703297
284
- 274 32.84671533
285
- 275 32.72727273
286
- 276 32.60869565
287
- 277 32.49097473
288
- 278 32.37410072
289
- 279 32.25806452
290
- 280 32.50000000
291
- 281 32.74021352
292
- 282 32.62411348
293
- 283 32.86219081
294
- 284 32.74647887
295
- 285 32.63157895
296
- 286 32.86713287
297
- 287 33.10104530
298
- 288 32.98611111
299
- 289 32.87197232
300
- 290 32.75862069
301
- 291 32.98969072
302
- 292 32.87671233
303
- 293 33.10580205
304
- 294 32.99319728
305
- 295 33.22033898
306
- 296 33.44594595
307
- 297 33.67003367
308
- 298 33.55704698
309
- 299 33.44481605
310
- 300 33.33333333
311
- 301 33.22259136
312
- 302 33.11258278
313
- 303 33.00330033
314
- 304 32.89473684
315
- 305 32.78688525
316
- 306 32.67973856
317
- 307 32.57328990
318
- 308 32.79220779
319
- 309 32.68608414
320
- 310 32.58064516
321
- 311 32.47588424
322
- 312 32.69230769
323
- 313 32.58785942
324
- 314 32.48407643
325
- 315 32.38095238
326
- 316 32.27848101
327
- 317 32.17665615
328
- 318 32.07547170
329
- 319 31.97492163
330
- 320 31.87500000
331
- 321 31.77570093
332
- 322 31.98757764
333
- 323 31.88854489
334
- 324 31.79012346
335
- 325 31.69230769
336
- 326 31.59509202
337
- 327 31.49847095
338
- 328 31.70731707
339
- 329 31.61094225
340
- 330 31.81818182
341
- 331 31.72205438
342
- 332 31.92771084
343
- 333 31.83183183
344
- 334 31.73652695
345
- 335 31.94029851
346
- 336 32.14285714
347
- 337 32.04747774
348
- 338 31.95266272
349
- 339 32.15339233
350
- 340 32.05882353
351
- 341 32.25806452
352
- 342 32.45614035
353
- 343 32.36151603
354
- 344 32.55813953
355
- 345 32.75362319
356
- 346 32.65895954
357
- 347 32.85302594
358
- 348 32.75862069
359
- 349 32.66475645
360
- 350 32.57142857
361
- 351 32.47863248
362
- 352 32.38636364
363
- 353 32.29461756
364
- 354 32.20338983
365
- 355 32.39436620
366
- 356 32.30337079
367
- 357 32.21288515
368
- 358 32.12290503
369
- 359 32.03342618
370
- 360 31.94444444
371
- 361 32.13296399
372
- 362 32.04419890
373
- 363 32.23140496
374
- 364 32.14285714
375
- 365 32.05479452
376
- 366 31.96721311
377
- 367 32.15258856
378
- 368 32.06521739
379
- 369 31.97831978
380
- 370 31.89189189
381
- 371 31.80592992
382
- 372 31.98924731
383
- 373 31.90348525
384
- 374 32.08556150
385
- 375 32.00000000
386
- 376 31.91489362
387
- 377 32.09549072
388
- 378 32.01058201
389
- 379 31.92612137
390
- 380 31.84210526
391
- 381 31.75853018
392
- 382 31.67539267
393
- 383 31.59268930
394
- 384 31.51041667
395
- 385 31.42857143
396
- 386 31.34715026
397
- 387 31.26614987
398
- 388 31.44329897
399
- 389 31.36246787
400
- 390 31.28205128
401
- 391 31.45780051
402
- 392 31.37755102
403
- 393 31.55216285
404
- 394 31.72588832
405
- 395 31.64556962
406
- 396 31.81818182
407
- 397 31.98992443
408
- 398 31.90954774
409
- 399 31.82957393
410
- 400 31.75000000
411
- 401 31.67082294
412
- 402 31.84079602
413
- 403 31.76178660
414
- 404 31.68316832
415
- 405 31.60493827
416
- 406 31.52709360
417
- 407 31.44963145
418
- 408 31.37254902
419
- 409 31.29584352
420
- 410 31.21951220
421
- 411 31.14355231
422
- 412 31.31067961
423
- 413 31.47699758
424
- 414 31.64251208
425
- 415 31.80722892
426
- 416 31.73076923
427
- 417 31.65467626
428
- 418 31.57894737
429
- 419 31.50357995
430
- 420 31.66666667
431
- 421 31.82897862
432
- 422 31.75355450
433
- 423 31.91489362
434
- 424 31.83962264
435
- 425 31.76470588
436
- 426 31.69014085
437
- 427 31.61592506
438
- 428 31.54205607
439
- 429 31.46853147
440
- 430 31.39534884
441
- 431 31.32250580
442
- 432 31.48148148
443
- 433 31.63972286
444
- 434 31.79723502
445
- 435 31.72413793
446
- 436 31.65137615
447
- 437 31.57894737
448
- 438 31.73515982
449
- 439 31.89066059
450
- 440 32.04545455
451
- 441 31.97278912
452
- 442 31.90045249
453
- 443 31.82844244
454
- 444 31.98198198
455
- 445 31.91011236
456
- 446 31.83856502
457
- 447 31.76733781
458
- 448 31.69642857
459
- 449 31.84855234
460
- 450 31.77777778
461
- 451 31.70731707
462
- 452 31.63716814
463
- 453 31.78807947
464
- 454 31.71806167
465
- 455 31.64835165
466
- 456 31.57894737
467
- 457 31.50984683
468
- 458 31.44104803
469
- 459 31.37254902
470
- 460 31.30434783
471
- 461 31.23644252
472
- 462 31.38528139
473
- 463 31.53347732
474
- 464 31.68103448
475
- 465 31.82795699
476
- 466 31.75965665
477
- 467 31.69164882
478
- 468 31.83760684
479
- 469 31.76972281
480
- 470 31.70212766
481
- 471 31.84713376
482
- 472 31.99152542
483
- 473 31.92389006
484
- 474 31.85654008
485
- 475 32.00000000
486
- 476 31.93277311
487
- 477 31.86582809
488
- 478 31.79916318
489
- 479 31.73277662
490
- 480 31.66666667
491
- 481 31.60083160
492
- 482 31.74273859
493
- 483 31.67701863
494
- 484 31.81818182
495
- 485 31.95876289
496
- 486 31.89300412
497
- 487 31.82751540
498
- 488 31.76229508
499
- 489 31.90184049
500
- 490 32.04081633
501
- 491 32.17922607
502
- 492 32.31707317
503
- 493 32.25152130
504
- 494 32.38866397
505
- 495 32.52525253
506
- 496 32.45967742
507
- 497 32.39436620
508
- 498 32.32931727
509
- 499 32.26452906
510
- 500 32.20000000
511
- 501 32.13572854
512
- 502 32.07171315
513
- 503 32.00795229
514
- 504 32.14285714
515
- 505 32.07920792
516
- 506 32.01581028
517
- 507 32.14990138
518
- 508 32.28346457
519
- 509 32.41650295
520
- 510 32.54901961
521
- 511 32.48532290
522
- 512 32.42187500
523
- 513 32.35867446
524
- 514 32.29571984
525
- 515 32.42718447
526
- 516 32.36434109
527
- 517 32.49516441
528
- 518 32.43243243
529
- 519 32.36994220
530
- 520 32.30769231
531
- 521 32.24568138
532
- 522 32.18390805
533
- 523 32.12237094
534
- 524 32.06106870
535
- 525 32.00000000
536
- 526 31.93916350
537
- 527 32.06831120
538
- 528 32.00757576
539
- 529 32.13610586
540
- 530 32.07547170
541
- 531 32.01506591
542
- 532 31.95488722
543
- 533 32.08255159
544
- 534 32.02247191
545
- 535 31.96261682
546
- 536 31.90298507
547
- 537 32.02979516
548
- 538 31.97026022
549
- 539 31.91094620
550
- 540 32.03703704
551
- 541 31.97781885
552
- 542 31.91881919
553
- 543 31.86003683
554
- 544 31.80147059
555
- 545 31.74311927
556
- 546 31.68498168
557
- 547 31.80987203
558
- 548 31.93430657
559
- 549 31.87613843
560
- 550 31.81818182
561
- 551 31.76043557
562
- 552 31.88405797
563
- 553 32.00723327
564
- 554 31.94945848
565
- 555 31.89189189
566
- 556 31.83453237
567
- 557 31.77737882
568
- 558 31.72043011
569
- 559 31.66368515
570
- 560 31.78571429
571
- 561 31.72905526
572
- 562 31.67259786
573
- 563 31.61634103
574
- 564 31.73758865
575
- 565 31.85840708
576
- 566 31.97879859
577
- 567 31.92239859
578
- 568 32.04225352
579
- 569 32.16168717
580
- 570 32.10526316
581
- 571 32.22416813
582
- 572 32.16783217
583
- 573 32.11169284
584
- 574 32.22996516
585
- 575 32.34782609
586
- 576 32.46527778
587
- 577 32.58232236
588
- 578 32.69896194
589
- 579 32.81519862
590
- 580 32.93103448
591
- 581 32.87435456
592
- 582 32.98969072
593
- 583 32.93310463
594
- 584 33.04794521
595
- 585 32.99145299
596
- 586 33.10580205
597
- 587 33.04940375
598
- 588 32.99319728
599
- 589 32.93718166
600
- 590 33.05084746
601
- 591 32.99492386
602
- 592 32.93918919
603
- 593 32.88364250
604
- 594 32.99663300
605
- 595 32.94117647
606
- 596 32.88590604
607
- 597 32.83082077
608
- 598 32.77591973
609
- 599 32.72120200
610
- 600 32.66666667
611
- 601 32.77870216
612
- 602 32.89036545
613
- 603 33.00165837
614
- 604 32.94701987
615
- 605 32.89256198
616
- 606 33.00330033
617
- 607 32.94892916
618
- 608 32.89473684
619
- 609 32.84072250
620
- 610 32.78688525
621
- 611 32.73322422
622
- 612 32.84313725
623
- 613 32.78955954
624
- 614 32.73615635
625
- 615 32.68292683
626
- 616 32.79220779
627
- 617 32.90113452
628
- 618 32.84789644
629
- 619 32.95638126
630
- 620 32.90322581
631
- 621 32.85024155
632
- 622 32.95819936
633
- 623 32.90529695
634
- 624 32.85256410
635
- 625 32.80000000
636
- 626 32.74760383
637
- 627 32.69537480
638
- 628 32.80254777
639
- 629 32.75039746
640
- 630 32.69841270
641
- 631 32.80507132
642
- 632 32.75316456
643
- 633 32.70142180
644
- 634 32.80757098
645
- 635 32.75590551
646
- 636 32.70440252
647
- 637 32.65306122
648
- 638 32.60188088
649
- 639 32.55086072
650
- 640 32.65625000
651
- 641 32.60530421
652
- 642 32.71028037
653
- 643 32.65940902
654
- 644 32.60869565
655
- 645 32.71317829
656
- 646 32.66253870
657
- 647 32.61205564
658
- 648 32.71604938
659
- 649 32.66563945
660
- 650 32.76923077
661
- 651 32.87250384
662
- 652 32.97546012
663
- 653 33.07810107
664
- 654 33.02752294
665
- 655 32.97709924
666
- 656 33.07926829
667
- 657 33.18112633
668
- 658 33.13069909
669
- 659 33.23216995
670
- 660 33.18181818
671
- 661 33.13161876
672
- 662 33.08157100
673
- 663 33.03167421
674
- 664 32.98192771
675
- 665 32.93233083
676
- 666 33.03303303
677
- 667 32.98350825
678
- 668 32.93413174
679
- 669 32.88490284
680
- 670 32.98507463
681
- 671 32.93591654
682
- 672 32.88690476
683
- 673 32.83803863
684
- 674 32.78931751
685
- 675 32.88888889
686
- 676 32.98816568
687
- 677 32.93943870
688
- 678 32.89085546
689
- 679 32.84241532
690
- 680 32.79411765
691
- 681 32.74596182
692
- 682 32.84457478
693
- 683 32.79648609
694
- 684 32.89473684
695
- 685 32.84671533
696
- 686 32.79883382
697
- 687 32.75109170
698
- 688 32.70348837
699
- 689 32.80116110
700
- 690 32.75362319
701
- 691 32.85094067
702
- 692 32.94797688
703
- 693 33.04473304
704
- 694 32.99711816
705
- 695 32.94964029
706
- 696 33.04597701
707
- 697 32.99856528
708
- 698 32.95128940
709
- 699 32.90414878
710
- 700 33.00000000
711
- 701 32.95292439
712
- 702 32.90598291
713
- 703 32.85917496
714
- 704 32.81250000
715
- 705 32.76595745
716
- 706 32.86118980
717
- 707 32.95615276
718
- 708 32.90960452
719
- 709 32.86318759
720
- 710 32.95774648
721
- 711 32.91139241
722
- 712 33.00561798
723
- 713 32.95932679
724
- 714 32.91316527
725
- 715 33.00699301
726
- 716 32.96089385
727
- 717 33.05439331
728
- 718 33.14763231
729
- 719 33.24061196
730
- 720 33.19444444
731
- 721 33.14840499
732
- 722 33.10249307
733
- 723 33.05670816
734
- 724 33.14917127
735
- 725 33.10344828
736
- 726 33.19559229
737
- 727 33.14993122
738
- 728 33.10439560
739
- 729 33.19615912
740
- 730 33.15068493
741
- 731 33.10533516
742
- 732 33.19672131
743
- 733 33.15143247
744
- 734 33.10626703
745
- 735 33.06122449
746
- 736 33.01630435
747
- 737 33.10719132
748
- 738 33.06233062
749
- 739 33.01759134
750
- 740 32.97297297
751
- 741 33.06342780
752
- 742 33.01886792
753
- 743 32.97442799
754
- 744 33.06451613
755
- 745 33.02013423
756
- 746 33.10991957
757
- 747 33.19946452
758
- 748 33.15508021
759
- 749 33.24432577
760
- 750 33.33333333
761
 
762
- Final result: 33.3333 +/- 1.7225
763
- Random chance: 19.8992 +/- 1.4588
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q3_K_L.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 35.4037 +/- 2.6692
6
+ Random chance: 28.5968 +/- 2.5221
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 307.41 ms
10
+ llama_perf_context_print: prompt eval time = 81969.79 ms / 17455 tokens ( 4.70 ms per token, 212.94 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 83215.27 ms / 17456 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-q3_k_l.wng CHANGED
@@ -1,760 +1,11 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- winogrande_score : loaded 1267 tasks from prompt.
6
- winogrande_score : selecting 750 random tasks
7
- winogrande_score : tokenizing selected tasks
8
- winogrande_score : calculating winogrande score over selected tasks.
9
- 1 100.0000 -2.907583 -3.803999 1 1
10
- 2 100.0000 -1.692355 -1.924335 1 1
11
- 3 100.0000 -2.869413 -2.718204 2 2
12
- 4 75.0000 -4.830220 -4.749911 2 1
13
- 5 60.0000 -2.752985 -3.079111 1 2
14
- 6 50.0000 -5.032448 -4.867825 2 1
15
- 7 57.1429 -4.140249 -2.837141 2 2
16
- 8 62.5000 -3.203068 -2.523867 2 2
17
- 9 66.6667 -4.139743 -5.582026 1 1
18
- 10 70.0000 -1.795674 -1.660755 2 2
19
- 11 72.7273 -8.994802 -4.708105 2 2
20
- 12 75.0000 -2.646854 -1.741683 2 2
21
- 13 76.9231 -2.930563 -4.570487 1 1
22
- 14 78.5714 -3.193429 -4.008543 1 1
23
- 15 73.3333 -4.200741 -6.247772 1 2
24
- 16 75.0000 -4.248100 -3.835441 2 2
25
- 17 70.5882 -7.257449 -7.472361 1 2
26
- 18 72.2222 -1.542092 -3.297827 1 1
27
- 19 73.6842 -3.504145 -4.972578 1 1
28
- 20 75.0000 -2.757241 -2.890451 1 1
29
- 21 76.1905 -2.132241 -3.424561 1 1
30
- 22 72.7273 -2.531060 -2.563687 1 2
31
- 23 73.9130 -4.724509 -5.097672 1 1
32
- 24 70.8333 -2.651444 -2.456048 2 1
33
- 25 72.0000 -1.884954 -1.847693 2 2
34
- 26 73.0769 -3.466794 -4.493444 1 1
35
- 27 74.0741 -5.382220 -5.242719 2 2
36
- 28 75.0000 -4.703293 -5.267189 1 1
37
- 29 72.4138 -4.159544 -4.088655 2 1
38
- 30 73.3333 -2.014174 -2.011087 2 2
39
- 31 74.1935 -5.064910 -5.511064 1 1
40
- 32 75.0000 -4.389097 -6.325314 1 1
41
- 33 75.7576 -4.660306 -5.996384 1 1
42
- 34 76.4706 -2.532510 -2.722643 1 1
43
- 35 77.1429 -4.903158 -4.799878 2 2
44
- 36 77.7778 -4.003703 -3.852717 2 2
45
- 37 78.3784 -3.140848 -3.256392 1 1
46
- 38 76.3158 -4.143925 -4.082227 2 1
47
- 39 76.9231 -5.557262 -4.685828 2 2
48
- 40 75.0000 -6.081103 -6.175181 1 2
49
- 41 75.6098 -2.026120 -2.509788 1 1
50
- 42 76.1905 -2.326648 -3.311518 1 1
51
- 43 74.4186 -3.234551 -3.238431 1 2
52
- 44 75.0000 -2.381371 -1.773752 2 2
53
- 45 75.5556 -0.100998 -12.027444 1 1
54
- 46 76.0870 -2.588329 -2.682551 1 1
55
- 47 74.4681 -3.976639 -4.221735 1 2
56
- 48 75.0000 -4.063995 -3.888061 2 2
57
- 49 73.4694 -2.888681 -1.693157 2 1
58
- 50 72.0000 -2.756531 -2.821922 1 2
59
- 51 72.5490 -2.506640 -3.063721 1 1
60
- 52 73.0769 -3.558265 -2.267377 2 2
61
- 53 73.5849 -3.806557 -4.276422 1 1
62
- 54 74.0741 -2.703078 -2.205066 2 2
63
- 55 74.5455 -2.865460 -2.630393 2 2
64
- 56 73.2143 -3.964649 -4.904109 1 2
65
- 57 73.6842 -2.903156 -2.641858 2 2
66
- 58 74.1379 -3.772213 -4.647779 1 1
67
- 59 74.5763 -3.506344 -3.470757 2 2
68
- 60 75.0000 -4.683885 -4.498544 2 2
69
- 61 75.4098 -3.081809 -3.322363 1 1
70
- 62 75.8065 -5.800490 -6.071895 1 1
71
- 63 74.6032 -4.754691 -4.895220 1 2
72
- 64 73.4375 -3.805163 -3.686909 2 1
73
- 65 73.8462 -5.397812 -4.522152 2 2
74
- 66 74.2424 -3.314953 -3.601342 1 1
75
- 67 74.6269 -3.099614 -2.488757 2 2
76
- 68 73.5294 -5.332057 -5.439553 1 2
77
- 69 73.9130 -4.752548 -7.069320 1 1
78
- 70 74.2857 -3.784065 -3.120995 2 2
79
- 71 73.2394 -3.127587 -2.424565 2 1
80
- 72 73.6111 -3.955679 -3.525416 2 2
81
- 73 73.9726 -2.532634 -2.548937 1 1
82
- 74 74.3243 -2.169942 -2.794400 1 1
83
- 75 74.6667 -2.062663 -2.931119 1 1
84
- 76 75.0000 -2.346954 -4.798073 1 1
85
- 77 75.3247 -2.943148 -2.477373 2 2
86
- 78 75.6410 -1.002248 -1.661147 1 1
87
- 79 74.6835 -2.267207 -2.807933 1 2
88
- 80 75.0000 -2.413518 -3.831530 1 1
89
- 81 75.3086 -6.256596 -4.609384 2 2
90
- 82 74.3902 -3.442162 -3.747318 1 2
91
- 83 74.6988 -6.226698 -5.184180 2 2
92
- 84 75.0000 -2.426614 -2.279968 2 2
93
- 85 75.2941 -5.197813 -5.496329 1 1
94
- 86 74.4186 -5.925068 -5.975219 1 2
95
- 87 73.5632 -1.767306 -1.592768 2 1
96
- 88 72.7273 -5.244862 -4.648801 2 1
97
- 89 73.0337 -3.260030 -3.500960 1 1
98
- 90 73.3333 -4.625819 -3.749420 2 2
99
- 91 72.5275 -4.989864 -4.671907 2 1
100
- 92 72.8261 -4.988968 -4.541605 2 2
101
- 93 73.1183 -2.725417 -2.665505 2 2
102
- 94 73.4043 -4.833009 -4.053913 2 2
103
- 95 73.6842 -3.755160 -4.019369 1 1
104
- 96 73.9583 -1.544636 -2.845248 1 1
105
- 97 74.2268 -4.721530 -5.267744 1 1
106
- 98 74.4898 -3.909052 -3.478139 2 2
107
- 99 73.7374 -2.654876 -2.546609 2 1
108
- 100 74.0000 -5.934973 -4.945826 2 2
109
- 101 73.2673 -4.804340 -4.820332 1 2
110
- 102 73.5294 -3.728794 -3.775764 1 1
111
- 103 73.7864 -3.220513 -2.950880 2 2
112
- 104 74.0385 -7.578451 -7.203027 2 2
113
- 105 74.2857 -1.076577 -2.241513 1 1
114
- 106 74.5283 -2.792448 -2.455632 2 2
115
- 107 74.7664 -2.330571 -2.031398 2 2
116
- 108 75.0000 -4.255724 -3.383344 2 2
117
- 109 74.3119 -4.747777 -5.465454 1 2
118
- 110 74.5455 -4.774388 -3.748404 2 2
119
- 111 74.7748 -3.211093 -2.945482 2 2
120
- 112 74.1071 -4.040776 -3.284443 2 1
121
- 113 73.4513 -5.138307 -6.394355 1 2
122
- 114 73.6842 -4.177353 -5.081490 1 1
123
- 115 73.9130 -1.515202 -2.241961 1 1
124
- 116 74.1379 -3.322567 -2.802502 2 2
125
- 117 73.5043 -4.415501 -4.653490 1 2
126
- 118 73.7288 -1.365483 -2.300375 1 1
127
- 119 73.9496 -1.963738 -2.875325 1 1
128
- 120 73.3333 -3.111470 -2.799199 2 1
129
- 121 73.5537 -0.954586 -0.757157 2 2
130
- 122 73.7705 -6.721692 -5.951168 2 2
131
- 123 73.9837 -3.615697 -4.202824 1 1
132
- 124 74.1935 -2.713786 -2.451111 2 2
133
- 125 74.4000 -2.678940 -2.371234 2 2
134
- 126 74.6032 -7.166794 -6.701796 2 2
135
- 127 74.8031 -2.921472 -2.057755 2 2
136
- 128 75.0000 -1.680120 -1.793529 1 1
137
- 129 75.1938 -2.293719 -1.786393 2 2
138
- 130 75.3846 -6.261825 -5.533078 2 2
139
- 131 74.8092 -3.479991 -3.494766 1 2
140
- 132 75.0000 -2.364025 -2.064741 2 2
141
- 133 75.1880 -2.051158 -4.219316 1 1
142
- 134 75.3731 -5.234619 -4.474929 2 2
143
- 135 75.5556 -4.723645 -5.524661 1 1
144
- 136 75.7353 -3.604099 -2.514437 2 2
145
- 137 75.9124 -5.405852 -5.638381 1 1
146
- 138 75.3623 -3.764777 -3.974575 1 2
147
- 139 75.5396 -4.553334 -5.064232 1 1
148
- 140 75.7143 -1.351445 -1.038132 2 2
149
- 141 75.8865 -5.180614 -3.976336 2 2
150
- 142 76.0563 -3.190860 -2.383941 2 2
151
- 143 76.2238 -4.630512 -7.313096 1 1
152
- 144 76.3889 -2.315805 -2.341810 1 1
153
- 145 75.8621 -1.439435 -14.899495 1 2
154
- 146 75.3425 -3.833098 -3.937038 1 2
155
- 147 75.5102 -4.890879 -4.142333 2 2
156
- 148 75.6757 -1.814803 -2.464769 1 1
157
- 149 75.8389 -3.961458 -3.774420 2 2
158
- 150 76.0000 -4.391448 -3.380520 2 2
159
- 151 75.4967 -1.891254 -1.835047 2 1
160
- 152 75.0000 -4.057164 -4.376886 1 2
161
- 153 75.1634 -2.744061 -3.297084 1 1
162
- 154 75.3247 -3.570136 -3.825418 1 1
163
- 155 74.8387 -2.485600 -2.705277 1 2
164
- 156 75.0000 -2.022888 -2.299520 1 1
165
- 157 75.1592 -2.866172 -4.043398 1 1
166
- 158 75.3165 -4.763031 -4.190783 2 2
167
- 159 75.4717 -3.593562 -4.909765 1 1
168
- 160 75.6250 -3.785105 -3.536138 2 2
169
- 161 75.7764 -1.849502 -2.087705 1 1
170
- 162 75.3086 -0.073175 -12.879461 1 2
171
- 163 75.4601 -2.215839 -2.501852 1 1
172
- 164 75.6098 -5.066327 -3.836260 2 2
173
- 165 75.1515 -3.854185 -4.003379 1 2
174
- 166 75.3012 -5.003896 -4.296535 2 2
175
- 167 74.8503 -4.843620 -4.617793 2 1
176
- 168 75.0000 -4.322782 -3.629503 2 2
177
- 169 74.5562 -4.182545 -4.698117 1 2
178
- 170 74.7059 -2.389002 -3.007776 1 1
179
- 171 74.8538 -3.847777 -4.693920 1 1
180
- 172 75.0000 -3.787814 -5.245786 1 1
181
- 173 75.1445 -2.979184 -2.650157 2 2
182
- 174 75.2874 -5.562902 -5.214144 2 2
183
- 175 75.4286 -3.863761 -4.170976 1 1
184
- 176 75.5682 -2.227906 -3.736512 1 1
185
- 177 75.7062 -5.976926 -5.807745 2 2
186
- 178 75.2809 -2.699059 -2.425066 2 1
187
- 179 75.4190 -2.457998 -2.763863 1 1
188
- 180 75.5556 -6.019433 -5.501569 2 2
189
- 181 75.6906 -3.799884 -2.872020 2 2
190
- 182 75.8242 -4.048113 -3.127023 2 2
191
- 183 75.9563 -5.236752 -4.846253 2 2
192
- 184 76.0870 -1.484578 -2.407790 1 1
193
- 185 75.6757 -4.504518 -4.798019 1 2
194
- 186 75.8065 -3.162872 -3.919480 1 1
195
- 187 75.9358 -4.896515 -4.305174 2 2
196
- 188 75.5319 -4.447998 -3.505265 2 1
197
- 189 75.1323 -4.751043 -5.180445 1 2
198
- 190 75.2632 -4.678551 -4.502705 2 2
199
- 191 75.3927 -2.989015 -3.734836 1 1
200
- 192 75.5208 -5.033810 -3.933192 2 2
201
- 193 75.6477 -4.045457 -3.999330 2 2
202
- 194 75.7732 -1.856347 -1.545907 2 2
203
- 195 75.8974 -3.103245 -3.177436 1 1
204
- 196 75.5102 -3.357856 -3.548646 1 2
205
- 197 75.6345 -0.830488 -4.277023 1 1
206
- 198 75.7576 -4.139133 -7.305645 1 1
207
- 199 75.8794 -5.495628 -4.490810 2 2
208
- 200 76.0000 -3.278700 -3.016630 2 2
209
- 201 76.1194 -3.068680 -3.164790 1 1
210
- 202 76.2376 -9.100143 -8.375798 2 2
211
- 203 76.3547 -3.377599 -2.985166 2 2
212
- 204 76.4706 -2.169682 -3.026294 1 1
213
- 205 76.5854 -3.620867 -2.383338 2 2
214
- 206 76.2136 -4.275328 -4.554796 1 2
215
- 207 76.3285 -3.092697 -3.746993 1 1
216
- 208 76.4423 -3.507107 -2.740535 2 2
217
- 209 76.5550 -3.915269 -3.324216 2 2
218
- 210 76.6667 -6.739055 -6.620506 2 2
219
- 211 76.3033 -5.059892 -4.906681 2 1
220
- 212 76.4151 -4.845002 -3.994756 2 2
221
- 213 76.0563 -3.595459 -4.962977 1 2
222
- 214 75.7009 -3.357013 -3.258168 2 1
223
- 215 75.8140 -4.484510 -4.002797 2 2
224
- 216 75.4630 -3.213620 -3.134326 2 1
225
- 217 75.5760 -1.229037 -1.842573 1 1
226
- 218 75.6881 -2.877757 -2.427233 2 2
227
- 219 75.7991 -3.195300 -3.259522 1 1
228
- 220 75.4545 -3.958401 -4.379782 1 2
229
- 221 75.5656 -4.424751 -8.885927 1 1
230
- 222 75.2252 -0.511913 -9.500249 1 2
231
- 223 75.3363 -4.120785 -3.189393 2 2
232
- 224 75.4464 -5.935741 -5.005898 2 2
233
- 225 75.1111 -2.962243 -3.112580 1 2
234
- 226 75.2212 -2.967889 -3.712960 1 1
235
- 227 75.3304 -3.074818 -3.087136 1 1
236
- 228 75.4386 -1.741042 -2.648229 1 1
237
- 229 75.5459 -3.164381 -2.967195 2 2
238
- 230 75.6522 -3.211317 -3.695328 1 1
239
- 231 75.7576 -2.444014 -2.664121 1 1
240
- 232 75.8621 -1.975608 -1.787008 2 2
241
- 233 75.9657 -1.348736 -1.936096 1 1
242
- 234 76.0684 -3.650460 -3.136969 2 2
243
- 235 76.1702 -3.421951 -2.854602 2 2
244
- 236 75.8475 -1.709700 -1.424582 2 1
245
- 237 75.9494 -2.607247 -2.684552 1 1
246
- 238 76.0504 -2.750936 -2.447459 2 2
247
- 239 75.7322 -2.711424 -2.818043 1 2
248
- 240 75.8333 -7.279760 -7.139847 2 2
249
- 241 75.5187 -3.622898 -4.292024 1 2
250
- 242 75.6198 -5.686004 -5.370683 2 2
251
- 243 75.3086 -4.237090 -3.896119 2 1
252
- 244 75.4098 -1.998318 -4.041572 1 1
253
- 245 75.5102 -4.219659 -4.719025 1 1
254
- 246 75.6098 -2.477879 -3.014289 1 1
255
- 247 75.7085 -3.370323 -3.336628 2 2
256
- 248 75.8065 -3.342562 -3.030790 2 2
257
- 249 75.9036 -2.904778 -2.100368 2 2
258
- 250 76.0000 -2.421578 -3.015761 1 1
259
- 251 76.0956 -1.447500 -2.179036 1 1
260
- 252 76.1905 -7.298458 -6.416162 2 2
261
- 253 75.8893 -5.559126 -5.324866 2 1
262
- 254 75.9843 -4.695471 -4.159447 2 2
263
- 255 76.0784 -3.296144 -4.492035 1 1
264
- 256 75.7812 -3.069317 -2.927357 2 1
265
- 257 75.4864 -2.613685 -2.024665 2 1
266
- 258 75.1938 -3.813113 -3.984035 1 2
267
- 259 75.2896 -5.207438 -5.445568 1 1
268
- 260 75.0000 -5.912292 -5.611626 2 1
269
- 261 75.0958 -3.761583 -3.733000 2 2
270
- 262 75.1908 -4.902678 -4.559943 2 2
271
- 263 75.2852 -3.507805 -4.318757 1 1
272
- 264 75.3788 -4.063264 -3.427114 2 2
273
- 265 75.4717 -1.131432 -2.194524 1 1
274
- 266 75.5639 -5.267309 -6.471044 1 1
275
- 267 75.2809 -3.735959 -3.418087 2 1
276
- 268 75.0000 -3.038782 -2.992093 2 1
277
- 269 75.0929 -3.048217 -2.903071 2 2
278
- 270 75.1852 -2.623777 -3.315277 1 1
279
- 271 75.2768 -3.148788 -2.786349 2 2
280
- 272 75.3676 -4.035127 -3.615057 2 2
281
- 273 75.4579 -3.355333 -3.237514 2 2
282
- 274 75.5474 -3.597148 -3.460104 2 2
283
- 275 75.6364 -2.669858 -2.844119 1 1
284
- 276 75.3623 -2.618412 -2.533853 2 1
285
- 277 75.4513 -2.710283 -3.500794 1 1
286
- 278 75.5396 -2.952472 -3.073151 1 1
287
- 279 75.6272 -4.566366 -2.795009 2 2
288
- 280 75.7143 -3.763848 -3.322875 2 2
289
- 281 75.8007 -2.529173 -1.652402 2 2
290
- 282 75.8865 -4.080788 -3.570040 2 2
291
- 283 75.9717 -3.325534 -3.182465 2 2
292
- 284 76.0563 -2.791281 -3.648704 1 1
293
- 285 76.1404 -3.403263 -5.670537 1 1
294
- 286 76.2238 -3.780062 -3.902052 1 1
295
- 287 76.3066 -3.156338 -4.776638 1 1
296
- 288 76.3889 -3.709810 -3.043038 2 2
297
- 289 76.4706 -2.240078 -3.108900 1 1
298
- 290 76.5517 -4.137402 -4.791507 1 1
299
- 291 76.6323 -2.130457 -2.804418 1 1
300
- 292 76.7123 -2.798812 -2.199756 2 2
301
- 293 76.7918 -3.891319 -3.294811 2 2
302
- 294 76.5306 -3.939539 -3.363028 2 1
303
- 295 76.2712 -2.148370 -2.093177 2 1
304
- 296 76.3514 -5.241541 -4.975015 2 2
305
- 297 76.0943 -1.550812 -1.886290 1 2
306
- 298 76.1745 -5.490792 -5.031482 2 2
307
- 299 76.2542 -2.314991 -2.206178 2 2
308
- 300 76.3333 -2.086814 -2.474054 1 1
309
- 301 76.4120 -4.215546 -4.454723 1 1
310
- 302 76.1589 -4.056640 -4.122735 1 2
311
- 303 75.9076 -8.789881 -8.555219 2 1
312
- 304 75.9868 -4.444102 -3.148021 2 2
313
- 305 76.0656 -3.601351 -3.364747 2 2
314
- 306 76.1438 -2.244508 -1.933598 2 2
315
- 307 76.2215 -2.660453 -3.268669 1 1
316
- 308 76.2987 -4.922258 -4.115546 2 2
317
- 309 76.3754 -4.162449 -4.808254 1 1
318
- 310 76.1290 -5.815636 -5.870791 1 2
319
- 311 76.2058 -0.598358 -4.687351 1 1
320
- 312 75.9615 -4.267676 -4.154979 2 1
321
- 313 76.0383 -4.906751 -4.758641 2 2
322
- 314 76.1146 -4.728469 -4.421876 2 2
323
- 315 76.1905 -1.986280 -3.441134 1 1
324
- 316 76.2658 -7.056142 -4.836820 2 2
325
- 317 76.0252 -4.935123 -4.929174 2 1
326
- 318 76.1006 -4.797128 -4.481421 2 2
327
- 319 76.1755 -1.394660 -1.614995 1 1
328
- 320 76.2500 -3.816496 -4.814373 1 1
329
- 321 76.0125 -2.898412 -2.908198 1 2
330
- 322 75.7764 -2.524740 -2.809752 1 2
331
- 323 75.8514 -3.583074 -4.011282 1 1
332
- 324 75.6173 -4.495127 -4.282003 2 1
333
- 325 75.6923 -5.643873 -5.276181 2 2
334
- 326 75.7669 -2.677631 -1.984794 2 2
335
- 327 75.8410 -2.224206 -2.815942 1 1
336
- 328 75.9146 -4.256442 -4.357086 1 1
337
- 329 75.9878 -3.591678 -7.885498 1 1
338
- 330 76.0606 -1.356892 -2.061393 1 1
339
- 331 75.8308 -3.830631 -3.360066 2 1
340
- 332 75.9036 -1.598573 -2.564519 1 1
341
- 333 75.9760 -3.854169 -1.358419 2 2
342
- 334 76.0479 -2.860685 -5.551284 1 1
343
- 335 75.8209 -5.457644 -4.943791 2 1
344
- 336 75.5952 -3.228648 -3.818838 1 2
345
- 337 75.6677 -2.110184 -1.541741 2 2
346
- 338 75.4438 -3.642691 -3.014620 2 1
347
- 339 75.2212 -5.561210 -5.590662 1 2
348
- 340 75.2941 -4.887138 -5.598681 1 1
349
- 341 75.3666 -3.836068 -4.885107 1 1
350
- 342 75.4386 -5.539666 -4.874341 2 2
351
- 343 75.5102 -3.153946 -3.789643 1 1
352
- 344 75.5814 -1.987747 -3.130948 1 1
353
- 345 75.6522 -0.744080 -2.912490 1 1
354
- 346 75.7225 -6.046284 -5.890066 2 2
355
- 347 75.7925 -3.345294 -2.805161 2 2
356
- 348 75.8621 -0.904148 -1.743983 1 1
357
- 349 75.6447 -3.371418 -4.484200 1 2
358
- 350 75.4286 -2.187053 -4.165557 1 2
359
- 351 75.4986 -2.999509 -3.808236 1 1
360
- 352 75.5682 -3.493578 -2.646553 2 2
361
- 353 75.6374 -2.264035 -2.354458 1 1
362
- 354 75.7062 -5.803949 -5.171781 2 2
363
- 355 75.7746 -2.835742 -2.360867 2 2
364
- 356 75.5618 -3.185820 -3.777185 1 2
365
- 357 75.6303 -4.369053 -4.098013 2 2
366
- 358 75.6983 -2.110005 -1.806577 2 2
367
- 359 75.7660 -5.452168 -3.337437 2 2
368
- 360 75.5556 -3.559513 -3.612020 1 2
369
- 361 75.6233 -2.944733 -1.494101 2 2
370
- 362 75.6906 -6.160593 -5.136144 2 2
371
- 363 75.4821 -2.750495 -2.547756 2 1
372
- 364 75.5495 -6.213554 -6.550668 1 1
373
- 365 75.6164 -5.345643 -5.436150 1 1
374
- 366 75.4098 -2.557686 -7.122428 1 2
375
- 367 75.4768 -5.080946 -5.357377 1 1
376
- 368 75.5435 -4.109037 -3.399486 2 2
377
- 369 75.3388 -3.509646 -3.833158 1 2
378
- 370 75.4054 -4.772972 -4.934324 1 1
379
- 371 75.4717 -2.170468 -3.453233 1 1
380
- 372 75.5376 -2.309317 -2.532939 1 1
381
- 373 75.3351 -4.100992 -3.384680 2 1
382
- 374 75.4011 -2.257812 -2.974429 1 1
383
- 375 75.4667 -3.496527 -2.466250 2 2
384
- 376 75.5319 -4.504804 -3.633504 2 2
385
- 377 75.5968 -4.150199 -3.827150 2 2
386
- 378 75.3968 -3.040668 -3.417627 1 2
387
- 379 75.1979 -0.629060 -1.954349 1 2
388
- 380 75.2632 -6.123642 -6.700254 1 1
389
- 381 75.3281 -10.549230 -9.299952 2 2
390
- 382 75.3927 -2.600703 -2.538908 2 2
391
- 383 75.4569 -2.052328 -1.389470 2 2
392
- 384 75.2604 -3.567934 -2.708439 2 1
393
- 385 75.0649 -0.765746 -0.743069 2 1
394
- 386 75.1295 -2.748534 -14.197248 1 1
395
- 387 75.1938 -1.662318 -2.453434 1 1
396
- 388 75.2577 -5.797068 -5.728326 2 2
397
- 389 75.3213 -2.942791 -2.861813 2 2
398
- 390 75.1282 -2.988605 -2.924577 2 1
399
- 391 74.9361 -4.183996 -4.752492 1 2
400
- 392 74.7449 -4.653978 -3.713541 2 1
401
- 393 74.5547 -6.178029 -6.228019 1 2
402
- 394 74.6193 -2.515577 -3.170799 1 1
403
- 395 74.6835 -3.183637 -3.061860 2 2
404
- 396 74.7475 -3.451034 -4.017899 1 1
405
- 397 74.8111 -6.472212 -7.091749 1 1
406
- 398 74.8744 -2.782765 -5.048210 1 1
407
- 399 74.9373 -2.945872 -3.864199 1 1
408
- 400 75.0000 -1.752115 -7.287173 1 1
409
- 401 75.0623 -3.525935 -3.952518 1 1
410
- 402 75.1244 -3.672424 -3.540745 2 2
411
- 403 74.9380 -5.706786 -6.478608 1 2
412
- 404 75.0000 -4.376811 -2.917417 2 2
413
- 405 75.0617 -3.155027 -4.277821 1 1
414
- 406 75.1232 -7.848854 -8.427352 1 1
415
- 407 74.9386 -5.115051 -5.152937 1 2
416
- 408 74.7549 -3.188613 -3.340531 1 2
417
- 409 74.8166 -3.980111 -4.706951 1 1
418
- 410 74.8780 -3.623977 -3.671301 1 1
419
- 411 74.9392 -3.791473 -5.034071 1 1
420
- 412 75.0000 -3.913579 -3.317090 2 2
421
- 413 75.0605 -3.418798 -2.558263 2 2
422
- 414 74.8792 -4.111796 -4.629603 1 2
423
- 415 74.9398 -3.787318 -3.319336 2 2
424
- 416 75.0000 -4.161789 -1.990889 2 2
425
- 417 75.0600 -4.062176 -5.050584 1 1
426
- 418 75.1196 -5.334681 -5.931920 1 1
427
- 419 75.1790 -3.564157 -3.380878 2 2
428
- 420 75.0000 -3.202935 -3.406956 1 2
429
- 421 75.0594 -4.257522 -3.922893 2 2
430
- 422 75.1185 -2.017487 -2.344226 1 1
431
- 423 75.1773 -3.439120 -3.503446 1 1
432
- 424 75.2358 -4.810982 -5.651562 1 1
433
- 425 75.2941 -5.149172 -2.941122 2 2
434
- 426 75.3521 -1.927414 -1.642814 2 2
435
- 427 75.4098 -3.729618 -4.441014 1 1
436
- 428 75.4673 -3.242109 -3.851976 1 1
437
- 429 75.5245 -3.237835 -3.209292 2 2
438
- 430 75.5814 -5.061032 -5.291825 1 1
439
- 431 75.6381 -3.697881 -5.388578 1 1
440
- 432 75.4630 -3.119182 -3.839663 1 2
441
- 433 75.5196 -2.308137 -1.966434 2 2
442
- 434 75.5760 -6.941461 -7.788227 1 1
443
- 435 75.6322 -3.705199 -3.774015 1 1
444
- 436 75.4587 -2.340396 -2.216386 2 1
445
- 437 75.5149 -4.074306 -4.028910 2 2
446
- 438 75.5708 -3.848799 -3.530167 2 2
447
- 439 75.6264 -2.823722 -3.368750 1 1
448
- 440 75.6818 -3.784134 -4.404306 1 1
449
- 441 75.7370 -3.601819 -4.803755 1 1
450
- 442 75.7919 -4.280212 -4.348256 1 1
451
- 443 75.8465 -1.708810 -3.406029 1 1
452
- 444 75.6757 -5.905006 -6.026115 1 2
453
- 445 75.7303 -1.653794 -2.302348 1 1
454
- 446 75.7848 -4.300918 -2.999596 2 2
455
- 447 75.6152 -3.144322 -2.786745 2 1
456
- 448 75.6696 -5.047641 -4.708449 2 2
457
- 449 75.7238 -2.461446 -2.121894 2 2
458
- 450 75.7778 -6.133792 -5.338863 2 2
459
- 451 75.8315 -3.959711 -3.829292 2 2
460
- 452 75.6637 -4.388934 -4.013073 2 1
461
- 453 75.7174 -4.194315 -3.416071 2 2
462
- 454 75.7709 -3.669598 -4.065974 1 1
463
- 455 75.6044 -2.192533 -10.918139 1 2
464
- 456 75.6579 -3.736434 -3.658490 2 2
465
- 457 75.4923 -8.850690 -8.408135 2 1
466
- 458 75.5459 -3.377010 -2.412436 2 2
467
- 459 75.5991 -4.020337 -3.782721 2 2
468
- 460 75.6522 -0.667576 -9.360743 1 1
469
- 461 75.7050 -2.366160 -3.538671 1 1
470
- 462 75.5411 -5.661176 -5.299925 2 1
471
- 463 75.5940 -1.327648 -1.581504 1 1
472
- 464 75.6466 -2.276721 -2.759466 1 1
473
- 465 75.6989 -3.537862 -2.307130 2 2
474
- 466 75.7511 -5.535852 -5.485519 2 2
475
- 467 75.5889 -5.011769 -4.692547 2 1
476
- 468 75.6410 -4.280712 -3.384286 2 2
477
- 469 75.6930 -1.225477 -1.549065 1 1
478
- 470 75.7447 -1.980907 -1.506910 2 2
479
- 471 75.7962 -2.494808 -2.960757 1 1
480
- 472 75.6356 -3.443797 -3.539916 1 2
481
- 473 75.6871 -3.879374 -3.083358 2 2
482
- 474 75.7384 -5.041722 -6.173255 1 1
483
- 475 75.7895 -2.610695 -2.303227 2 2
484
- 476 75.6303 -4.414364 -4.460398 1 2
485
- 477 75.4717 -4.306448 -4.675563 1 2
486
- 478 75.5230 -2.470951 -2.597736 1 1
487
- 479 75.3653 -3.040916 -2.999226 2 1
488
- 480 75.4167 -5.752794 -5.280787 2 2
489
- 481 75.4678 -2.197838 -2.416918 1 1
490
- 482 75.3112 -4.470367 -5.233910 1 2
491
- 483 75.3623 -4.000229 -4.456536 1 1
492
- 484 75.2066 -2.740991 -2.124690 2 1
493
- 485 75.2577 -1.371292 -1.912002 1 1
494
- 486 75.3086 -2.577526 -3.351085 1 1
495
- 487 75.3593 -3.560248 -3.180624 2 2
496
- 488 75.4098 -4.098474 -3.025200 2 2
497
- 489 75.4601 -2.021596 -2.777590 1 1
498
- 490 75.5102 -2.696670 -3.029559 1 1
499
- 491 75.5601 -5.168604 -4.240366 2 2
500
- 492 75.6098 -5.168438 -4.243580 2 2
501
- 493 75.6592 -1.345368 -2.250503 1 1
502
- 494 75.7085 -3.479796 -2.839432 2 2
503
- 495 75.7576 -4.015556 -4.081096 1 1
504
- 496 75.8065 -3.301469 -3.054182 2 2
505
- 497 75.8551 -4.871363 -5.184426 1 1
506
- 498 75.7028 -1.772877 -2.085286 1 2
507
- 499 75.7515 -2.820192 -3.397775 1 1
508
- 500 75.8000 -5.807620 -5.026447 2 2
509
- 501 75.8483 -2.656950 -2.792606 1 1
510
- 502 75.8964 -1.262148 -2.484801 1 1
511
- 503 75.9443 -3.759994 -3.401260 2 2
512
- 504 75.9921 -2.768531 -2.827919 1 1
513
- 505 76.0396 -7.200277 -6.868652 2 2
514
- 506 76.0870 -4.097976 -4.887414 1 1
515
- 507 76.1341 -3.186639 -3.644165 1 1
516
- 508 76.1811 -5.059479 -2.911945 2 2
517
- 509 76.2279 -6.854768 -10.133657 1 1
518
- 510 76.0784 -3.581489 -3.387447 2 1
519
- 511 76.1252 -5.664878 -4.089642 2 2
520
- 512 76.1719 -3.022454 -3.433969 1 1
521
- 513 76.0234 -2.674391 -2.738991 1 2
522
- 514 75.8755 -4.649695 -3.907766 2 1
523
- 515 75.9223 -6.453047 -9.190205 1 1
524
- 516 75.7752 -4.512825 -4.506096 2 1
525
- 517 75.8221 -1.594395 -1.830382 1 1
526
- 518 75.8687 -2.589224 -2.538609 2 2
527
- 519 75.9152 -7.039417 -6.598190 2 2
528
- 520 75.9615 -2.589269 -2.569717 2 2
529
- 521 76.0077 -1.862302 -2.071094 1 1
530
- 522 76.0536 -3.905791 -2.374030 2 2
531
- 523 76.0994 -3.903749 -3.554855 2 2
532
- 524 76.1450 -4.084810 -4.571526 1 1
533
- 525 76.1905 -4.414790 -2.589859 2 2
534
- 526 76.2357 -1.026425 -14.246471 1 1
535
- 527 76.2808 -6.525455 -4.169796 2 2
536
- 528 76.3258 -5.926240 -5.791212 2 2
537
- 529 76.3705 -5.521376 -6.251120 1 1
538
- 530 76.4151 -4.649323 -2.878068 2 2
539
- 531 76.4595 -3.476515 -3.854250 1 1
540
- 532 76.5038 -2.027778 -3.392522 1 1
541
- 533 76.5478 -3.489665 -2.472489 2 2
542
- 534 76.5918 -2.299031 -2.866709 1 1
543
- 535 76.4486 -3.541853 -3.605466 1 2
544
- 536 76.3060 -2.448860 -2.217975 2 1
545
- 537 76.3501 -4.977170 -6.183281 1 1
546
- 538 76.3941 -2.324200 -4.880809 1 1
547
- 539 76.4378 -3.880577 -1.930328 2 2
548
- 540 76.4815 -2.813288 -2.642084 2 2
549
- 541 76.5250 -2.666698 -3.431723 1 1
550
- 542 76.3838 -4.099127 -4.699935 1 2
551
- 543 76.4273 -5.651197 -3.984096 2 2
552
- 544 76.4706 -3.471979 -3.840936 1 1
553
- 545 76.3303 -2.162662 -3.929876 1 2
554
- 546 76.3736 -2.880491 -4.747735 1 1
555
- 547 76.4168 -5.948875 -5.378456 2 2
556
- 548 76.2774 -4.337137 -4.246847 2 1
557
- 549 76.3206 -5.363940 -5.640406 1 1
558
- 550 76.1818 -2.682342 -2.737406 1 2
559
- 551 76.0436 -0.912113 -13.339571 1 2
560
- 552 75.9058 -1.989716 -1.709679 2 1
561
- 553 75.9494 -5.112836 -5.090178 2 2
562
- 554 75.8123 -4.805760 -4.817634 1 2
563
- 555 75.8559 -1.830882 -1.377471 2 2
564
- 556 75.8993 -5.317393 -5.011626 2 2
565
- 557 75.9425 -4.861079 -4.967060 1 1
566
- 558 75.9857 -1.523373 -3.002577 1 1
567
- 559 76.0286 -2.572911 -2.466426 2 2
568
- 560 76.0714 -2.297816 -2.423772 1 1
569
- 561 76.1141 -2.354695 -2.946059 1 1
570
- 562 76.1566 -0.726961 -1.899624 1 1
571
- 563 76.0213 -1.381280 -2.448614 1 2
572
- 564 76.0638 -2.888596 -3.180009 1 1
573
- 565 76.1062 -4.286591 -3.874127 2 2
574
- 566 76.1484 -4.084661 -4.598557 1 1
575
- 567 76.1905 -1.920294 -1.616245 2 2
576
- 568 76.2324 -7.877369 -3.399316 2 2
577
- 569 76.0984 -5.580405 -4.546635 2 1
578
- 570 76.1404 -4.024067 -5.034360 1 1
579
- 571 76.1821 -4.051126 -3.183553 2 2
580
- 572 76.2238 -2.366670 -2.125248 2 2
581
- 573 76.0908 -4.025572 -4.270216 1 2
582
- 574 76.1324 -3.967357 -3.865241 2 2
583
- 575 76.1739 -5.365466 -5.307762 2 2
584
- 576 76.2153 -3.567473 -4.131577 1 1
585
- 577 76.2565 -4.110363 -3.426773 2 2
586
- 578 76.2976 -2.848560 -2.748712 2 2
587
- 579 76.1658 -4.268475 -4.717311 1 2
588
- 580 76.2069 -1.978422 -2.405109 1 1
589
- 581 76.0757 -6.741199 -6.180228 2 1
590
- 582 76.1168 -5.735488 -5.574275 2 2
591
- 583 76.1578 -4.781911 -4.505411 2 2
592
- 584 76.0274 -4.894618 -4.593395 2 1
593
- 585 76.0684 -0.457700 -1.138071 1 1
594
- 586 76.1092 -3.677970 -3.765078 1 1
595
- 587 76.1499 -5.360595 -5.371619 1 1
596
- 588 76.1905 -3.651240 -2.787786 2 2
597
- 589 76.2309 -3.876093 -3.853065 2 2
598
- 590 76.2712 -3.987072 -3.265087 2 2
599
- 591 76.3113 -3.711125 -3.412356 2 2
600
- 592 76.3514 -2.996173 -3.846171 1 1
601
- 593 76.3912 -3.928472 -3.369241 2 2
602
- 594 76.2626 -4.419807 -4.172922 2 1
603
- 595 76.1345 -1.536119 -1.563012 1 2
604
- 596 76.1745 -5.160928 -4.895800 2 2
605
- 597 76.2144 -1.887084 -2.065745 1 1
606
- 598 76.2542 -3.091282 -3.844700 1 1
607
- 599 76.2938 -2.983325 -3.061582 1 1
608
- 600 76.3333 -4.141816 -3.976379 2 2
609
- 601 76.3727 -3.226695 -3.166833 2 2
610
- 602 76.4120 -2.243173 -3.772404 1 1
611
- 603 76.4511 -5.992207 -5.593081 2 2
612
- 604 76.3245 -2.987515 -3.029588 1 2
613
- 605 76.1983 -2.866509 -2.360199 2 1
614
- 606 76.2376 -3.311217 -2.663459 2 2
615
- 607 76.1120 -2.429266 -1.856273 2 1
616
- 608 76.1513 -2.347032 -2.905406 1 1
617
- 609 76.1905 -5.781290 -6.526100 1 1
618
- 610 76.0656 -2.664124 -2.605593 2 1
619
- 611 76.1047 -5.065400 -5.158870 1 1
620
- 612 76.1438 -3.339250 -5.753202 1 1
621
- 613 76.1827 -2.479705 -2.092921 2 2
622
- 614 76.2215 -2.554749 -1.772408 2 2
623
- 615 76.2602 -2.414392 -2.190584 2 2
624
- 616 76.2987 -3.554322 -3.369440 2 2
625
- 617 76.3371 -12.103456 -3.476103 2 2
626
- 618 76.2136 -3.977639 -4.096760 1 2
627
- 619 76.2520 -3.439955 -2.769356 2 2
628
- 620 76.1290 -4.762887 -4.471746 2 1
629
- 621 76.1675 -3.421144 -3.087451 2 2
630
- 622 76.0450 -2.503959 -2.437519 2 1
631
- 623 75.9230 -2.567982 -2.345865 2 1
632
- 624 75.9615 -4.416440 -3.832741 2 2
633
- 625 76.0000 -3.436603 -5.472248 1 1
634
- 626 76.0383 -2.303326 -3.194900 1 1
635
- 627 76.0766 -3.879554 -3.928421 1 1
636
- 628 76.1146 -2.539125 -2.296863 2 2
637
- 629 76.1526 -1.559105 -2.002126 1 1
638
- 630 76.1905 -1.265086 -1.872371 1 1
639
- 631 76.2282 -1.486301 -2.582404 1 1
640
- 632 76.2658 -2.094362 -1.282009 2 2
641
- 633 76.3033 -3.069804 -2.491330 2 2
642
- 634 76.3407 -2.397671 -2.854271 1 1
643
- 635 76.2205 -3.815829 -4.743527 1 2
644
- 636 76.2579 -2.176522 -4.040909 1 1
645
- 637 76.2951 -3.409198 -5.545778 1 1
646
- 638 76.3323 -4.953012 -5.296922 1 1
647
- 639 76.3693 -1.664075 -2.157493 1 1
648
- 640 76.2500 -1.363817 -2.111424 1 2
649
- 641 76.2871 -4.143036 -4.961105 1 1
650
- 642 76.3240 -4.502244 -2.363125 2 2
651
- 643 76.3608 -1.186888 -0.583270 2 2
652
- 644 76.3975 -3.601633 -4.758821 1 1
653
- 645 76.4341 -1.585566 -1.657425 1 1
654
- 646 76.3158 -4.212878 -4.025835 2 1
655
- 647 76.1978 -4.622797 -3.980535 2 1
656
- 648 76.2346 -3.362607 -3.143490 2 2
657
- 649 76.2712 -2.792945 -3.035083 1 1
658
- 650 76.3077 -3.375988 -4.156296 1 1
659
- 651 76.3441 -5.651464 -5.415671 2 2
660
- 652 76.3804 -5.850210 -5.144272 2 2
661
- 653 76.4165 -6.522597 -5.438709 2 2
662
- 654 76.2997 -3.154461 -3.432470 1 2
663
- 655 76.3359 -2.608589 -2.541746 2 2
664
- 656 76.3720 -3.953304 -1.968076 2 2
665
- 657 76.2557 -5.383312 -5.990216 1 2
666
- 658 76.2918 -5.605238 -3.550627 2 2
667
- 659 76.3278 -2.260438 -2.072905 2 2
668
- 660 76.3636 -3.367128 -2.400516 2 2
669
- 661 76.3994 -2.798123 -2.884034 1 1
670
- 662 76.4350 -3.232407 -3.640919 1 1
671
- 663 76.4706 -5.997135 -4.668238 2 2
672
- 664 76.3554 -2.867210 -2.841692 2 1
673
- 665 76.3910 -3.959906 -3.845268 2 2
674
- 666 76.4264 -4.330925 -4.579209 1 1
675
- 667 76.3118 -5.319892 -9.236569 1 2
676
- 668 76.3473 -2.837915 -3.557476 1 1
677
- 669 76.3827 -2.882022 -3.237558 1 1
678
- 670 76.2687 -3.536850 -2.845000 2 1
679
- 671 76.3040 -3.463813 -4.964452 1 1
680
- 672 76.3393 -2.815099 -2.700773 2 2
681
- 673 76.2259 -1.867627 -1.893861 1 2
682
- 674 76.2611 -5.976880 -6.005303 1 1
683
- 675 76.1481 -3.952022 -3.656991 2 1
684
- 676 76.0355 -1.975106 -2.719159 1 2
685
- 677 76.0709 -3.835747 -4.049139 1 1
686
- 678 75.9587 -5.231865 -5.316621 1 2
687
- 679 75.9941 -5.381824 -4.819479 2 2
688
- 680 76.0294 -3.164604 -4.877905 1 1
689
- 681 75.9178 -3.857631 -2.817287 2 1
690
- 682 75.8065 -2.238639 -2.362845 1 2
691
- 683 75.8419 -1.556160 -2.702363 1 1
692
- 684 75.8772 -4.825953 -8.636149 1 1
693
- 685 75.9124 -3.524755 -1.484637 2 2
694
- 686 75.9475 -5.170671 -4.831746 2 2
695
- 687 75.9825 -2.836606 -1.666663 2 2
696
- 688 76.0174 -6.897691 -7.345090 1 1
697
- 689 76.0522 -2.945344 -2.403212 2 2
698
- 690 75.9420 -2.762531 -2.952115 1 2
699
- 691 75.8321 -3.804464 -3.468500 2 1
700
- 692 75.8671 -1.575988 -2.679235 1 1
701
- 693 75.9019 -4.657864 -4.855135 1 1
702
- 694 75.9366 -3.533150 -3.193708 2 2
703
- 695 75.9712 -5.145913 -5.545474 1 1
704
- 696 75.8621 -4.839345 -4.419505 2 1
705
- 697 75.8967 -2.728415 -2.774689 1 1
706
- 698 75.9312 -4.641103 -4.158267 2 2
707
- 699 75.9657 -3.709432 -5.439933 1 1
708
- 700 76.0000 -3.410856 -2.572101 2 2
709
- 701 75.8916 -2.185054 -1.777430 2 1
710
- 702 75.7835 -4.288454 -4.772275 1 2
711
- 703 75.6757 -3.923208 -3.295961 2 1
712
- 704 75.7102 -1.683617 -2.010596 1 1
713
- 705 75.7447 -2.718125 -2.249116 2 2
714
- 706 75.7790 -2.909504 -2.896289 2 2
715
- 707 75.6719 -5.091342 -4.904042 2 1
716
- 708 75.7062 -4.184419 -3.613837 2 2
717
- 709 75.7405 -3.906547 -3.333334 2 2
718
- 710 75.7746 -3.034245 -3.967609 1 1
719
- 711 75.8087 -2.722308 -2.318383 2 2
720
- 712 75.7022 -2.867235 -4.041706 1 2
721
- 713 75.5961 -3.472218 -3.622409 1 2
722
- 714 75.6303 -3.047014 -2.912795 2 2
723
- 715 75.6643 -5.558347 -4.331776 2 2
724
- 716 75.6983 -4.467770 -3.683494 2 2
725
- 717 75.7322 -2.473203 -0.536250 2 2
726
- 718 75.7660 -2.853095 -3.630617 1 1
727
- 719 75.6606 -5.256259 -4.886644 2 1
728
- 720 75.5556 -1.049066 -4.586788 1 2
729
- 721 75.4508 -4.766860 -5.467299 1 2
730
- 722 75.4848 -2.449831 -2.493209 1 1
731
- 723 75.5187 -2.881475 -3.017959 1 1
732
- 724 75.4144 -3.195180 -3.701813 1 2
733
- 725 75.4483 -2.828868 -2.715412 2 2
734
- 726 75.3444 -4.150347 -4.310081 1 2
735
- 727 75.3783 -3.331345 -2.568950 2 2
736
- 728 75.2747 -4.595504 -4.509327 2 1
737
- 729 75.3086 -2.263471 -2.993543 1 1
738
- 730 75.2055 -3.774438 -3.531267 2 1
739
- 731 75.2394 -3.604242 -4.055017 1 1
740
- 732 75.2732 -3.216992 -2.383692 2 2
741
- 733 75.3070 -4.221098 -3.857611 2 2
742
- 734 75.3406 -0.098407 -7.968916 1 1
743
- 735 75.3741 -2.725444 -2.755885 1 1
744
- 736 75.4076 -3.311194 -4.124228 1 1
745
- 737 75.4410 -4.109104 -4.357525 1 1
746
- 738 75.4743 -4.312621 -4.170076 2 2
747
- 739 75.5074 -0.518089 -9.303288 1 1
748
- 740 75.5405 -4.238755 -3.289378 2 2
749
- 741 75.4386 -4.259760 -4.102266 2 1
750
- 742 75.4717 -2.124413 -1.621954 2 2
751
- 743 75.5047 -3.470392 -3.354162 2 2
752
- 744 75.5376 -4.945076 -3.981754 2 2
753
- 745 75.5705 -4.169211 -5.022873 1 1
754
- 746 75.6032 -5.285230 -5.694309 1 1
755
- 747 75.5020 -5.355791 -4.841535 2 1
756
- 748 75.5348 -3.317398 -3.239745 2 2
757
- 749 75.5674 -3.280246 -3.692564 1 1
758
- 750 75.6000 -1.942605 -9.303097 1 1
759
 
760
- Final Winogrande score(750 tasks): 75.6000 +/- 1.5693
 
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q3_K_L.gguf (version GGUF V3 (latest))
4
 
5
+ Final Winogrande score(750 tasks): 72.8000 +/- 1.6260
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
 
7
+ llama_perf_context_print: load time = 308.84 ms
8
+ llama_perf_context_print: prompt eval time = 95382.87 ms / 22219 tokens ( 4.29 ms per token, 232.95 tokens per second)
9
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
10
+ llama_perf_context_print: total time = 96563.37 ms / 22220 tokens
11
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-q3_k_m.arc CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 869 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 869 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 100.00000000
13
- 3 100.00000000
14
- 4 75.00000000
15
- 5 80.00000000
16
- 6 83.33333333
17
- 7 85.71428571
18
- 8 87.50000000
19
- 9 77.77777778
20
- 10 80.00000000
21
- 11 72.72727273
22
- 12 75.00000000
23
- 13 69.23076923
24
- 14 71.42857143
25
- 15 73.33333333
26
- 16 75.00000000
27
- 17 76.47058824
28
- 18 72.22222222
29
- 19 73.68421053
30
- 20 70.00000000
31
- 21 71.42857143
32
- 22 72.72727273
33
- 23 69.56521739
34
- 24 70.83333333
35
- 25 68.00000000
36
- 26 65.38461538
37
- 27 66.66666667
38
- 28 64.28571429
39
- 29 65.51724138
40
- 30 66.66666667
41
- 31 64.51612903
42
- 32 65.62500000
43
- 33 63.63636364
44
- 34 61.76470588
45
- 35 62.85714286
46
- 36 61.11111111
47
- 37 62.16216216
48
- 38 60.52631579
49
- 39 61.53846154
50
- 40 62.50000000
51
- 41 63.41463415
52
- 42 61.90476190
53
- 43 60.46511628
54
- 44 59.09090909
55
- 45 60.00000000
56
- 46 60.86956522
57
- 47 61.70212766
58
- 48 62.50000000
59
- 49 63.26530612
60
- 50 62.00000000
61
- 51 60.78431373
62
- 52 61.53846154
63
- 53 60.37735849
64
- 54 59.25925926
65
- 55 58.18181818
66
- 56 58.92857143
67
- 57 59.64912281
68
- 58 60.34482759
69
- 59 61.01694915
70
- 60 61.66666667
71
- 61 62.29508197
72
- 62 61.29032258
73
- 63 61.90476190
74
- 64 62.50000000
75
- 65 61.53846154
76
- 66 62.12121212
77
- 67 62.68656716
78
- 68 63.23529412
79
- 69 63.76811594
80
- 70 62.85714286
81
- 71 61.97183099
82
- 72 61.11111111
83
- 73 61.64383562
84
- 74 62.16216216
85
- 75 62.66666667
86
- 76 63.15789474
87
- 77 63.63636364
88
- 78 64.10256410
89
- 79 63.29113924
90
- 80 63.75000000
91
- 81 62.96296296
92
- 82 63.41463415
93
- 83 62.65060241
94
- 84 63.09523810
95
- 85 63.52941176
96
- 86 63.95348837
97
- 87 63.21839080
98
- 88 63.63636364
99
- 89 64.04494382
100
- 90 64.44444444
101
- 91 64.83516484
102
- 92 65.21739130
103
- 93 64.51612903
104
- 94 63.82978723
105
- 95 64.21052632
106
- 96 64.58333333
107
- 97 63.91752577
108
- 98 63.26530612
109
- 99 63.63636364
110
- 100 64.00000000
111
- 101 63.36633663
112
- 102 63.72549020
113
- 103 63.10679612
114
- 104 62.50000000
115
- 105 62.85714286
116
- 106 62.26415094
117
- 107 62.61682243
118
- 108 62.03703704
119
- 109 61.46788991
120
- 110 61.81818182
121
- 111 62.16216216
122
- 112 62.50000000
123
- 113 61.94690265
124
- 114 62.28070175
125
- 115 62.60869565
126
- 116 62.93103448
127
- 117 63.24786325
128
- 118 63.55932203
129
- 119 63.02521008
130
- 120 63.33333333
131
- 121 62.80991736
132
- 122 63.11475410
133
- 123 63.41463415
134
- 124 62.90322581
135
- 125 63.20000000
136
- 126 63.49206349
137
- 127 63.77952756
138
- 128 64.06250000
139
- 129 63.56589147
140
- 130 63.07692308
141
- 131 63.35877863
142
- 132 63.63636364
143
- 133 63.90977444
144
- 134 63.43283582
145
- 135 62.96296296
146
- 136 62.50000000
147
- 137 62.77372263
148
- 138 63.04347826
149
- 139 63.30935252
150
- 140 63.57142857
151
- 141 63.12056738
152
- 142 63.38028169
153
- 143 63.63636364
154
- 144 63.88888889
155
- 145 64.13793103
156
- 146 63.69863014
157
- 147 63.94557823
158
- 148 64.18918919
159
- 149 64.42953020
160
- 150 64.66666667
161
- 151 64.90066225
162
- 152 65.13157895
163
- 153 65.35947712
164
- 154 64.93506494
165
- 155 65.16129032
166
- 156 65.38461538
167
- 157 65.60509554
168
- 158 65.18987342
169
- 159 65.40880503
170
- 160 65.00000000
171
- 161 65.21739130
172
- 162 65.43209877
173
- 163 65.03067485
174
- 164 64.63414634
175
- 165 64.84848485
176
- 166 65.06024096
177
- 167 64.67065868
178
- 168 64.88095238
179
- 169 64.49704142
180
- 170 64.11764706
181
- 171 64.32748538
182
- 172 63.95348837
183
- 173 64.16184971
184
- 174 64.36781609
185
- 175 64.00000000
186
- 176 64.20454545
187
- 177 64.40677966
188
- 178 64.60674157
189
- 179 64.80446927
190
- 180 65.00000000
191
- 181 64.64088398
192
- 182 64.83516484
193
- 183 64.48087432
194
- 184 64.67391304
195
- 185 64.86486486
196
- 186 65.05376344
197
- 187 64.70588235
198
- 188 64.36170213
199
- 189 64.02116402
200
- 190 63.68421053
201
- 191 63.87434555
202
- 192 64.06250000
203
- 193 64.24870466
204
- 194 64.43298969
205
- 195 64.61538462
206
- 196 64.79591837
207
- 197 64.97461929
208
- 198 65.15151515
209
- 199 64.82412060
210
- 200 65.00000000
211
- 201 65.17412935
212
- 202 64.85148515
213
- 203 65.02463054
214
- 204 65.19607843
215
- 205 64.87804878
216
- 206 64.56310680
217
- 207 64.25120773
218
- 208 64.42307692
219
- 209 64.59330144
220
- 210 64.76190476
221
- 211 64.92890995
222
- 212 64.62264151
223
- 213 64.78873239
224
- 214 64.95327103
225
- 215 65.11627907
226
- 216 64.81481481
227
- 217 64.51612903
228
- 218 64.22018349
229
- 219 64.38356164
230
- 220 64.09090909
231
- 221 63.80090498
232
- 222 63.51351351
233
- 223 63.22869955
234
- 224 63.39285714
235
- 225 63.11111111
236
- 226 63.27433628
237
- 227 63.43612335
238
- 228 63.59649123
239
- 229 63.75545852
240
- 230 63.91304348
241
- 231 63.63636364
242
- 232 63.36206897
243
- 233 63.51931330
244
- 234 63.67521368
245
- 235 63.40425532
246
- 236 63.13559322
247
- 237 62.86919831
248
- 238 63.02521008
249
- 239 63.17991632
250
- 240 63.33333333
251
- 241 63.48547718
252
- 242 63.63636364
253
- 243 63.37448560
254
- 244 63.11475410
255
- 245 63.26530612
256
- 246 63.41463415
257
- 247 63.56275304
258
- 248 63.70967742
259
- 249 63.45381526
260
- 250 63.60000000
261
- 251 63.74501992
262
- 252 63.88888889
263
- 253 64.03162055
264
- 254 63.77952756
265
- 255 63.92156863
266
- 256 63.67187500
267
- 257 63.42412451
268
- 258 63.56589147
269
- 259 63.32046332
270
- 260 63.46153846
271
- 261 63.60153257
272
- 262 63.74045802
273
- 263 63.87832700
274
- 264 64.01515152
275
- 265 63.77358491
276
- 266 63.53383459
277
- 267 63.67041199
278
- 268 63.80597015
279
- 269 63.94052045
280
- 270 64.07407407
281
- 271 64.20664207
282
- 272 63.97058824
283
- 273 63.73626374
284
- 274 63.50364964
285
- 275 63.27272727
286
- 276 63.40579710
287
- 277 63.53790614
288
- 278 63.30935252
289
- 279 63.44086022
290
- 280 63.57142857
291
- 281 63.70106762
292
- 282 63.82978723
293
- 283 63.60424028
294
- 284 63.73239437
295
- 285 63.85964912
296
- 286 63.98601399
297
- 287 64.11149826
298
- 288 64.23611111
299
- 289 64.01384083
300
- 290 64.13793103
301
- 291 64.26116838
302
- 292 64.38356164
303
- 293 64.50511945
304
- 294 64.28571429
305
- 295 64.40677966
306
- 296 64.52702703
307
- 297 64.64646465
308
- 298 64.76510067
309
- 299 64.88294314
310
- 300 65.00000000
311
- 301 64.78405316
312
- 302 64.90066225
313
- 303 65.01650165
314
- 304 64.80263158
315
- 305 64.91803279
316
- 306 64.70588235
317
- 307 64.82084691
318
- 308 64.93506494
319
- 309 65.04854369
320
- 310 64.83870968
321
- 311 64.95176849
322
- 312 65.06410256
323
- 313 65.17571885
324
- 314 64.96815287
325
- 315 65.07936508
326
- 316 65.18987342
327
- 317 65.29968454
328
- 318 65.09433962
329
- 319 64.89028213
330
- 320 64.68750000
331
- 321 64.79750779
332
- 322 64.90683230
333
- 323 65.01547988
334
- 324 65.12345679
335
- 325 65.23076923
336
- 326 65.33742331
337
- 327 65.13761468
338
- 328 65.24390244
339
- 329 65.34954407
340
- 330 65.15151515
341
- 331 65.25679758
342
- 332 65.36144578
343
- 333 65.46546547
344
- 334 65.56886228
345
- 335 65.37313433
346
- 336 65.47619048
347
- 337 65.57863501
348
- 338 65.38461538
349
- 339 65.48672566
350
- 340 65.58823529
351
- 341 65.68914956
352
- 342 65.78947368
353
- 343 65.88921283
354
- 344 65.98837209
355
- 345 65.79710145
356
- 346 65.89595376
357
- 347 65.70605187
358
- 348 65.51724138
359
- 349 65.61604585
360
- 350 65.42857143
361
- 351 65.52706553
362
- 352 65.62500000
363
- 353 65.72237960
364
- 354 65.53672316
365
- 355 65.35211268
366
- 356 65.44943820
367
- 357 65.54621849
368
- 358 65.64245810
369
- 359 65.73816156
370
- 360 65.83333333
371
- 361 65.92797784
372
- 362 66.02209945
373
- 363 65.84022039
374
- 364 65.65934066
375
- 365 65.75342466
376
- 366 65.57377049
377
- 367 65.39509537
378
- 368 65.48913043
379
- 369 65.58265583
380
- 370 65.67567568
381
- 371 65.76819407
382
- 372 65.86021505
383
- 373 65.95174263
384
- 374 66.04278075
385
- 375 65.86666667
386
- 376 65.69148936
387
- 377 65.51724138
388
- 378 65.60846561
389
- 379 65.43535620
390
- 380 65.52631579
391
- 381 65.35433071
392
- 382 65.18324607
393
- 383 65.01305483
394
- 384 65.10416667
395
- 385 65.19480519
396
- 386 65.28497409
397
- 387 65.37467700
398
- 388 65.20618557
399
- 389 65.29562982
400
- 390 65.12820513
401
- 391 65.21739130
402
- 392 65.30612245
403
- 393 65.39440204
404
- 394 65.48223350
405
- 395 65.56962025
406
- 396 65.40404040
407
- 397 65.49118388
408
- 398 65.32663317
409
- 399 65.41353383
410
- 400 65.25000000
411
- 401 65.08728180
412
- 402 65.17412935
413
- 403 65.26054591
414
- 404 65.34653465
415
- 405 65.43209877
416
- 406 65.51724138
417
- 407 65.60196560
418
- 408 65.68627451
419
- 409 65.52567237
420
- 410 65.60975610
421
- 411 65.69343066
422
- 412 65.77669903
423
- 413 65.61743341
424
- 414 65.70048309
425
- 415 65.54216867
426
- 416 65.62500000
427
- 417 65.46762590
428
- 418 65.55023923
429
- 419 65.39379475
430
- 420 65.47619048
431
- 421 65.32066508
432
- 422 65.40284360
433
- 423 65.48463357
434
- 424 65.33018868
435
- 425 65.41176471
436
- 426 65.25821596
437
- 427 65.33957845
438
- 428 65.18691589
439
- 429 65.26806527
440
- 430 65.11627907
441
- 431 65.19721578
442
- 432 65.27777778
443
- 433 65.12702079
444
- 434 65.20737327
445
- 435 65.28735632
446
- 436 65.36697248
447
- 437 65.44622426
448
- 438 65.29680365
449
- 439 65.37585421
450
- 440 65.22727273
451
- 441 65.07936508
452
- 442 64.93212670
453
- 443 64.78555305
454
- 444 64.63963964
455
- 445 64.71910112
456
- 446 64.79820628
457
- 447 64.65324385
458
- 448 64.73214286
459
- 449 64.81069042
460
- 450 64.66666667
461
- 451 64.74501109
462
- 452 64.82300885
463
- 453 64.67991170
464
- 454 64.75770925
465
- 455 64.61538462
466
- 456 64.69298246
467
- 457 64.77024070
468
- 458 64.84716157
469
- 459 64.92374728
470
- 460 65.00000000
471
- 461 65.07592191
472
- 462 65.15151515
473
- 463 65.01079914
474
- 464 64.87068966
475
- 465 64.94623656
476
- 466 65.02145923
477
- 467 65.09635974
478
- 468 65.17094017
479
- 469 65.03198294
480
- 470 65.10638298
481
- 471 65.18046709
482
- 472 65.04237288
483
- 473 64.90486258
484
- 474 64.97890295
485
- 475 65.05263158
486
- 476 65.12605042
487
- 477 65.19916143
488
- 478 65.27196653
489
- 479 65.34446764
490
- 480 65.41666667
491
- 481 65.48856549
492
- 482 65.56016598
493
- 483 65.63146998
494
- 484 65.70247934
495
- 485 65.77319588
496
- 486 65.84362140
497
- 487 65.70841889
498
- 488 65.77868852
499
- 489 65.84867076
500
- 490 65.71428571
501
- 491 65.58044807
502
- 492 65.65040650
503
- 493 65.51724138
504
- 494 65.58704453
505
- 495 65.45454545
506
- 496 65.52419355
507
- 497 65.39235412
508
- 498 65.46184739
509
- 499 65.53106212
510
- 500 65.60000000
511
- 501 65.46906188
512
- 502 65.53784861
513
- 503 65.60636183
514
- 504 65.67460317
515
- 505 65.74257426
516
- 506 65.61264822
517
- 507 65.48323471
518
- 508 65.55118110
519
- 509 65.42239686
520
- 510 65.29411765
521
- 511 65.16634051
522
- 512 65.23437500
523
- 513 65.10721248
524
- 514 65.17509728
525
- 515 65.24271845
526
- 516 65.31007752
527
- 517 65.37717602
528
- 518 65.44401544
529
- 519 65.51059730
530
- 520 65.38461538
531
- 521 65.45105566
532
- 522 65.51724138
533
- 523 65.39196941
534
- 524 65.26717557
535
- 525 65.14285714
536
- 526 65.01901141
537
- 527 64.89563567
538
- 528 64.77272727
539
- 529 64.65028355
540
- 530 64.71698113
541
- 531 64.78342750
542
- 532 64.84962406
543
- 533 64.91557223
544
- 534 64.79400749
545
- 535 64.67289720
546
- 536 64.73880597
547
- 537 64.80446927
548
- 538 64.86988848
549
- 539 64.93506494
550
- 540 64.81481481
551
- 541 64.69500924
552
- 542 64.76014760
553
- 543 64.64088398
554
- 544 64.70588235
555
- 545 64.77064220
556
- 546 64.65201465
557
- 547 64.53382084
558
- 548 64.59854015
559
- 549 64.66302368
560
- 550 64.72727273
561
- 551 64.60980036
562
- 552 64.49275362
563
- 553 64.55696203
564
- 554 64.62093863
565
- 555 64.68468468
566
- 556 64.56834532
567
- 557 64.63195691
568
- 558 64.51612903
569
- 559 64.57960644
570
- 560 64.64285714
571
- 561 64.70588235
572
- 562 64.76868327
573
- 563 64.83126110
574
- 564 64.89361702
575
- 565 64.77876106
576
- 566 64.84098940
577
- 567 64.72663139
578
- 568 64.78873239
579
- 569 64.67486819
580
- 570 64.56140351
581
- 571 64.62346760
582
- 572 64.51048951
583
- 573 64.57242583
584
- 574 64.63414634
585
- 575 64.52173913
586
- 576 64.58333333
587
- 577 64.64471404
588
- 578 64.70588235
589
- 579 64.76683938
590
- 580 64.82758621
591
- 581 64.71600688
592
- 582 64.60481100
593
- 583 64.66552316
594
- 584 64.72602740
595
- 585 64.78632479
596
- 586 64.67576792
597
- 587 64.73594549
598
- 588 64.79591837
599
- 589 64.85568761
600
- 590 64.91525424
601
- 591 64.97461929
602
- 592 64.86486486
603
- 593 64.75548061
604
- 594 64.81481481
605
- 595 64.87394958
606
- 596 64.76510067
607
- 597 64.65661642
608
- 598 64.54849498
609
- 599 64.60767947
610
- 600 64.50000000
611
- 601 64.55906822
612
- 602 64.45182724
613
- 603 64.34494196
614
- 604 64.40397351
615
- 605 64.46280992
616
- 606 64.52145215
617
- 607 64.57990115
618
- 608 64.63815789
619
- 609 64.53201970
620
- 610 64.59016393
621
- 611 64.48445172
622
- 612 64.54248366
623
- 613 64.43719413
624
- 614 64.49511401
625
- 615 64.39024390
626
- 616 64.28571429
627
- 617 64.34359806
628
- 618 64.40129450
629
- 619 64.29725363
630
- 620 64.35483871
631
- 621 64.41223833
632
- 622 64.30868167
633
- 623 64.20545746
634
- 624 64.10256410
635
- 625 64.00000000
636
- 626 63.89776358
637
- 627 63.95534290
638
- 628 64.01273885
639
- 629 63.91096979
640
- 630 63.96825397
641
- 631 64.02535658
642
- 632 63.92405063
643
- 633 63.82306477
644
- 634 63.88012618
645
- 635 63.93700787
646
- 636 63.99371069
647
- 637 63.89324961
648
- 638 63.94984326
649
- 639 64.00625978
650
- 640 63.90625000
651
- 641 63.96255850
652
- 642 63.86292835
653
- 643 63.76360809
654
- 644 63.81987578
655
- 645 63.72093023
656
- 646 63.77708978
657
- 647 63.83307573
658
- 648 63.73456790
659
- 649 63.63636364
660
- 650 63.69230769
661
- 651 63.74807988
662
- 652 63.65030675
663
- 653 63.55283308
664
- 654 63.60856269
665
- 655 63.51145038
666
- 656 63.41463415
667
- 657 63.31811263
668
- 658 63.22188450
669
- 659 63.12594841
670
- 660 63.18181818
671
- 661 63.08623298
672
- 662 63.14199396
673
- 663 63.19758673
674
- 664 63.25301205
675
- 665 63.30827068
676
- 666 63.21321321
677
- 667 63.26836582
678
- 668 63.17365269
679
- 669 63.07922272
680
- 670 63.13432836
681
- 671 63.04023845
682
- 672 63.09523810
683
- 673 63.00148588
684
- 674 62.90801187
685
- 675 62.81481481
686
- 676 62.86982249
687
- 677 62.92466765
688
- 678 62.97935103
689
- 679 63.03387334
690
- 680 63.08823529
691
- 681 62.99559471
692
- 682 62.90322581
693
- 683 62.81112738
694
- 684 62.86549708
695
- 685 62.91970803
696
- 686 62.82798834
697
- 687 62.73653566
698
- 688 62.79069767
699
- 689 62.69956459
700
- 690 62.75362319
701
- 691 62.80752533
702
- 692 62.86127168
703
- 693 62.91486291
704
- 694 62.96829971
705
- 695 62.87769784
706
- 696 62.78735632
707
- 697 62.69727403
708
- 698 62.60744986
709
- 699 62.66094421
710
- 700 62.71428571
711
- 701 62.62482168
712
- 702 62.67806268
713
- 703 62.73115220
714
- 704 62.78409091
715
- 705 62.83687943
716
- 706 62.88951841
717
- 707 62.94200849
718
- 708 62.99435028
719
- 709 63.04654443
720
- 710 63.09859155
721
- 711 63.15049226
722
- 712 63.20224719
723
- 713 63.25385694
724
- 714 63.16526611
725
- 715 63.07692308
726
- 716 63.12849162
727
- 717 63.17991632
728
- 718 63.09192201
729
- 719 63.00417246
730
- 720 63.05555556
731
- 721 63.10679612
732
- 722 63.15789474
733
- 723 63.20885201
734
- 724 63.12154696
735
- 725 63.17241379
736
- 726 63.08539945
737
- 727 62.99862448
738
- 728 63.04945055
739
- 729 63.10013717
740
- 730 63.15068493
741
- 731 63.20109439
742
- 732 63.25136612
743
- 733 63.30150068
744
- 734 63.35149864
745
- 735 63.40136054
746
- 736 63.45108696
747
- 737 63.36499322
748
- 738 63.41463415
749
- 739 63.32882273
750
- 740 63.24324324
751
- 741 63.29284750
752
- 742 63.20754717
753
- 743 63.25706595
754
- 744 63.17204301
755
- 745 63.22147651
756
- 746 63.27077748
757
- 747 63.18607764
758
- 748 63.10160428
759
- 749 63.01735648
760
- 750 63.06666667
761
 
762
- Final result: 63.0667 +/- 1.7635
763
- Random chance: 25.0083 +/- 1.5824
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q3_K_M.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 63.1016 +/- 1.7655
6
+ Random chance: 25.0251 +/- 1.5848
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 1629.74 ms
10
+ llama_perf_context_print: prompt eval time = 172545.34 ms / 36557 tokens ( 4.72 ms per token, 211.87 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 174780.50 ms / 36558 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-q3_k_m.hsw CHANGED
@@ -1,760 +1,12 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- hellaswag_score : loaded 10042 tasks from prompt.
6
- ================================= is_spm = 0
7
- hellaswag_score : selecting 750 randomized tasks.
8
- hellaswag_score : calculating hellaswag score over selected tasks.
9
 
10
- task acc_norm
11
- 1 0.00000000
12
- 2 50.00000000
13
- 3 66.66666667
14
- 4 75.00000000
15
- 5 80.00000000
16
- 6 83.33333333
17
- 7 71.42857143
18
- 8 75.00000000
19
- 9 66.66666667
20
- 10 70.00000000
21
- 11 72.72727273
22
- 12 66.66666667
23
- 13 69.23076923
24
- 14 71.42857143
25
- 15 73.33333333
26
- 16 75.00000000
27
- 17 70.58823529
28
- 18 72.22222222
29
- 19 68.42105263
30
- 20 70.00000000
31
- 21 71.42857143
32
- 22 72.72727273
33
- 23 73.91304348
34
- 24 70.83333333
35
- 25 68.00000000
36
- 26 65.38461538
37
- 27 62.96296296
38
- 28 64.28571429
39
- 29 62.06896552
40
- 30 60.00000000
41
- 31 61.29032258
42
- 32 59.37500000
43
- 33 60.60606061
44
- 34 61.76470588
45
- 35 62.85714286
46
- 36 61.11111111
47
- 37 59.45945946
48
- 38 60.52631579
49
- 39 61.53846154
50
- 40 60.00000000
51
- 41 60.97560976
52
- 42 59.52380952
53
- 43 60.46511628
54
- 44 61.36363636
55
- 45 60.00000000
56
- 46 60.86956522
57
- 47 61.70212766
58
- 48 62.50000000
59
- 49 63.26530612
60
- 50 62.00000000
61
- 51 62.74509804
62
- 52 61.53846154
63
- 53 60.37735849
64
- 54 59.25925926
65
- 55 58.18181818
66
- 56 57.14285714
67
- 57 57.89473684
68
- 58 58.62068966
69
- 59 59.32203390
70
- 60 60.00000000
71
- 61 59.01639344
72
- 62 59.67741935
73
- 63 60.31746032
74
- 64 60.93750000
75
- 65 61.53846154
76
- 66 60.60606061
77
- 67 61.19402985
78
- 68 61.76470588
79
- 69 62.31884058
80
- 70 62.85714286
81
- 71 61.97183099
82
- 72 62.50000000
83
- 73 63.01369863
84
- 74 63.51351351
85
- 75 62.66666667
86
- 76 63.15789474
87
- 77 63.63636364
88
- 78 64.10256410
89
- 79 64.55696203
90
- 80 65.00000000
91
- 81 65.43209877
92
- 82 65.85365854
93
- 83 66.26506024
94
- 84 66.66666667
95
- 85 67.05882353
96
- 86 67.44186047
97
- 87 67.81609195
98
- 88 68.18181818
99
- 89 67.41573034
100
- 90 67.77777778
101
- 91 68.13186813
102
- 92 67.39130435
103
- 93 67.74193548
104
- 94 68.08510638
105
- 95 68.42105263
106
- 96 68.75000000
107
- 97 69.07216495
108
- 98 69.38775510
109
- 99 69.69696970
110
- 100 69.00000000
111
- 101 69.30693069
112
- 102 68.62745098
113
- 103 67.96116505
114
- 104 68.26923077
115
- 105 68.57142857
116
- 106 68.86792453
117
- 107 69.15887850
118
- 108 69.44444444
119
- 109 68.80733945
120
- 110 69.09090909
121
- 111 69.36936937
122
- 112 69.64285714
123
- 113 69.91150442
124
- 114 70.17543860
125
- 115 70.43478261
126
- 116 69.82758621
127
- 117 69.23076923
128
- 118 69.49152542
129
- 119 69.74789916
130
- 120 69.16666667
131
- 121 69.42148760
132
- 122 69.67213115
133
- 123 69.91869919
134
- 124 70.16129032
135
- 125 70.40000000
136
- 126 70.63492063
137
- 127 70.86614173
138
- 128 70.31250000
139
- 129 70.54263566
140
- 130 70.00000000
141
- 131 70.22900763
142
- 132 69.69696970
143
- 133 69.92481203
144
- 134 70.14925373
145
- 135 70.37037037
146
- 136 70.58823529
147
- 137 70.80291971
148
- 138 70.28985507
149
- 139 69.78417266
150
- 140 70.00000000
151
- 141 70.21276596
152
- 142 70.42253521
153
- 143 70.62937063
154
- 144 70.83333333
155
- 145 71.03448276
156
- 146 71.23287671
157
- 147 71.42857143
158
- 148 71.62162162
159
- 149 71.81208054
160
- 150 72.00000000
161
- 151 71.52317881
162
- 152 71.05263158
163
- 153 71.24183007
164
- 154 71.42857143
165
- 155 71.61290323
166
- 156 71.79487179
167
- 157 71.97452229
168
- 158 72.15189873
169
- 159 71.69811321
170
- 160 71.87500000
171
- 161 72.04968944
172
- 162 72.22222222
173
- 163 72.39263804
174
- 164 72.56097561
175
- 165 72.72727273
176
- 166 72.89156627
177
- 167 73.05389222
178
- 168 73.21428571
179
- 169 72.78106509
180
- 170 72.94117647
181
- 171 73.09941520
182
- 172 73.25581395
183
- 173 72.83236994
184
- 174 72.98850575
185
- 175 72.57142857
186
- 176 72.72727273
187
- 177 72.88135593
188
- 178 73.03370787
189
- 179 72.62569832
190
- 180 72.77777778
191
- 181 72.37569061
192
- 182 72.52747253
193
- 183 72.67759563
194
- 184 72.28260870
195
- 185 72.43243243
196
- 186 72.04301075
197
- 187 71.65775401
198
- 188 71.80851064
199
- 189 71.42857143
200
- 190 71.57894737
201
- 191 71.72774869
202
- 192 71.87500000
203
- 193 71.50259067
204
- 194 71.64948454
205
- 195 71.28205128
206
- 196 71.42857143
207
- 197 71.06598985
208
- 198 70.70707071
209
- 199 70.85427136
210
- 200 71.00000000
211
- 201 71.14427861
212
- 202 71.28712871
213
- 203 70.93596059
214
- 204 71.07843137
215
- 205 71.21951220
216
- 206 71.35922330
217
- 207 71.49758454
218
- 208 71.15384615
219
- 209 71.29186603
220
- 210 71.42857143
221
- 211 71.56398104
222
- 212 71.69811321
223
- 213 71.83098592
224
- 214 71.96261682
225
- 215 71.62790698
226
- 216 71.75925926
227
- 217 71.42857143
228
- 218 71.55963303
229
- 219 71.68949772
230
- 220 71.81818182
231
- 221 71.49321267
232
- 222 71.62162162
233
- 223 71.74887892
234
- 224 71.87500000
235
- 225 72.00000000
236
- 226 72.12389381
237
- 227 71.80616740
238
- 228 71.92982456
239
- 229 72.05240175
240
- 230 72.17391304
241
- 231 71.86147186
242
- 232 71.55172414
243
- 233 71.67381974
244
- 234 71.79487179
245
- 235 71.48936170
246
- 236 71.18644068
247
- 237 71.30801688
248
- 238 71.42857143
249
- 239 71.54811715
250
- 240 71.66666667
251
- 241 71.78423237
252
- 242 71.90082645
253
- 243 72.01646091
254
- 244 71.72131148
255
- 245 71.42857143
256
- 246 71.13821138
257
- 247 71.25506073
258
- 248 71.37096774
259
- 249 71.48594378
260
- 250 71.60000000
261
- 251 71.71314741
262
- 252 71.82539683
263
- 253 71.54150198
264
- 254 71.65354331
265
- 255 71.76470588
266
- 256 71.87500000
267
- 257 71.59533074
268
- 258 71.70542636
269
- 259 71.81467181
270
- 260 71.92307692
271
- 261 72.03065134
272
- 262 71.75572519
273
- 263 71.86311787
274
- 264 71.96969697
275
- 265 72.07547170
276
- 266 72.18045113
277
- 267 72.28464419
278
- 268 72.38805970
279
- 269 72.49070632
280
- 270 72.59259259
281
- 271 72.32472325
282
- 272 72.42647059
283
- 273 72.16117216
284
- 274 72.26277372
285
- 275 72.00000000
286
- 276 72.10144928
287
- 277 72.20216606
288
- 278 72.30215827
289
- 279 72.40143369
290
- 280 72.50000000
291
- 281 72.59786477
292
- 282 72.69503546
293
- 283 72.79151943
294
- 284 72.88732394
295
- 285 72.63157895
296
- 286 72.72727273
297
- 287 72.82229965
298
- 288 72.91666667
299
- 289 73.01038062
300
- 290 73.10344828
301
- 291 73.19587629
302
- 292 72.94520548
303
- 293 73.03754266
304
- 294 73.12925170
305
- 295 73.22033898
306
- 296 73.31081081
307
- 297 73.40067340
308
- 298 73.48993289
309
- 299 73.24414716
310
- 300 73.00000000
311
- 301 73.08970100
312
- 302 73.17880795
313
- 303 72.93729373
314
- 304 72.69736842
315
- 305 72.78688525
316
- 306 72.87581699
317
- 307 72.63843648
318
- 308 72.72727273
319
- 309 72.81553398
320
- 310 72.90322581
321
- 311 72.99035370
322
- 312 73.07692308
323
- 313 73.16293930
324
- 314 73.24840764
325
- 315 73.33333333
326
- 316 73.41772152
327
- 317 73.50157729
328
- 318 73.27044025
329
- 319 73.04075235
330
- 320 73.12500000
331
- 321 72.89719626
332
- 322 72.67080745
333
- 323 72.75541796
334
- 324 72.83950617
335
- 325 72.92307692
336
- 326 73.00613497
337
- 327 72.78287462
338
- 328 72.86585366
339
- 329 72.94832827
340
- 330 73.03030303
341
- 331 73.11178248
342
- 332 73.19277108
343
- 333 73.27327327
344
- 334 73.05389222
345
- 335 73.13432836
346
- 336 73.21428571
347
- 337 72.99703264
348
- 338 72.78106509
349
- 339 72.86135693
350
- 340 72.94117647
351
- 341 73.02052786
352
- 342 73.09941520
353
- 343 73.17784257
354
- 344 72.96511628
355
- 345 73.04347826
356
- 346 72.83236994
357
- 347 72.91066282
358
- 348 72.70114943
359
- 349 72.49283668
360
- 350 72.57142857
361
- 351 72.64957265
362
- 352 72.72727273
363
- 353 72.80453258
364
- 354 72.88135593
365
- 355 72.95774648
366
- 356 73.03370787
367
- 357 73.10924370
368
- 358 73.18435754
369
- 359 73.25905292
370
- 360 73.05555556
371
- 361 73.13019391
372
- 362 73.20441989
373
- 363 73.27823691
374
- 364 73.35164835
375
- 365 73.42465753
376
- 366 73.49726776
377
- 367 73.56948229
378
- 368 73.64130435
379
- 369 73.71273713
380
- 370 73.78378378
381
- 371 73.85444744
382
- 372 73.92473118
383
- 373 73.99463807
384
- 374 74.06417112
385
- 375 74.13333333
386
- 376 74.20212766
387
- 377 74.00530504
388
- 378 74.07407407
389
- 379 74.14248021
390
- 380 74.21052632
391
- 381 74.27821522
392
- 382 74.34554974
393
- 383 74.41253264
394
- 384 74.47916667
395
- 385 74.54545455
396
- 386 74.61139896
397
- 387 74.67700258
398
- 388 74.48453608
399
- 389 74.55012853
400
- 390 74.61538462
401
- 391 74.42455243
402
- 392 74.48979592
403
- 393 74.55470738
404
- 394 74.61928934
405
- 395 74.68354430
406
- 396 74.74747475
407
- 397 74.81108312
408
- 398 74.87437186
409
- 399 74.93734336
410
- 400 75.00000000
411
- 401 75.06234414
412
- 402 75.12437811
413
- 403 74.93796526
414
- 404 75.00000000
415
- 405 74.81481481
416
- 406 74.87684729
417
- 407 74.93857494
418
- 408 75.00000000
419
- 409 75.06112469
420
- 410 75.12195122
421
- 411 74.93917275
422
- 412 75.00000000
423
- 413 74.81840194
424
- 414 74.87922705
425
- 415 74.69879518
426
- 416 74.75961538
427
- 417 74.82014388
428
- 418 74.88038278
429
- 419 74.94033413
430
- 420 75.00000000
431
- 421 75.05938242
432
- 422 75.11848341
433
- 423 75.17730496
434
- 424 75.23584906
435
- 425 75.29411765
436
- 426 75.35211268
437
- 427 75.17564403
438
- 428 75.23364486
439
- 429 75.29137529
440
- 430 75.34883721
441
- 431 75.17401392
442
- 432 75.00000000
443
- 433 74.82678984
444
- 434 74.88479263
445
- 435 74.71264368
446
- 436 74.77064220
447
- 437 74.59954233
448
- 438 74.42922374
449
- 439 74.48747153
450
- 440 74.31818182
451
- 441 74.37641723
452
- 442 74.43438914
453
- 443 74.49209932
454
- 444 74.54954955
455
- 445 74.60674157
456
- 446 74.66367713
457
- 447 74.72035794
458
- 448 74.77678571
459
- 449 74.83296214
460
- 450 74.88888889
461
- 451 74.72283814
462
- 452 74.55752212
463
- 453 74.61368653
464
- 454 74.66960352
465
- 455 74.72527473
466
- 456 74.78070175
467
- 457 74.83588621
468
- 458 74.89082969
469
- 459 74.94553377
470
- 460 75.00000000
471
- 461 75.05422993
472
- 462 75.10822511
473
- 463 75.16198704
474
- 464 75.21551724
475
- 465 75.05376344
476
- 466 75.10729614
477
- 467 75.16059957
478
- 468 75.00000000
479
- 469 74.84008529
480
- 470 74.68085106
481
- 471 74.73460722
482
- 472 74.78813559
483
- 473 74.84143763
484
- 474 74.89451477
485
- 475 74.94736842
486
- 476 75.00000000
487
- 477 74.84276730
488
- 478 74.89539749
489
- 479 74.94780793
490
- 480 75.00000000
491
- 481 75.05197505
492
- 482 74.89626556
493
- 483 74.94824017
494
- 484 74.79338843
495
- 485 74.63917526
496
- 486 74.69135802
497
- 487 74.74332649
498
- 488 74.59016393
499
- 489 74.64212679
500
- 490 74.69387755
501
- 491 74.74541752
502
- 492 74.79674797
503
- 493 74.84787018
504
- 494 74.89878543
505
- 495 74.94949495
506
- 496 75.00000000
507
- 497 75.05030181
508
- 498 75.10040161
509
- 499 74.94989980
510
- 500 75.00000000
511
- 501 74.85029940
512
- 502 74.90039841
513
- 503 74.95029821
514
- 504 75.00000000
515
- 505 75.04950495
516
- 506 75.09881423
517
- 507 75.14792899
518
- 508 75.19685039
519
- 509 75.24557957
520
- 510 75.29411765
521
- 511 75.34246575
522
- 512 75.19531250
523
- 513 75.04873294
524
- 514 75.09727626
525
- 515 75.14563107
526
- 516 75.00000000
527
- 517 74.85493230
528
- 518 74.90347490
529
- 519 74.95183044
530
- 520 75.00000000
531
- 521 74.85604607
532
- 522 74.71264368
533
- 523 74.76099426
534
- 524 74.80916031
535
- 525 74.85714286
536
- 526 74.71482890
537
- 527 74.57305503
538
- 528 74.62121212
539
- 529 74.66918715
540
- 530 74.71698113
541
- 531 74.76459510
542
- 532 74.81203008
543
- 533 74.85928705
544
- 534 74.90636704
545
- 535 74.95327103
546
- 536 75.00000000
547
- 537 75.04655493
548
- 538 75.09293680
549
- 539 74.95361781
550
- 540 75.00000000
551
- 541 75.04621072
552
- 542 75.09225092
553
- 543 74.95395948
554
- 544 75.00000000
555
- 545 75.04587156
556
- 546 75.09157509
557
- 547 75.13711152
558
- 548 75.18248175
559
- 549 75.22768670
560
- 550 75.09090909
561
- 551 75.13611615
562
- 552 75.00000000
563
- 553 75.04520796
564
- 554 75.09025271
565
- 555 74.95495495
566
- 556 75.00000000
567
- 557 74.86535009
568
- 558 74.73118280
569
- 559 74.77638640
570
- 560 74.82142857
571
- 561 74.86631016
572
- 562 74.91103203
573
- 563 74.95559503
574
- 564 75.00000000
575
- 565 74.86725664
576
- 566 74.73498233
577
- 567 74.77954145
578
- 568 74.82394366
579
- 569 74.69244288
580
- 570 74.73684211
581
- 571 74.78108581
582
- 572 74.82517483
583
- 573 74.86910995
584
- 574 74.91289199
585
- 575 74.95652174
586
- 576 75.00000000
587
- 577 75.04332756
588
- 578 75.08650519
589
- 579 75.12953368
590
- 580 75.17241379
591
- 581 75.04302926
592
- 582 75.08591065
593
- 583 75.12864494
594
- 584 75.17123288
595
- 585 75.21367521
596
- 586 75.25597270
597
- 587 75.29812606
598
- 588 75.17006803
599
- 589 75.04244482
600
- 590 75.08474576
601
- 591 75.12690355
602
- 592 75.16891892
603
- 593 75.04215852
604
- 594 74.91582492
605
- 595 74.95798319
606
- 596 75.00000000
607
- 597 75.04187605
608
- 598 75.08361204
609
- 599 74.95826377
610
- 600 75.00000000
611
- 601 75.04159734
612
- 602 75.08305648
613
- 603 75.12437811
614
- 604 75.16556291
615
- 605 75.20661157
616
- 606 75.08250825
617
- 607 75.12355848
618
- 608 75.00000000
619
- 609 75.04105090
620
- 610 75.08196721
621
- 611 75.12274959
622
- 612 75.00000000
623
- 613 74.87765090
624
- 614 74.91856678
625
- 615 74.95934959
626
- 616 74.83766234
627
- 617 74.87844408
628
- 618 74.91909385
629
- 619 74.95961228
630
- 620 75.00000000
631
- 621 75.04025765
632
- 622 75.08038585
633
- 623 75.12038523
634
- 624 75.16025641
635
- 625 75.20000000
636
- 626 75.23961661
637
- 627 75.27910686
638
- 628 75.31847134
639
- 629 75.19872814
640
- 630 75.07936508
641
- 631 75.11885895
642
- 632 75.00000000
643
- 633 74.88151659
644
- 634 74.92113565
645
- 635 74.96062992
646
- 636 75.00000000
647
- 637 75.03924647
648
- 638 75.07836991
649
- 639 75.11737089
650
- 640 75.00000000
651
- 641 75.03900156
652
- 642 75.07788162
653
- 643 75.11664075
654
- 644 75.15527950
655
- 645 75.03875969
656
- 646 75.07739938
657
- 647 75.11591963
658
- 648 75.15432099
659
- 649 75.03852080
660
- 650 75.07692308
661
- 651 75.11520737
662
- 652 75.15337423
663
- 653 75.19142420
664
- 654 75.22935780
665
- 655 75.26717557
666
- 656 75.30487805
667
- 657 75.34246575
668
- 658 75.37993921
669
- 659 75.26555387
670
- 660 75.15151515
671
- 661 75.18910741
672
- 662 75.22658610
673
- 663 75.26395173
674
- 664 75.30120482
675
- 665 75.33834586
676
- 666 75.37537538
677
- 667 75.41229385
678
- 668 75.44910180
679
- 669 75.48579970
680
- 670 75.52238806
681
- 671 75.55886736
682
- 672 75.44642857
683
- 673 75.48291233
684
- 674 75.51928783
685
- 675 75.55555556
686
- 676 75.59171598
687
- 677 75.62776957
688
- 678 75.66371681
689
- 679 75.69955817
690
- 680 75.73529412
691
- 681 75.62408223
692
- 682 75.65982405
693
- 683 75.69546120
694
- 684 75.73099415
695
- 685 75.76642336
696
- 686 75.80174927
697
- 687 75.83697234
698
- 688 75.87209302
699
- 689 75.90711176
700
- 690 75.94202899
701
- 691 75.97684515
702
- 692 76.01156069
703
- 693 75.90187590
704
- 694 75.93659942
705
- 695 75.97122302
706
- 696 76.00574713
707
- 697 76.04017217
708
- 698 76.07449857
709
- 699 75.96566524
710
- 700 76.00000000
711
- 701 76.03423680
712
- 702 76.06837607
713
- 703 76.10241821
714
- 704 76.13636364
715
- 705 76.17021277
716
- 706 76.20396601
717
- 707 76.23762376
718
- 708 76.27118644
719
- 709 76.30465444
720
- 710 76.33802817
721
- 711 76.23066104
722
- 712 76.26404494
723
- 713 76.29733520
724
- 714 76.33053221
725
- 715 76.22377622
726
- 716 76.25698324
727
- 717 76.29009763
728
- 718 76.32311978
729
- 719 76.35605007
730
- 720 76.38888889
731
- 721 76.42163662
732
- 722 76.45429363
733
- 723 76.48686030
734
- 724 76.51933702
735
- 725 76.55172414
736
- 726 76.58402204
737
- 727 76.61623109
738
- 728 76.64835165
739
- 729 76.68038409
740
- 730 76.71232877
741
- 731 76.74418605
742
- 732 76.63934426
743
- 733 76.67121419
744
- 734 76.70299728
745
- 735 76.59863946
746
- 736 76.63043478
747
- 737 76.52645862
748
- 738 76.42276423
749
- 739 76.31935047
750
- 740 76.35135135
751
- 741 76.38326586
752
- 742 76.41509434
753
- 743 76.44683715
754
- 744 76.47849462
755
- 745 76.51006711
756
- 746 76.54155496
757
- 747 76.43908969
758
- 748 76.47058824
759
- 749 76.50200267
760
- 750 76.53333333
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q3_K_M.gguf (version GGUF V3 (latest))
4
 
5
+ 750 75.33333333
 
 
 
 
6
 
7
+
8
+ llama_perf_context_print: load time = 308.43 ms
9
+ llama_perf_context_print: prompt eval time = 462302.43 ms / 122058 tokens ( 3.79 ms per token, 264.02 tokens per second)
10
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
11
+ llama_perf_context_print: total time = 467959.83 ms / 122059 tokens
12
+ ggml_metal_free: deallocating
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scores/Watt-Tool-8B-q3_k_m.mmlu CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 1548 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 1548 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 50.00000000
13
- 3 33.33333333
14
- 4 50.00000000
15
- 5 40.00000000
16
- 6 33.33333333
17
- 7 42.85714286
18
- 8 50.00000000
19
- 9 44.44444444
20
- 10 50.00000000
21
- 11 45.45454545
22
- 12 50.00000000
23
- 13 46.15384615
24
- 14 42.85714286
25
- 15 46.66666667
26
- 16 50.00000000
27
- 17 52.94117647
28
- 18 50.00000000
29
- 19 52.63157895
30
- 20 55.00000000
31
- 21 57.14285714
32
- 22 54.54545455
33
- 23 52.17391304
34
- 24 54.16666667
35
- 25 56.00000000
36
- 26 53.84615385
37
- 27 55.55555556
38
- 28 57.14285714
39
- 29 58.62068966
40
- 30 60.00000000
41
- 31 61.29032258
42
- 32 62.50000000
43
- 33 60.60606061
44
- 34 61.76470588
45
- 35 62.85714286
46
- 36 63.88888889
47
- 37 62.16216216
48
- 38 60.52631579
49
- 39 58.97435897
50
- 40 57.50000000
51
- 41 56.09756098
52
- 42 54.76190476
53
- 43 55.81395349
54
- 44 54.54545455
55
- 45 53.33333333
56
- 46 52.17391304
57
- 47 51.06382979
58
- 48 52.08333333
59
- 49 51.02040816
60
- 50 50.00000000
61
- 51 49.01960784
62
- 52 48.07692308
63
- 53 47.16981132
64
- 54 46.29629630
65
- 55 47.27272727
66
- 56 48.21428571
67
- 57 49.12280702
68
- 58 50.00000000
69
- 59 50.84745763
70
- 60 50.00000000
71
- 61 50.81967213
72
- 62 50.00000000
73
- 63 49.20634921
74
- 64 48.43750000
75
- 65 47.69230769
76
- 66 46.96969697
77
- 67 46.26865672
78
- 68 45.58823529
79
- 69 44.92753623
80
- 70 45.71428571
81
- 71 46.47887324
82
- 72 45.83333333
83
- 73 45.20547945
84
- 74 45.94594595
85
- 75 46.66666667
86
- 76 47.36842105
87
- 77 48.05194805
88
- 78 48.71794872
89
- 79 48.10126582
90
- 80 48.75000000
91
- 81 49.38271605
92
- 82 48.78048780
93
- 83 49.39759036
94
- 84 50.00000000
95
- 85 49.41176471
96
- 86 48.83720930
97
- 87 49.42528736
98
- 88 50.00000000
99
- 89 49.43820225
100
- 90 48.88888889
101
- 91 48.35164835
102
- 92 47.82608696
103
- 93 47.31182796
104
- 94 46.80851064
105
- 95 46.31578947
106
- 96 46.87500000
107
- 97 47.42268041
108
- 98 46.93877551
109
- 99 47.47474747
110
- 100 47.00000000
111
- 101 47.52475248
112
- 102 47.05882353
113
- 103 46.60194175
114
- 104 47.11538462
115
- 105 47.61904762
116
- 106 48.11320755
117
- 107 47.66355140
118
- 108 47.22222222
119
- 109 46.78899083
120
- 110 47.27272727
121
- 111 47.74774775
122
- 112 48.21428571
123
- 113 47.78761062
124
- 114 48.24561404
125
- 115 48.69565217
126
- 116 49.13793103
127
- 117 49.57264957
128
- 118 49.15254237
129
- 119 49.57983193
130
- 120 49.16666667
131
- 121 49.58677686
132
- 122 49.18032787
133
- 123 48.78048780
134
- 124 49.19354839
135
- 125 49.60000000
136
- 126 49.20634921
137
- 127 48.81889764
138
- 128 48.43750000
139
- 129 48.06201550
140
- 130 47.69230769
141
- 131 47.32824427
142
- 132 47.72727273
143
- 133 47.36842105
144
- 134 47.01492537
145
- 135 46.66666667
146
- 136 47.05882353
147
- 137 46.71532847
148
- 138 46.37681159
149
- 139 46.76258993
150
- 140 47.14285714
151
- 141 47.51773050
152
- 142 47.88732394
153
- 143 48.25174825
154
- 144 47.91666667
155
- 145 47.58620690
156
- 146 47.94520548
157
- 147 47.61904762
158
- 148 47.97297297
159
- 149 47.65100671
160
- 150 47.33333333
161
- 151 47.01986755
162
- 152 47.36842105
163
- 153 47.71241830
164
- 154 47.40259740
165
- 155 47.09677419
166
- 156 46.79487179
167
- 157 47.13375796
168
- 158 47.46835443
169
- 159 47.16981132
170
- 160 47.50000000
171
- 161 47.20496894
172
- 162 46.91358025
173
- 163 46.62576687
174
- 164 46.34146341
175
- 165 46.66666667
176
- 166 46.38554217
177
- 167 46.70658683
178
- 168 46.42857143
179
- 169 46.15384615
180
- 170 45.88235294
181
- 171 46.19883041
182
- 172 45.93023256
183
- 173 45.66473988
184
- 174 45.40229885
185
- 175 45.14285714
186
- 176 45.45454545
187
- 177 45.19774011
188
- 178 44.94382022
189
- 179 45.25139665
190
- 180 45.00000000
191
- 181 44.75138122
192
- 182 44.50549451
193
- 183 44.80874317
194
- 184 44.56521739
195
- 185 44.86486486
196
- 186 45.16129032
197
- 187 44.91978610
198
- 188 44.68085106
199
- 189 44.97354497
200
- 190 44.73684211
201
- 191 45.02617801
202
- 192 45.31250000
203
- 193 45.07772021
204
- 194 45.36082474
205
- 195 45.64102564
206
- 196 45.91836735
207
- 197 45.68527919
208
- 198 45.95959596
209
- 199 45.72864322
210
- 200 45.50000000
211
- 201 45.27363184
212
- 202 45.04950495
213
- 203 44.82758621
214
- 204 44.60784314
215
- 205 44.87804878
216
- 206 44.66019417
217
- 207 44.92753623
218
- 208 44.71153846
219
- 209 44.49760766
220
- 210 44.28571429
221
- 211 44.54976303
222
- 212 44.33962264
223
- 213 44.13145540
224
- 214 43.92523364
225
- 215 44.18604651
226
- 216 44.44444444
227
- 217 44.23963134
228
- 218 44.49541284
229
- 219 44.74885845
230
- 220 45.00000000
231
- 221 44.79638009
232
- 222 44.59459459
233
- 223 44.84304933
234
- 224 45.08928571
235
- 225 45.33333333
236
- 226 45.13274336
237
- 227 44.93392070
238
- 228 45.17543860
239
- 229 44.97816594
240
- 230 44.78260870
241
- 231 44.58874459
242
- 232 44.39655172
243
- 233 44.20600858
244
- 234 44.44444444
245
- 235 44.25531915
246
- 236 44.06779661
247
- 237 43.88185654
248
- 238 43.69747899
249
- 239 43.51464435
250
- 240 43.75000000
251
- 241 43.56846473
252
- 242 43.80165289
253
- 243 44.03292181
254
- 244 43.85245902
255
- 245 44.08163265
256
- 246 43.90243902
257
- 247 43.72469636
258
- 248 43.54838710
259
- 249 43.37349398
260
- 250 43.20000000
261
- 251 43.42629482
262
- 252 43.25396825
263
- 253 43.08300395
264
- 254 42.91338583
265
- 255 42.74509804
266
- 256 42.96875000
267
- 257 43.19066148
268
- 258 43.02325581
269
- 259 42.85714286
270
- 260 43.07692308
271
- 261 42.91187739
272
- 262 42.74809160
273
- 263 42.58555133
274
- 264 42.42424242
275
- 265 42.64150943
276
- 266 42.85714286
277
- 267 42.69662921
278
- 268 42.53731343
279
- 269 42.75092937
280
- 270 42.96296296
281
- 271 42.80442804
282
- 272 43.01470588
283
- 273 42.85714286
284
- 274 42.70072993
285
- 275 42.54545455
286
- 276 42.75362319
287
- 277 42.96028881
288
- 278 42.80575540
289
- 279 42.65232975
290
- 280 42.50000000
291
- 281 42.34875445
292
- 282 42.19858156
293
- 283 42.04946996
294
- 284 41.90140845
295
- 285 41.75438596
296
- 286 41.60839161
297
- 287 41.81184669
298
- 288 41.66666667
299
- 289 41.52249135
300
- 290 41.72413793
301
- 291 41.92439863
302
- 292 42.12328767
303
- 293 42.32081911
304
- 294 42.17687075
305
- 295 42.03389831
306
- 296 41.89189189
307
- 297 41.75084175
308
- 298 41.94630872
309
- 299 42.14046823
310
- 300 42.33333333
311
- 301 42.19269103
312
- 302 42.38410596
313
- 303 42.24422442
314
- 304 42.10526316
315
- 305 42.29508197
316
- 306 42.15686275
317
- 307 42.34527687
318
- 308 42.20779221
319
- 309 42.07119741
320
- 310 41.93548387
321
- 311 42.12218650
322
- 312 42.30769231
323
- 313 42.49201278
324
- 314 42.67515924
325
- 315 42.53968254
326
- 316 42.40506329
327
- 317 42.58675079
328
- 318 42.45283019
329
- 319 42.63322884
330
- 320 42.50000000
331
- 321 42.36760125
332
- 322 42.54658385
333
- 323 42.41486068
334
- 324 42.28395062
335
- 325 42.46153846
336
- 326 42.33128834
337
- 327 42.50764526
338
- 328 42.37804878
339
- 329 42.24924012
340
- 330 42.12121212
341
- 331 41.99395770
342
- 332 41.86746988
343
- 333 42.04204204
344
- 334 41.91616766
345
- 335 42.08955224
346
- 336 41.96428571
347
- 337 41.83976261
348
- 338 41.71597633
349
- 339 41.59292035
350
- 340 41.76470588
351
- 341 41.64222874
352
- 342 41.81286550
353
- 343 41.69096210
354
- 344 41.56976744
355
- 345 41.44927536
356
- 346 41.61849711
357
- 347 41.78674352
358
- 348 41.66666667
359
- 349 41.54727794
360
- 350 41.71428571
361
- 351 41.59544160
362
- 352 41.47727273
363
- 353 41.64305949
364
- 354 41.52542373
365
- 355 41.40845070
366
- 356 41.57303371
367
- 357 41.45658263
368
- 358 41.34078212
369
- 359 41.22562674
370
- 360 41.38888889
371
- 361 41.55124654
372
- 362 41.71270718
373
- 363 41.59779614
374
- 364 41.48351648
375
- 365 41.36986301
376
- 366 41.53005464
377
- 367 41.41689373
378
- 368 41.30434783
379
- 369 41.19241192
380
- 370 41.35135135
381
- 371 41.23989218
382
- 372 41.12903226
383
- 373 41.01876676
384
- 374 40.90909091
385
- 375 41.06666667
386
- 376 40.95744681
387
- 377 41.11405836
388
- 378 41.26984127
389
- 379 41.16094987
390
- 380 41.05263158
391
- 381 41.20734908
392
- 382 41.36125654
393
- 383 41.51436031
394
- 384 41.40625000
395
- 385 41.55844156
396
- 386 41.45077720
397
- 387 41.34366925
398
- 388 41.23711340
399
- 389 41.38817481
400
- 390 41.53846154
401
- 391 41.68797954
402
- 392 41.83673469
403
- 393 41.73027990
404
- 394 41.62436548
405
- 395 41.77215190
406
- 396 41.91919192
407
- 397 41.81360202
408
- 398 41.70854271
409
- 399 41.85463659
410
- 400 41.75000000
411
- 401 41.89526185
412
- 402 41.79104478
413
- 403 41.68734491
414
- 404 41.83168317
415
- 405 41.72839506
416
- 406 41.62561576
417
- 407 41.52334152
418
- 408 41.42156863
419
- 409 41.56479218
420
- 410 41.46341463
421
- 411 41.36253041
422
- 412 41.26213592
423
- 413 41.16222760
424
- 414 41.06280193
425
- 415 40.96385542
426
- 416 41.10576923
427
- 417 41.24700240
428
- 418 41.14832536
429
- 419 41.28878282
430
- 420 41.19047619
431
- 421 41.33016627
432
- 422 41.23222749
433
- 423 41.13475177
434
- 424 41.27358491
435
- 425 41.17647059
436
- 426 41.07981221
437
- 427 41.21779859
438
- 428 41.12149533
439
- 429 41.25874126
440
- 430 41.16279070
441
- 431 41.29930394
442
- 432 41.43518519
443
- 433 41.33949192
444
- 434 41.47465438
445
- 435 41.37931034
446
- 436 41.28440367
447
- 437 41.18993135
448
- 438 41.32420091
449
- 439 41.45785877
450
- 440 41.36363636
451
- 441 41.26984127
452
- 442 41.17647059
453
- 443 41.08352144
454
- 444 40.99099099
455
- 445 40.89887640
456
- 446 41.03139013
457
- 447 40.93959732
458
- 448 40.84821429
459
- 449 40.75723831
460
- 450 40.66666667
461
- 451 40.57649667
462
- 452 40.48672566
463
- 453 40.61810155
464
- 454 40.52863436
465
- 455 40.43956044
466
- 456 40.57017544
467
- 457 40.70021882
468
- 458 40.61135371
469
- 459 40.52287582
470
- 460 40.65217391
471
- 461 40.56399132
472
- 462 40.47619048
473
- 463 40.60475162
474
- 464 40.73275862
475
- 465 40.64516129
476
- 466 40.55793991
477
- 467 40.68522484
478
- 468 40.59829060
479
- 469 40.72494670
480
- 470 40.85106383
481
- 471 40.76433121
482
- 472 40.67796610
483
- 473 40.80338266
484
- 474 40.71729958
485
- 475 40.63157895
486
- 476 40.75630252
487
- 477 40.67085954
488
- 478 40.79497908
489
- 479 40.91858038
490
- 480 41.04166667
491
- 481 40.95634096
492
- 482 40.87136929
493
- 483 40.78674948
494
- 484 40.70247934
495
- 485 40.61855670
496
- 486 40.74074074
497
- 487 40.86242300
498
- 488 40.77868852
499
- 489 40.69529652
500
- 490 40.81632653
501
- 491 40.73319756
502
- 492 40.65040650
503
- 493 40.56795132
504
- 494 40.68825911
505
- 495 40.80808081
506
- 496 40.92741935
507
- 497 40.84507042
508
- 498 40.76305221
509
- 499 40.88176353
510
- 500 41.00000000
511
- 501 40.91816367
512
- 502 40.83665339
513
- 503 40.75546720
514
- 504 40.67460317
515
- 505 40.59405941
516
- 506 40.71146245
517
- 507 40.63116371
518
- 508 40.74803150
519
- 509 40.66797642
520
- 510 40.78431373
521
- 511 40.70450098
522
- 512 40.62500000
523
- 513 40.54580897
524
- 514 40.46692607
525
- 515 40.38834951
526
- 516 40.50387597
527
- 517 40.61895551
528
- 518 40.73359073
529
- 519 40.65510597
530
- 520 40.76923077
531
- 521 40.69097889
532
- 522 40.80459770
533
- 523 40.72657744
534
- 524 40.64885496
535
- 525 40.57142857
536
- 526 40.49429658
537
- 527 40.41745731
538
- 528 40.34090909
539
- 529 40.45368620
540
- 530 40.56603774
541
- 531 40.48964218
542
- 532 40.41353383
543
- 533 40.33771107
544
- 534 40.44943820
545
- 535 40.56074766
546
- 536 40.48507463
547
- 537 40.59590317
548
- 538 40.70631970
549
- 539 40.63079777
550
- 540 40.55555556
551
- 541 40.48059150
552
- 542 40.59040590
553
- 543 40.51565378
554
- 544 40.44117647
555
- 545 40.36697248
556
- 546 40.29304029
557
- 547 40.21937843
558
- 548 40.32846715
559
- 549 40.25500911
560
- 550 40.18181818
561
- 551 40.29038113
562
- 552 40.21739130
563
- 553 40.14466546
564
- 554 40.07220217
565
- 555 40.00000000
566
- 556 40.10791367
567
- 557 40.21543986
568
- 558 40.32258065
569
- 559 40.25044723
570
- 560 40.17857143
571
- 561 40.10695187
572
- 562 40.21352313
573
- 563 40.31971581
574
- 564 40.24822695
575
- 565 40.35398230
576
- 566 40.28268551
577
- 567 40.21164021
578
- 568 40.14084507
579
- 569 40.07029877
580
- 570 40.00000000
581
- 571 39.92994746
582
- 572 40.03496503
583
- 573 39.96509599
584
- 574 39.89547038
585
- 575 39.82608696
586
- 576 39.93055556
587
- 577 40.03466205
588
- 578 39.96539792
589
- 579 39.89637306
590
- 580 40.00000000
591
- 581 39.93115318
592
- 582 39.86254296
593
- 583 39.96569468
594
- 584 40.06849315
595
- 585 40.17094017
596
- 586 40.27303754
597
- 587 40.20442930
598
- 588 40.13605442
599
- 589 40.06791171
600
- 590 40.00000000
601
- 591 39.93231810
602
- 592 39.86486486
603
- 593 39.96627319
604
- 594 40.06734007
605
- 595 40.00000000
606
- 596 40.10067114
607
- 597 40.03350084
608
- 598 39.96655518
609
- 599 40.06677796
610
- 600 40.00000000
611
- 601 40.09983361
612
- 602 40.03322259
613
- 603 40.13266998
614
- 604 40.06622517
615
- 605 40.00000000
616
- 606 39.93399340
617
- 607 40.03294893
618
- 608 39.96710526
619
- 609 39.90147783
620
- 610 39.83606557
621
- 611 39.93453355
622
- 612 39.86928105
623
- 613 39.96737357
624
- 614 39.90228013
625
- 615 39.83739837
626
- 616 39.93506494
627
- 617 40.03241491
628
- 618 39.96763754
629
- 619 39.90306947
630
- 620 39.83870968
631
- 621 39.77455717
632
- 622 39.71061093
633
- 623 39.64686998
634
- 624 39.58333333
635
- 625 39.52000000
636
- 626 39.61661342
637
- 627 39.55342903
638
- 628 39.49044586
639
- 629 39.58664547
640
- 630 39.52380952
641
- 631 39.61965135
642
- 632 39.71518987
643
- 633 39.65244866
644
- 634 39.58990536
645
- 635 39.52755906
646
- 636 39.46540881
647
- 637 39.40345369
648
- 638 39.34169279
649
- 639 39.43661972
650
- 640 39.53125000
651
- 641 39.46957878
652
- 642 39.40809969
653
- 643 39.34681182
654
- 644 39.44099379
655
- 645 39.37984496
656
- 646 39.31888545
657
- 647 39.25811437
658
- 648 39.19753086
659
- 649 39.13713405
660
- 650 39.07692308
661
- 651 39.01689708
662
- 652 38.95705521
663
- 653 38.89739663
664
- 654 38.99082569
665
- 655 38.93129771
666
- 656 38.87195122
667
- 657 38.81278539
668
- 658 38.90577508
669
- 659 38.84673748
670
- 660 38.78787879
671
- 661 38.72919818
672
- 662 38.67069486
673
- 663 38.61236802
674
- 664 38.55421687
675
- 665 38.49624060
676
- 666 38.43843844
677
- 667 38.53073463
678
- 668 38.62275449
679
- 669 38.71449925
680
- 670 38.80597015
681
- 671 38.89716841
682
- 672 38.83928571
683
- 673 38.78157504
684
- 674 38.72403561
685
- 675 38.66666667
686
- 676 38.60946746
687
- 677 38.55243722
688
- 678 38.64306785
689
- 679 38.58615611
690
- 680 38.67647059
691
- 681 38.76651982
692
- 682 38.70967742
693
- 683 38.65300146
694
- 684 38.74269006
695
- 685 38.68613139
696
- 686 38.77551020
697
- 687 38.71906841
698
- 688 38.66279070
699
- 689 38.60667634
700
- 690 38.55072464
701
- 691 38.63965268
702
- 692 38.58381503
703
- 693 38.52813853
704
- 694 38.61671470
705
- 695 38.56115108
706
- 696 38.64942529
707
- 697 38.59397418
708
- 698 38.68194842
709
- 699 38.76967096
710
- 700 38.71428571
711
- 701 38.80171184
712
- 702 38.88888889
713
- 703 38.97581792
714
- 704 39.06250000
715
- 705 39.00709220
716
- 706 38.95184136
717
- 707 38.89674682
718
- 708 38.84180791
719
- 709 38.78702398
720
- 710 38.73239437
721
- 711 38.81856540
722
- 712 38.90449438
723
- 713 38.99018233
724
- 714 39.07563025
725
- 715 39.02097902
726
- 716 38.96648045
727
- 717 38.91213389
728
- 718 38.85793872
729
- 719 38.94297636
730
- 720 38.88888889
731
- 721 38.83495146
732
- 722 38.78116343
733
- 723 38.86583679
734
- 724 38.95027624
735
- 725 39.03448276
736
- 726 38.98071625
737
- 727 38.92709766
738
- 728 39.01098901
739
- 729 38.95747599
740
- 730 38.90410959
741
- 731 38.85088919
742
- 732 38.79781421
743
- 733 38.88130969
744
- 734 38.82833787
745
- 735 38.77551020
746
- 736 38.72282609
747
- 737 38.80597015
748
- 738 38.88888889
749
- 739 38.97158322
750
- 740 38.91891892
751
- 741 38.86639676
752
- 742 38.94878706
753
- 743 39.03095559
754
- 744 39.11290323
755
- 745 39.06040268
756
- 746 39.00804290
757
- 747 39.08969210
758
- 748 39.03743316
759
- 749 38.98531375
760
- 750 39.06666667
761
 
762
- Final result: 39.0667 +/- 1.7827
763
- Random chance: 25.0000 +/- 1.5822
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q3_K_M.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 38.8000 +/- 1.7805
6
+ Random chance: 25.0000 +/- 1.5822
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 324.30 ms
10
+ llama_perf_context_print: prompt eval time = 282248.57 ms / 70434 tokens ( 4.01 ms per token, 249.55 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 285034.26 ms / 70435 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-q3_k_m.ppx CHANGED
@@ -1,37 +1,37 @@
1
  ====== Perplexity statistics ======
2
- Mean PPL(Q) : 9.016990 ± 0.059441
3
- Mean PPL(base) : 7.534124 ± 0.048206
4
- Cor(ln(PPL(Q)), ln(PPL(base))): 97.13%
5
- Mean ln(PPL(Q)/PPL(base)) : 0.179668 ± 0.001569
6
- Mean PPL(Q)/PPL(base) : 1.196820 ± 0.001878
7
- Mean PPL(Q)-PPL(base) : 1.482867 ± 0.017057
8
 
9
  ====== KL divergence statistics ======
10
- Mean KLD: 0.157666 ± 0.000482
11
- Maximum KLD: 9.472803
12
- 99.9% KLD: 2.049311
13
- 99.0% KLD: 0.765471
14
- 99.0% KLD: 0.765471
15
- Median KLD: 0.127495
16
- 10.0% KLD: 0.007270
17
- 5.0% KLD: 0.002221
18
- 1.0% KLD: 0.000296
19
- Minimum KLD: -0.000001
20
 
21
  ====== Token probability statistics ======
22
- Mean Δp: -3.373 ± 0.030 %
23
- Maximum Δp: 68.206%
24
- 99.9% Δp: 40.367%
25
- 99.0% Δp: 24.601%
26
- 95.0% Δp: 11.563%
27
- 90.0% Δp: 5.922%
28
- 75.0% Δp: 0.342%
29
- Median Δp: -0.591%
30
- 25.0% Δp: -6.307%
31
- 10.0% Δp: -16.931%
32
- 5.0% Δp: -24.574%
33
- 1.0% Δp: -41.728%
34
- 0.1% Δp: -69.844%
35
- Minimum Δp: -98.781%
36
- RMS Δp : 11.741 ± 0.045 %
37
- Same top p: 78.308 ± 0.109 %
 
1
  ====== Perplexity statistics ======
2
+ Mean PPL(Q) : 9.855009 ± 0.061412
3
+ Mean PPL(base) : 7.237090 ± 0.045539
4
+ Cor(ln(PPL(Q)), ln(PPL(base))): 94.48%
5
+ Mean ln(PPL(Q)/PPL(base)) : 0.308761 ± 0.002082
6
+ Mean PPL(Q)/PPL(base) : 1.361736 ± 0.002836
7
+ Mean PPL(Q)-PPL(base) : 2.617919 ± 0.023684
8
 
9
  ====== KL divergence statistics ======
10
+ Mean KLD: 0.292188 ± 0.000927
11
+ Maximum KLD: 8.990653
12
+ 99.9% KLD: 3.324761
13
+ 99.0% KLD: 1.855327
14
+ 99.0% KLD: 1.855327
15
+ Median KLD: 0.215862
16
+ 10.0% KLD: 0.020007
17
+ 5.0% KLD: 0.005427
18
+ 1.0% KLD: 0.000564
19
+ Minimum KLD: 0.000000
20
 
21
  ====== Token probability statistics ======
22
+ Mean Δp: -8.052 ± 0.043 %
23
+ Maximum Δp: 83.209%
24
+ 99.9% Δp: 40.828%
25
+ 99.0% Δp: 21.532%
26
+ 95.0% Δp: 8.004%
27
+ 90.0% Δp: 2.972%
28
+ 75.0% Δp: 0.009%
29
+ Median Δp: -2.217%
30
+ 25.0% Δp: -12.532%
31
+ 10.0% Δp: -27.827%
32
+ 5.0% Δp: -40.790%
33
+ 1.0% Δp: -72.837%
34
+ 0.1% Δp: -88.971%
35
+ Minimum Δp: -99.209%
36
+ RMS Δp : 18.125 ± 0.066 %
37
+ Same top p: 74.435 ± 0.115 %
scores/Watt-Tool-8B-q3_k_m.tqa CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 817 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 817 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 50.00000000
13
- 3 33.33333333
14
- 4 25.00000000
15
- 5 20.00000000
16
- 6 16.66666667
17
- 7 14.28571429
18
- 8 25.00000000
19
- 9 22.22222222
20
- 10 20.00000000
21
- 11 18.18181818
22
- 12 25.00000000
23
- 13 30.76923077
24
- 14 28.57142857
25
- 15 26.66666667
26
- 16 31.25000000
27
- 17 29.41176471
28
- 18 27.77777778
29
- 19 26.31578947
30
- 20 25.00000000
31
- 21 23.80952381
32
- 22 22.72727273
33
- 23 21.73913043
34
- 24 25.00000000
35
- 25 24.00000000
36
- 26 23.07692308
37
- 27 22.22222222
38
- 28 21.42857143
39
- 29 24.13793103
40
- 30 23.33333333
41
- 31 25.80645161
42
- 32 28.12500000
43
- 33 30.30303030
44
- 34 29.41176471
45
- 35 28.57142857
46
- 36 27.77777778
47
- 37 29.72972973
48
- 38 28.94736842
49
- 39 30.76923077
50
- 40 30.00000000
51
- 41 29.26829268
52
- 42 28.57142857
53
- 43 30.23255814
54
- 44 29.54545455
55
- 45 31.11111111
56
- 46 30.43478261
57
- 47 31.91489362
58
- 48 33.33333333
59
- 49 32.65306122
60
- 50 34.00000000
61
- 51 33.33333333
62
- 52 32.69230769
63
- 53 32.07547170
64
- 54 31.48148148
65
- 55 32.72727273
66
- 56 32.14285714
67
- 57 31.57894737
68
- 58 31.03448276
69
- 59 32.20338983
70
- 60 33.33333333
71
- 61 32.78688525
72
- 62 33.87096774
73
- 63 33.33333333
74
- 64 32.81250000
75
- 65 32.30769231
76
- 66 33.33333333
77
- 67 32.83582090
78
- 68 32.35294118
79
- 69 33.33333333
80
- 70 32.85714286
81
- 71 32.39436620
82
- 72 33.33333333
83
- 73 32.87671233
84
- 74 32.43243243
85
- 75 32.00000000
86
- 76 32.89473684
87
- 77 32.46753247
88
- 78 32.05128205
89
- 79 31.64556962
90
- 80 32.50000000
91
- 81 32.09876543
92
- 82 32.92682927
93
- 83 32.53012048
94
- 84 32.14285714
95
- 85 31.76470588
96
- 86 31.39534884
97
- 87 31.03448276
98
- 88 30.68181818
99
- 89 31.46067416
100
- 90 31.11111111
101
- 91 30.76923077
102
- 92 30.43478261
103
- 93 30.10752688
104
- 94 29.78723404
105
- 95 30.52631579
106
- 96 31.25000000
107
- 97 31.95876289
108
- 98 31.63265306
109
- 99 32.32323232
110
- 100 33.00000000
111
- 101 33.66336634
112
- 102 33.33333333
113
- 103 33.00970874
114
- 104 32.69230769
115
- 105 32.38095238
116
- 106 32.07547170
117
- 107 32.71028037
118
- 108 33.33333333
119
- 109 33.02752294
120
- 110 33.63636364
121
- 111 33.33333333
122
- 112 33.92857143
123
- 113 34.51327434
124
- 114 35.08771930
125
- 115 34.78260870
126
- 116 34.48275862
127
- 117 34.18803419
128
- 118 33.89830508
129
- 119 33.61344538
130
- 120 34.16666667
131
- 121 34.71074380
132
- 122 34.42622951
133
- 123 34.14634146
134
- 124 34.67741935
135
- 125 35.20000000
136
- 126 35.71428571
137
- 127 35.43307087
138
- 128 35.15625000
139
- 129 35.65891473
140
- 130 36.15384615
141
- 131 35.87786260
142
- 132 35.60606061
143
- 133 35.33834586
144
- 134 35.07462687
145
- 135 35.55555556
146
- 136 35.29411765
147
- 137 35.76642336
148
- 138 36.23188406
149
- 139 36.69064748
150
- 140 36.42857143
151
- 141 36.17021277
152
- 142 35.91549296
153
- 143 35.66433566
154
- 144 36.11111111
155
- 145 35.86206897
156
- 146 35.61643836
157
- 147 36.05442177
158
- 148 35.81081081
159
- 149 35.57046980
160
- 150 35.33333333
161
- 151 35.76158940
162
- 152 35.52631579
163
- 153 35.29411765
164
- 154 35.06493506
165
- 155 35.48387097
166
- 156 35.89743590
167
- 157 35.66878981
168
- 158 36.07594937
169
- 159 35.84905660
170
- 160 35.62500000
171
- 161 35.40372671
172
- 162 35.80246914
173
- 163 35.58282209
174
- 164 35.36585366
175
- 165 35.75757576
176
- 166 36.14457831
177
- 167 35.92814371
178
- 168 36.30952381
179
- 169 36.09467456
180
- 170 35.88235294
181
- 171 35.67251462
182
- 172 35.46511628
183
- 173 35.26011561
184
- 174 35.63218391
185
- 175 35.42857143
186
- 176 35.79545455
187
- 177 35.59322034
188
- 178 35.39325843
189
- 179 35.75418994
190
- 180 36.11111111
191
- 181 35.91160221
192
- 182 35.71428571
193
- 183 35.51912568
194
- 184 35.32608696
195
- 185 35.13513514
196
- 186 35.48387097
197
- 187 35.29411765
198
- 188 35.10638298
199
- 189 35.44973545
200
- 190 35.26315789
201
- 191 35.07853403
202
- 192 34.89583333
203
- 193 34.71502591
204
- 194 35.05154639
205
- 195 34.87179487
206
- 196 34.69387755
207
- 197 35.02538071
208
- 198 35.35353535
209
- 199 35.17587940
210
- 200 35.50000000
211
- 201 35.32338308
212
- 202 35.64356436
213
- 203 35.46798030
214
- 204 35.29411765
215
- 205 35.60975610
216
- 206 35.43689320
217
- 207 35.74879227
218
- 208 35.57692308
219
- 209 35.40669856
220
- 210 35.23809524
221
- 211 35.07109005
222
- 212 34.90566038
223
- 213 34.74178404
224
- 214 34.57943925
225
- 215 34.41860465
226
- 216 34.72222222
227
- 217 35.02304147
228
- 218 35.32110092
229
- 219 35.15981735
230
- 220 35.00000000
231
- 221 34.84162896
232
- 222 34.68468468
233
- 223 34.52914798
234
- 224 34.82142857
235
- 225 34.66666667
236
- 226 34.95575221
237
- 227 35.24229075
238
- 228 35.08771930
239
- 229 34.93449782
240
- 230 34.78260870
241
- 231 34.63203463
242
- 232 34.48275862
243
- 233 34.33476395
244
- 234 34.61538462
245
- 235 34.46808511
246
- 236 34.74576271
247
- 237 35.02109705
248
- 238 35.29411765
249
- 239 35.14644351
250
- 240 35.00000000
251
- 241 34.85477178
252
- 242 34.71074380
253
- 243 34.97942387
254
- 244 34.83606557
255
- 245 34.69387755
256
- 246 34.55284553
257
- 247 34.81781377
258
- 248 34.67741935
259
- 249 34.93975904
260
- 250 34.80000000
261
- 251 35.05976096
262
- 252 34.92063492
263
- 253 34.78260870
264
- 254 34.64566929
265
- 255 34.50980392
266
- 256 34.37500000
267
- 257 34.24124514
268
- 258 34.10852713
269
- 259 33.97683398
270
- 260 33.84615385
271
- 261 34.09961686
272
- 262 33.96946565
273
- 263 33.84030418
274
- 264 34.09090909
275
- 265 34.33962264
276
- 266 34.21052632
277
- 267 34.08239700
278
- 268 33.95522388
279
- 269 33.82899628
280
- 270 33.70370370
281
- 271 33.57933579
282
- 272 33.45588235
283
- 273 33.69963370
284
- 274 33.57664234
285
- 275 33.45454545
286
- 276 33.33333333
287
- 277 33.21299639
288
- 278 33.09352518
289
- 279 32.97491039
290
- 280 33.21428571
291
- 281 33.45195730
292
- 282 33.33333333
293
- 283 33.56890459
294
- 284 33.45070423
295
- 285 33.33333333
296
- 286 33.56643357
297
- 287 33.44947735
298
- 288 33.68055556
299
- 289 33.56401384
300
- 290 33.44827586
301
- 291 33.67697595
302
- 292 33.56164384
303
- 293 33.78839590
304
- 294 33.67346939
305
- 295 33.89830508
306
- 296 34.12162162
307
- 297 34.34343434
308
- 298 34.22818792
309
- 299 34.11371237
310
- 300 34.00000000
311
- 301 33.88704319
312
- 302 33.77483444
313
- 303 33.66336634
314
- 304 33.55263158
315
- 305 33.44262295
316
- 306 33.33333333
317
- 307 33.22475570
318
- 308 33.44155844
319
- 309 33.33333333
320
- 310 33.22580645
321
- 311 33.11897106
322
- 312 33.33333333
323
- 313 33.22683706
324
- 314 33.12101911
325
- 315 33.01587302
326
- 316 32.91139241
327
- 317 32.80757098
328
- 318 32.70440252
329
- 319 32.60188088
330
- 320 32.50000000
331
- 321 32.39875389
332
- 322 32.60869565
333
- 323 32.50773994
334
- 324 32.40740741
335
- 325 32.30769231
336
- 326 32.20858896
337
- 327 32.11009174
338
- 328 32.31707317
339
- 329 32.21884498
340
- 330 32.42424242
341
- 331 32.32628399
342
- 332 32.53012048
343
- 333 32.73273273
344
- 334 32.63473054
345
- 335 32.83582090
346
- 336 33.03571429
347
- 337 32.93768546
348
- 338 32.84023669
349
- 339 33.03834808
350
- 340 32.94117647
351
- 341 33.13782991
352
- 342 33.33333333
353
- 343 33.23615160
354
- 344 33.43023256
355
- 345 33.62318841
356
- 346 33.52601156
357
- 347 33.71757925
358
- 348 33.62068966
359
- 349 33.81088825
360
- 350 33.71428571
361
- 351 33.61823362
362
- 352 33.80681818
363
- 353 33.71104816
364
- 354 33.61581921
365
- 355 33.80281690
366
- 356 33.70786517
367
- 357 33.61344538
368
- 358 33.51955307
369
- 359 33.42618384
370
- 360 33.33333333
371
- 361 33.51800554
372
- 362 33.42541436
373
- 363 33.60881543
374
- 364 33.51648352
375
- 365 33.42465753
376
- 366 33.33333333
377
- 367 33.51498638
378
- 368 33.42391304
379
- 369 33.33333333
380
- 370 33.24324324
381
- 371 33.15363881
382
- 372 33.33333333
383
- 373 33.24396783
384
- 374 33.42245989
385
- 375 33.33333333
386
- 376 33.24468085
387
- 377 33.42175066
388
- 378 33.33333333
389
- 379 33.24538259
390
- 380 33.15789474
391
- 381 33.07086614
392
- 382 32.98429319
393
- 383 32.89817232
394
- 384 32.81250000
395
- 385 32.72727273
396
- 386 32.64248705
397
- 387 32.55813953
398
- 388 32.73195876
399
- 389 32.64781491
400
- 390 32.56410256
401
- 391 32.73657289
402
- 392 32.65306122
403
- 393 32.82442748
404
- 394 32.99492386
405
- 395 32.91139241
406
- 396 33.08080808
407
- 397 33.24937028
408
- 398 33.16582915
409
- 399 33.08270677
410
- 400 33.00000000
411
- 401 32.91770574
412
- 402 33.08457711
413
- 403 33.00248139
414
- 404 32.92079208
415
- 405 32.83950617
416
- 406 32.75862069
417
- 407 32.67813268
418
- 408 32.59803922
419
- 409 32.51833741
420
- 410 32.43902439
421
- 411 32.36009732
422
- 412 32.52427184
423
- 413 32.68765133
424
- 414 32.85024155
425
- 415 32.77108434
426
- 416 32.69230769
427
- 417 32.61390887
428
- 418 32.53588517
429
- 419 32.45823389
430
- 420 32.61904762
431
- 421 32.77909739
432
- 422 32.70142180
433
- 423 32.86052009
434
- 424 32.78301887
435
- 425 32.70588235
436
- 426 32.62910798
437
- 427 32.55269321
438
- 428 32.47663551
439
- 429 32.40093240
440
- 430 32.32558140
441
- 431 32.25058005
442
- 432 32.40740741
443
- 433 32.56351039
444
- 434 32.71889401
445
- 435 32.64367816
446
- 436 32.56880734
447
- 437 32.49427918
448
- 438 32.64840183
449
- 439 32.80182232
450
- 440 32.95454545
451
- 441 32.87981859
452
- 442 32.80542986
453
- 443 32.73137698
454
- 444 32.88288288
455
- 445 32.80898876
456
- 446 32.73542601
457
- 447 32.66219239
458
- 448 32.58928571
459
- 449 32.73942094
460
- 450 32.66666667
461
- 451 32.59423503
462
- 452 32.52212389
463
- 453 32.67108168
464
- 454 32.59911894
465
- 455 32.52747253
466
- 456 32.45614035
467
- 457 32.38512035
468
- 458 32.31441048
469
- 459 32.24400871
470
- 460 32.17391304
471
- 461 32.10412148
472
- 462 32.25108225
473
- 463 32.39740821
474
- 464 32.54310345
475
- 465 32.68817204
476
- 466 32.61802575
477
- 467 32.54817987
478
- 468 32.69230769
479
- 469 32.62260128
480
- 470 32.55319149
481
- 471 32.48407643
482
- 472 32.62711864
483
- 473 32.55813953
484
- 474 32.70042194
485
- 475 32.84210526
486
- 476 32.77310924
487
- 477 32.70440252
488
- 478 32.63598326
489
- 479 32.56784969
490
- 480 32.50000000
491
- 481 32.43243243
492
- 482 32.57261411
493
- 483 32.50517598
494
- 484 32.64462810
495
- 485 32.78350515
496
- 486 32.71604938
497
- 487 32.64887064
498
- 488 32.58196721
499
- 489 32.71983640
500
- 490 32.85714286
501
- 491 32.99389002
502
- 492 33.13008130
503
- 493 33.06288032
504
- 494 33.19838057
505
- 495 33.33333333
506
- 496 33.26612903
507
- 497 33.19919517
508
- 498 33.13253012
509
- 499 33.06613226
510
- 500 33.00000000
511
- 501 33.13373253
512
- 502 33.06772908
513
- 503 33.00198807
514
- 504 33.13492063
515
- 505 33.06930693
516
- 506 33.00395257
517
- 507 33.13609467
518
- 508 33.26771654
519
- 509 33.39882122
520
- 510 33.52941176
521
- 511 33.46379648
522
- 512 33.39843750
523
- 513 33.33333333
524
- 514 33.26848249
525
- 515 33.39805825
526
- 516 33.33333333
527
- 517 33.46228240
528
- 518 33.39768340
529
- 519 33.33333333
530
- 520 33.26923077
531
- 521 33.20537428
532
- 522 33.14176245
533
- 523 33.07839388
534
- 524 33.01526718
535
- 525 32.95238095
536
- 526 32.88973384
537
- 527 33.01707780
538
- 528 32.95454545
539
- 529 33.08128544
540
- 530 33.01886792
541
- 531 32.95668550
542
- 532 32.89473684
543
- 533 33.02063790
544
- 534 32.95880150
545
- 535 32.89719626
546
- 536 32.83582090
547
- 537 32.96089385
548
- 538 32.89962825
549
- 539 32.83858998
550
- 540 32.96296296
551
- 541 32.90203327
552
- 542 32.84132841
553
- 543 32.78084715
554
- 544 32.90441176
555
- 545 32.84403670
556
- 546 32.78388278
557
- 547 32.90676417
558
- 548 33.02919708
559
- 549 32.96903461
560
- 550 32.90909091
561
- 551 32.84936479
562
- 552 32.97101449
563
- 553 33.09222423
564
- 554 33.03249097
565
- 555 32.97297297
566
- 556 32.91366906
567
- 557 32.85457810
568
- 558 32.79569892
569
- 559 32.73703041
570
- 560 32.85714286
571
- 561 32.79857398
572
- 562 32.74021352
573
- 563 32.68206039
574
- 564 32.80141844
575
- 565 32.92035398
576
- 566 33.03886926
577
- 567 32.98059965
578
- 568 33.09859155
579
- 569 33.21616872
580
- 570 33.15789474
581
- 571 33.27495622
582
- 572 33.21678322
583
- 573 33.15881326
584
- 574 33.27526132
585
- 575 33.39130435
586
- 576 33.50694444
587
- 577 33.62218371
588
- 578 33.56401384
589
- 579 33.67875648
590
- 580 33.79310345
591
- 581 33.73493976
592
- 582 33.84879725
593
- 583 33.79073756
594
- 584 33.90410959
595
- 585 33.84615385
596
- 586 33.95904437
597
- 587 33.90119250
598
- 588 33.84353741
599
- 589 33.78607810
600
- 590 33.72881356
601
- 591 33.67174281
602
- 592 33.61486486
603
- 593 33.55817875
604
- 594 33.67003367
605
- 595 33.61344538
606
- 596 33.55704698
607
- 597 33.50083752
608
- 598 33.44481605
609
- 599 33.38898164
610
- 600 33.33333333
611
- 601 33.44425957
612
- 602 33.55481728
613
- 603 33.66500829
614
- 604 33.60927152
615
- 605 33.55371901
616
- 606 33.66336634
617
- 607 33.60790774
618
- 608 33.55263158
619
- 609 33.49753695
620
- 610 33.44262295
621
- 611 33.38788871
622
- 612 33.49673203
623
- 613 33.44208809
624
- 614 33.38762215
625
- 615 33.33333333
626
- 616 33.44155844
627
- 617 33.54943274
628
- 618 33.49514563
629
- 619 33.60258481
630
- 620 33.54838710
631
- 621 33.49436393
632
- 622 33.60128617
633
- 623 33.54735152
634
- 624 33.49358974
635
- 625 33.44000000
636
- 626 33.38658147
637
- 627 33.33333333
638
- 628 33.43949045
639
- 629 33.38632750
640
- 630 33.33333333
641
- 631 33.43898574
642
- 632 33.38607595
643
- 633 33.49131122
644
- 634 33.59621451
645
- 635 33.54330709
646
- 636 33.49056604
647
- 637 33.43799058
648
- 638 33.38557994
649
- 639 33.33333333
650
- 640 33.43750000
651
- 641 33.38533541
652
- 642 33.48909657
653
- 643 33.43701400
654
- 644 33.38509317
655
- 645 33.48837209
656
- 646 33.43653251
657
- 647 33.38485317
658
- 648 33.48765432
659
- 649 33.43605547
660
- 650 33.53846154
661
- 651 33.64055300
662
- 652 33.74233129
663
- 653 33.84379786
664
- 654 33.79204893
665
- 655 33.74045802
666
- 656 33.84146341
667
- 657 33.94216134
668
- 658 33.89057751
669
- 659 33.99089530
670
- 660 33.93939394
671
- 661 33.88804841
672
- 662 33.83685801
673
- 663 33.78582202
674
- 664 33.73493976
675
- 665 33.83458647
676
- 666 33.93393393
677
- 667 33.88305847
678
- 668 33.83233533
679
- 669 33.78176383
680
- 670 33.73134328
681
- 671 33.68107303
682
- 672 33.63095238
683
- 673 33.58098068
684
- 674 33.53115727
685
- 675 33.62962963
686
- 676 33.72781065
687
- 677 33.67799114
688
- 678 33.62831858
689
- 679 33.57879234
690
- 680 33.52941176
691
- 681 33.48017621
692
- 682 33.43108504
693
- 683 33.38213763
694
- 684 33.47953216
695
- 685 33.43065693
696
- 686 33.38192420
697
- 687 33.33333333
698
- 688 33.28488372
699
- 689 33.38171263
700
- 690 33.33333333
701
- 691 33.42981187
702
- 692 33.52601156
703
- 693 33.62193362
704
- 694 33.57348703
705
- 695 33.52517986
706
- 696 33.62068966
707
- 697 33.57245337
708
- 698 33.52435530
709
- 699 33.47639485
710
- 700 33.57142857
711
- 701 33.52353780
712
- 702 33.47578348
713
- 703 33.42816501
714
- 704 33.38068182
715
- 705 33.33333333
716
- 706 33.42776204
717
- 707 33.52192362
718
- 708 33.47457627
719
- 709 33.42736248
720
- 710 33.52112676
721
- 711 33.47398031
722
- 712 33.56741573
723
- 713 33.52033661
724
- 714 33.47338936
725
- 715 33.56643357
726
- 716 33.51955307
727
- 717 33.61227336
728
- 718 33.70473538
729
- 719 33.79694019
730
- 720 33.75000000
731
- 721 33.70319001
732
- 722 33.65650970
733
- 723 33.60995851
734
- 724 33.70165746
735
- 725 33.65517241
736
- 726 33.74655647
737
- 727 33.70013755
738
- 728 33.65384615
739
- 729 33.74485597
740
- 730 33.69863014
741
- 731 33.65253078
742
- 732 33.74316940
743
- 733 33.69713506
744
- 734 33.65122616
745
- 735 33.60544218
746
- 736 33.55978261
747
- 737 33.64993216
748
- 738 33.60433604
749
- 739 33.55886333
750
- 740 33.51351351
751
- 741 33.60323887
752
- 742 33.55795148
753
- 743 33.51278600
754
- 744 33.60215054
755
- 745 33.55704698
756
- 746 33.64611260
757
- 747 33.73493976
758
- 748 33.68983957
759
- 749 33.77837116
760
- 750 33.86666667
761
 
762
- Final result: 33.8667 +/- 1.7292
763
- Random chance: 19.8992 +/- 1.4588
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q3_K_M.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 35.3846 +/- 2.6565
6
+ Random chance: 28.4838 +/- 2.5074
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 305.46 ms
10
+ llama_perf_context_print: prompt eval time = 83329.07 ms / 17627 tokens ( 4.73 ms per token, 211.53 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 84576.13 ms / 17628 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-q3_k_m.wng CHANGED
@@ -1,760 +1,11 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- winogrande_score : loaded 1267 tasks from prompt.
6
- winogrande_score : selecting 750 random tasks
7
- winogrande_score : tokenizing selected tasks
8
- winogrande_score : calculating winogrande score over selected tasks.
9
- 1 100.0000 -2.897725 -3.827254 1 1
10
- 2 100.0000 -1.690543 -1.869228 1 1
11
- 3 100.0000 -2.844032 -2.672183 2 2
12
- 4 100.0000 -4.807497 -4.850924 1 1
13
- 5 80.0000 -2.782250 -3.161129 1 2
14
- 6 66.6667 -4.917640 -4.833240 2 1
15
- 7 71.4286 -4.146572 -2.885922 2 2
16
- 8 75.0000 -3.103183 -2.410702 2 2
17
- 9 77.7778 -4.387463 -5.713109 1 1
18
- 10 80.0000 -1.814376 -1.685671 2 2
19
- 11 81.8182 -8.894917 -4.574613 2 2
20
- 12 83.3333 -2.541389 -1.685823 2 2
21
- 13 84.6154 -2.798849 -4.418661 1 1
22
- 14 85.7143 -3.214237 -3.975608 1 1
23
- 15 80.0000 -4.310763 -6.156368 1 2
24
- 16 81.2500 -4.238556 -3.938653 2 2
25
- 17 76.4706 -7.130950 -7.189742 1 2
26
- 18 77.7778 -1.536705 -3.204909 1 1
27
- 19 78.9474 -3.490679 -5.009666 1 1
28
- 20 80.0000 -2.763325 -2.943631 1 1
29
- 21 80.9524 -2.103895 -3.335546 1 1
30
- 22 77.2727 -2.458061 -2.515393 1 2
31
- 23 78.2609 -4.866615 -5.201021 1 1
32
- 24 75.0000 -2.678564 -2.492952 2 1
33
- 25 72.0000 -1.897811 -1.897952 1 2
34
- 26 73.0769 -3.514012 -4.501553 1 1
35
- 27 70.3704 -5.160467 -5.184707 1 2
36
- 28 71.4286 -4.735009 -5.168111 1 1
37
- 29 72.4138 -4.173804 -4.177564 1 1
38
- 30 73.3333 -2.028756 -1.996568 2 2
39
- 31 74.1935 -5.128066 -5.487943 1 1
40
- 32 75.0000 -4.488111 -6.342593 1 1
41
- 33 75.7576 -4.706510 -5.927580 1 1
42
- 34 76.4706 -2.609157 -2.756131 1 1
43
- 35 77.1429 -4.856724 -4.763401 2 2
44
- 36 77.7778 -3.950265 -3.811170 2 2
45
- 37 78.3784 -3.114372 -3.273225 1 1
46
- 38 76.3158 -4.093525 -4.001323 2 1
47
- 39 76.9231 -5.536810 -4.635356 2 2
48
- 40 75.0000 -5.730164 -6.119761 1 2
49
- 41 75.6098 -2.063656 -2.511051 1 1
50
- 42 76.1905 -2.376156 -3.290011 1 1
51
- 43 74.4186 -3.317755 -3.351919 1 2
52
- 44 75.0000 -2.374400 -1.756511 2 2
53
- 45 75.5556 -0.089023 -11.682919 1 1
54
- 46 76.0870 -2.617247 -2.771026 1 1
55
- 47 74.4681 -3.993054 -4.257776 1 2
56
- 48 75.0000 -4.001210 -3.866456 2 2
57
- 49 73.4694 -2.906034 -1.746085 2 1
58
- 50 72.0000 -2.778499 -2.808255 1 2
59
- 51 72.5490 -2.334736 -2.948754 1 1
60
- 52 73.0769 -3.433385 -2.294690 2 2
61
- 53 73.5849 -3.892066 -4.397990 1 1
62
- 54 74.0741 -2.573168 -2.294309 2 2
63
- 55 74.5455 -2.850085 -2.653468 2 2
64
- 56 73.2143 -3.902386 -5.096895 1 2
65
- 57 73.6842 -2.972521 -2.678819 2 2
66
- 58 74.1379 -3.614313 -4.616115 1 1
67
- 59 74.5763 -3.468675 -3.338641 2 2
68
- 60 75.0000 -4.746070 -4.550670 2 2
69
- 61 75.4098 -3.151869 -3.295594 1 1
70
- 62 75.8065 -5.706081 -6.018679 1 1
71
- 63 74.6032 -4.720275 -4.869880 1 2
72
- 64 73.4375 -3.774018 -3.716827 2 1
73
- 65 73.8462 -5.427840 -4.370616 2 2
74
- 66 74.2424 -3.532460 -3.683557 1 1
75
- 67 74.6269 -3.167706 -2.522938 2 2
76
- 68 73.5294 -5.310918 -5.383486 1 2
77
- 69 73.9130 -4.641982 -6.884668 1 1
78
- 70 74.2857 -3.780939 -3.159370 2 2
79
- 71 73.2394 -3.073308 -2.422571 2 1
80
- 72 73.6111 -3.929049 -3.621956 2 2
81
- 73 73.9726 -2.576068 -2.605942 1 1
82
- 74 74.3243 -2.170206 -2.792475 1 1
83
- 75 74.6667 -2.089546 -3.088621 1 1
84
- 76 75.0000 -2.241910 -4.703063 1 1
85
- 77 75.3247 -2.856603 -2.522949 2 2
86
- 78 75.6410 -0.951247 -1.624057 1 1
87
- 79 74.6835 -2.131562 -2.766609 1 2
88
- 80 75.0000 -2.399091 -3.820045 1 1
89
- 81 75.3086 -6.364727 -4.739333 2 2
90
- 82 74.3902 -3.359231 -3.678342 1 2
91
- 83 74.6988 -6.205238 -5.149079 2 2
92
- 84 75.0000 -2.323246 -2.187297 2 2
93
- 85 75.2941 -4.811371 -5.208209 1 1
94
- 86 74.4186 -5.936085 -5.982716 1 2
95
- 87 73.5632 -1.759453 -1.586118 2 1
96
- 88 72.7273 -5.096473 -4.400588 2 1
97
- 89 73.0337 -3.287561 -3.605354 1 1
98
- 90 73.3333 -4.477499 -3.530020 2 2
99
- 91 72.5275 -5.015818 -4.708661 2 1
100
- 92 72.8261 -5.058893 -4.488576 2 2
101
- 93 73.1183 -2.888402 -2.717331 2 2
102
- 94 73.4043 -4.896978 -3.904374 2 2
103
- 95 73.6842 -3.729714 -3.930163 1 1
104
- 96 73.9583 -1.507540 -2.723957 1 1
105
- 97 74.2268 -4.778425 -5.278249 1 1
106
- 98 74.4898 -3.954344 -3.514691 2 2
107
- 99 73.7374 -2.595232 -2.446130 2 1
108
- 100 74.0000 -5.938007 -4.765345 2 2
109
- 101 73.2673 -4.756622 -4.838780 1 2
110
- 102 73.5294 -3.573605 -3.651100 1 1
111
- 103 73.7864 -3.189832 -2.956974 2 2
112
- 104 74.0385 -7.477604 -7.125298 2 2
113
- 105 74.2857 -1.031711 -2.252507 1 1
114
- 106 74.5283 -2.768587 -2.367343 2 2
115
- 107 74.7664 -2.324363 -2.050122 2 2
116
- 108 75.0000 -4.185159 -3.266205 2 2
117
- 109 74.3119 -4.668708 -5.316619 1 2
118
- 110 74.5455 -4.801520 -3.768039 2 2
119
- 111 74.7748 -3.179949 -2.911983 2 2
120
- 112 74.1071 -3.927269 -3.206461 2 1
121
- 113 73.4513 -5.254149 -6.547683 1 2
122
- 114 73.6842 -4.127022 -5.035754 1 1
123
- 115 73.9130 -1.473416 -2.166510 1 1
124
- 116 74.1379 -3.337085 -2.853154 2 2
125
- 117 73.5043 -4.347303 -4.605264 1 2
126
- 118 73.7288 -1.264472 -2.206322 1 1
127
- 119 73.9496 -1.963544 -2.863922 1 1
128
- 120 73.3333 -3.044242 -2.744879 2 1
129
- 121 73.5537 -1.001200 -0.836096 2 2
130
- 122 73.7705 -6.479999 -5.653314 2 2
131
- 123 73.9837 -3.631125 -4.129591 1 1
132
- 124 74.1935 -2.800851 -2.535443 2 2
133
- 125 74.4000 -2.603792 -2.322835 2 2
134
- 126 74.6032 -7.214783 -6.916715 2 2
135
- 127 74.8031 -3.234049 -2.020000 2 2
136
- 128 75.0000 -1.612474 -1.761419 1 1
137
- 129 75.1938 -2.334526 -1.840281 2 2
138
- 130 75.3846 -6.293570 -5.496342 2 2
139
- 131 74.8092 -3.455200 -3.484286 1 2
140
- 132 75.0000 -2.446664 -2.025586 2 2
141
- 133 75.1880 -1.957834 -4.086278 1 1
142
- 134 75.3731 -5.175178 -4.231399 2 2
143
- 135 75.5556 -4.732149 -5.550147 1 1
144
- 136 75.7353 -3.520872 -2.601637 2 2
145
- 137 75.9124 -5.431933 -5.651632 1 1
146
- 138 75.3623 -3.991044 -4.070690 1 2
147
- 139 75.5396 -4.589154 -5.076680 1 1
148
- 140 75.7143 -1.319132 -1.061210 2 2
149
- 141 75.8865 -5.127006 -3.950914 2 2
150
- 142 76.0563 -3.170365 -2.446596 2 2
151
- 143 76.2238 -4.406656 -7.315377 1 1
152
- 144 76.3889 -2.384481 -2.397433 1 1
153
- 145 75.8621 -1.253135 -14.945360 1 2
154
- 146 75.3425 -3.752378 -3.946097 1 2
155
- 147 75.5102 -4.881278 -4.061573 2 2
156
- 148 75.6757 -1.846693 -2.512559 1 1
157
- 149 75.8389 -3.767838 -3.561381 2 2
158
- 150 76.0000 -4.355097 -3.448127 2 2
159
- 151 75.4967 -1.906095 -1.826105 2 1
160
- 152 75.0000 -4.085503 -4.352087 1 2
161
- 153 75.1634 -2.656243 -3.334741 1 1
162
- 154 75.3247 -3.583132 -3.852615 1 1
163
- 155 74.8387 -2.533749 -2.750899 1 2
164
- 156 75.0000 -2.022381 -2.277342 1 1
165
- 157 75.1592 -2.860507 -4.012241 1 1
166
- 158 75.3165 -4.603150 -3.987932 2 2
167
- 159 75.4717 -3.502773 -4.880404 1 1
168
- 160 75.6250 -3.687972 -3.450878 2 2
169
- 161 75.7764 -1.770114 -1.985009 1 1
170
- 162 75.3086 -0.075492 -12.377515 1 2
171
- 163 75.4601 -2.278977 -2.556861 1 1
172
- 164 75.6098 -5.078033 -3.787768 2 2
173
- 165 75.1515 -3.946843 -3.960487 1 2
174
- 166 75.3012 -5.012219 -4.331424 2 2
175
- 167 74.8503 -4.765196 -4.490568 2 1
176
- 168 75.0000 -4.367093 -3.683910 2 2
177
- 169 74.5562 -4.191163 -4.834155 1 2
178
- 170 74.7059 -2.324059 -2.886841 1 1
179
- 171 74.8538 -3.858036 -4.824582 1 1
180
- 172 75.0000 -4.026914 -5.332679 1 1
181
- 173 75.1445 -3.026491 -2.702313 2 2
182
- 174 75.2874 -5.645791 -5.250638 2 2
183
- 175 75.4286 -3.818469 -4.087249 1 1
184
- 176 75.5682 -2.303064 -3.430075 1 1
185
- 177 75.7062 -5.945425 -5.818854 2 2
186
- 178 75.2809 -2.596851 -2.330609 2 1
187
- 179 75.4190 -2.465497 -2.756752 1 1
188
- 180 75.5556 -5.891842 -5.346068 2 2
189
- 181 75.6906 -3.880733 -3.083977 2 2
190
- 182 75.8242 -4.100728 -3.203448 2 2
191
- 183 75.9563 -5.078609 -4.727487 2 2
192
- 184 76.0870 -1.462284 -2.389743 1 1
193
- 185 75.6757 -4.586955 -4.862288 1 2
194
- 186 75.8065 -3.187478 -3.880481 1 1
195
- 187 75.9358 -4.791436 -4.232651 2 2
196
- 188 75.5319 -4.400719 -3.561031 2 1
197
- 189 75.1323 -4.539312 -4.929469 1 2
198
- 190 75.2632 -4.844040 -4.573697 2 2
199
- 191 75.3927 -3.134387 -3.981680 1 1
200
- 192 75.5208 -4.880809 -3.746491 2 2
201
- 193 75.6477 -4.069970 -4.004415 2 2
202
- 194 75.7732 -1.835722 -1.507483 2 2
203
- 195 75.8974 -3.215352 -3.251155 1 1
204
- 196 75.5102 -3.170391 -3.430298 1 2
205
- 197 75.6345 -0.863489 -4.105544 1 1
206
- 198 75.7576 -4.408813 -7.074252 1 1
207
- 199 75.8794 -5.436398 -4.371861 2 2
208
- 200 76.0000 -3.174676 -2.983687 2 2
209
- 201 76.1194 -3.109800 -3.240527 1 1
210
- 202 76.2376 -9.135680 -8.473200 2 2
211
- 203 76.3547 -3.389589 -2.904379 2 2
212
- 204 76.4706 -2.212495 -3.068541 1 1
213
- 205 76.5854 -3.714743 -2.464984 2 2
214
- 206 76.2136 -4.149986 -4.489581 1 2
215
- 207 76.3285 -3.219606 -3.912286 1 1
216
- 208 76.4423 -3.425579 -2.684986 2 2
217
- 209 76.5550 -3.875303 -3.156538 2 2
218
- 210 76.6667 -6.632048 -6.479352 2 2
219
- 211 76.3033 -5.042398 -4.918278 2 1
220
- 212 76.4151 -4.885408 -4.008152 2 2
221
- 213 76.0563 -3.684798 -4.855774 1 2
222
- 214 75.7009 -3.289373 -3.234942 2 1
223
- 215 75.8140 -4.501145 -4.070863 2 2
224
- 216 75.4630 -3.227167 -3.134656 2 1
225
- 217 75.5760 -1.353760 -1.928092 1 1
226
- 218 75.6881 -2.975834 -2.539461 2 2
227
- 219 75.7991 -3.152298 -3.265240 1 1
228
- 220 75.4545 -3.845394 -4.322571 1 2
229
- 221 75.5656 -4.363400 -8.814429 1 1
230
- 222 75.2252 -0.411702 -9.673657 1 2
231
- 223 75.3363 -4.119713 -3.275419 2 2
232
- 224 75.4464 -6.079876 -5.016834 2 2
233
- 225 75.1111 -3.058616 -3.243449 1 2
234
- 226 75.2212 -3.045366 -3.828146 1 1
235
- 227 74.8899 -3.079530 -3.072038 2 1
236
- 228 75.0000 -1.925561 -2.782117 1 1
237
- 229 75.1092 -3.194052 -3.022566 2 2
238
- 230 75.2174 -3.201863 -3.664705 1 1
239
- 231 75.3247 -2.600854 -2.779402 1 1
240
- 232 75.4310 -2.079968 -1.738300 2 2
241
- 233 75.5365 -1.327069 -1.958930 1 1
242
- 234 75.6410 -3.558593 -3.204527 2 2
243
- 235 75.7447 -3.370840 -3.120571 2 2
244
- 236 75.4237 -1.694526 -1.400921 2 1
245
- 237 75.5274 -2.544817 -2.764943 1 1
246
- 238 75.6303 -2.697994 -2.438376 2 2
247
- 239 75.3138 -2.640775 -2.696301 1 2
248
- 240 75.4167 -7.218674 -7.132032 2 2
249
- 241 75.1037 -3.567902 -3.919764 1 2
250
- 242 75.2066 -5.735713 -5.402500 2 2
251
- 243 74.8971 -4.238389 -3.966416 2 1
252
- 244 75.0000 -1.951082 -4.013280 1 1
253
- 245 75.1020 -4.234424 -4.764339 1 1
254
- 246 75.2033 -2.453883 -2.955570 1 1
255
- 247 75.3036 -3.262149 -3.174009 2 2
256
- 248 75.4032 -3.394363 -3.088863 2 2
257
- 249 75.5020 -3.040092 -2.197269 2 2
258
- 250 75.6000 -2.544877 -3.118218 1 1
259
- 251 75.6972 -1.422523 -2.199083 1 1
260
- 252 75.7937 -7.329779 -6.411672 2 2
261
- 253 75.4941 -5.612631 -5.329021 2 1
262
- 254 75.5906 -4.652215 -4.274127 2 2
263
- 255 75.6863 -3.262003 -4.098151 1 1
264
- 256 75.3906 -2.971432 -2.920233 2 1
265
- 257 75.0973 -2.660574 -2.106148 2 1
266
- 258 74.8062 -3.789223 -4.025429 1 2
267
- 259 74.9035 -5.155651 -5.443937 1 1
268
- 260 74.6154 -6.014371 -5.722334 2 1
269
- 261 74.7126 -3.711083 -3.662330 2 2
270
- 262 74.8092 -4.847834 -4.444467 2 2
271
- 263 74.9049 -3.498586 -4.151182 1 1
272
- 264 75.0000 -3.919844 -3.407949 2 2
273
- 265 75.0943 -1.174093 -2.155209 1 1
274
- 266 75.1880 -5.291402 -6.416101 1 1
275
- 267 74.9064 -3.792883 -3.458554 2 1
276
- 268 75.0000 -2.931404 -2.936107 1 1
277
- 269 75.0929 -3.102965 -2.916412 2 2
278
- 270 75.1852 -2.592316 -3.406011 1 1
279
- 271 75.2768 -3.120715 -2.824957 2 2
280
- 272 75.3676 -4.011140 -3.617922 2 2
281
- 273 75.4579 -3.363437 -3.292441 2 2
282
- 274 75.5474 -3.547414 -3.443841 2 2
283
- 275 75.6364 -2.731005 -2.919553 1 1
284
- 276 75.3623 -2.632166 -2.503979 2 1
285
- 277 75.4513 -2.668025 -3.288729 1 1
286
- 278 75.5396 -2.980398 -3.072336 1 1
287
- 279 75.6272 -4.666087 -2.804567 2 2
288
- 280 75.7143 -3.793388 -3.246049 2 2
289
- 281 75.8007 -2.515092 -1.713001 2 2
290
- 282 75.8865 -4.139671 -3.517613 2 2
291
- 283 75.9717 -3.412084 -3.365283 2 2
292
- 284 76.0563 -2.828299 -3.648053 1 1
293
- 285 76.1404 -3.129467 -5.537748 1 1
294
- 286 76.2238 -3.773650 -3.928455 1 1
295
- 287 76.3066 -3.145593 -4.766558 1 1
296
- 288 76.3889 -3.699560 -3.142471 2 2
297
- 289 76.4706 -2.385787 -3.018749 1 1
298
- 290 76.5517 -4.080774 -4.598639 1 1
299
- 291 76.6323 -2.259461 -2.801511 1 1
300
- 292 76.7123 -2.891855 -2.192483 2 2
301
- 293 76.7918 -3.915315 -3.256701 2 2
302
- 294 76.5306 -3.811965 -3.192056 2 1
303
- 295 76.2712 -2.279938 -2.148552 2 1
304
- 296 76.3514 -5.139627 -4.927138 2 2
305
- 297 76.0943 -1.529596 -1.940401 1 2
306
- 298 76.1745 -5.352979 -4.952059 2 2
307
- 299 76.2542 -2.404314 -2.244935 2 2
308
- 300 76.3333 -2.090508 -2.513633 1 1
309
- 301 76.4120 -4.140805 -4.482100 1 1
310
- 302 76.4901 -4.038043 -3.958821 2 2
311
- 303 76.2376 -8.958722 -8.580853 2 1
312
- 304 76.3158 -4.488789 -3.173470 2 2
313
- 305 76.3934 -3.541176 -3.335291 2 2
314
- 306 76.4706 -2.271872 -1.960553 2 2
315
- 307 76.5472 -2.706049 -3.293228 1 1
316
- 308 76.6234 -4.795634 -3.916766 2 2
317
- 309 76.6990 -4.255225 -4.744320 1 1
318
- 310 76.4516 -5.799532 -5.898750 1 2
319
- 311 76.5273 -0.576266 -4.622948 1 1
320
- 312 76.2821 -4.203097 -4.133868 2 1
321
- 313 76.3578 -4.796872 -4.695756 2 2
322
- 314 76.4331 -4.655089 -4.284960 2 2
323
- 315 76.5079 -2.065889 -3.516617 1 1
324
- 316 76.5823 -6.960261 -4.524920 2 2
325
- 317 76.6562 -4.913806 -4.989497 1 1
326
- 318 76.7296 -4.834039 -4.504354 2 2
327
- 319 76.8025 -1.402860 -1.715029 1 1
328
- 320 76.8750 -3.849400 -4.777585 1 1
329
- 321 76.6355 -2.867092 -2.891833 1 2
330
- 322 76.3975 -2.452866 -2.665157 1 2
331
- 323 76.4706 -3.631874 -4.072611 1 1
332
- 324 76.2346 -4.509383 -4.259326 2 1
333
- 325 76.3077 -5.690351 -5.397398 2 2
334
- 326 76.3804 -2.696059 -1.957798 2 2
335
- 327 76.4526 -2.171698 -2.829783 1 1
336
- 328 76.5244 -4.281621 -4.345227 1 1
337
- 329 76.5957 -3.449321 -7.745543 1 1
338
- 330 76.6667 -1.279104 -2.005857 1 1
339
- 331 76.4350 -4.055899 -3.420377 2 1
340
- 332 76.5060 -1.474170 -2.561345 1 1
341
- 333 76.5766 -3.780957 -1.425056 2 2
342
- 334 76.6467 -2.974995 -5.553373 1 1
343
- 335 76.4179 -5.470964 -5.070045 2 1
344
- 336 76.1905 -3.247934 -3.872577 1 2
345
- 337 76.2611 -2.032821 -1.568944 2 2
346
- 338 76.0355 -3.664464 -2.994153 2 1
347
- 339 75.8112 -5.446874 -5.632403 1 2
348
- 340 75.8824 -4.820940 -5.624248 1 1
349
- 341 75.9531 -3.934210 -4.915382 1 1
350
- 342 76.0234 -5.558977 -4.874511 2 2
351
- 343 76.0933 -3.145358 -3.752979 1 1
352
- 344 76.1628 -1.984483 -3.171114 1 1
353
- 345 76.2319 -0.739195 -2.930582 1 1
354
- 346 76.3006 -6.011398 -5.715317 2 2
355
- 347 76.3689 -3.360342 -2.872844 2 2
356
- 348 76.4368 -0.952340 -1.986416 1 1
357
- 349 76.2178 -3.334168 -4.564617 1 2
358
- 350 76.0000 -2.119540 -4.130608 1 2
359
- 351 76.0684 -3.082042 -3.851934 1 1
360
- 352 76.1364 -3.623686 -2.792311 2 2
361
- 353 76.2040 -2.242264 -2.351948 1 1
362
- 354 76.2712 -5.790556 -4.515013 2 2
363
- 355 76.3380 -2.808961 -2.353269 2 2
364
- 356 76.1236 -3.358327 -3.891202 1 2
365
- 357 76.1905 -4.384218 -4.216889 2 2
366
- 358 76.2570 -2.076740 -1.843544 2 2
367
- 359 76.3231 -5.515186 -3.250063 2 2
368
- 360 76.1111 -3.649798 -3.689238 1 2
369
- 361 76.1773 -3.005630 -1.487863 2 2
370
- 362 76.2431 -5.981671 -5.098458 2 2
371
- 363 76.0331 -2.764821 -2.548034 2 1
372
- 364 76.0989 -6.163332 -6.579532 1 1
373
- 365 76.1644 -5.422678 -5.506975 1 1
374
- 366 75.9563 -2.537046 -7.232204 1 2
375
- 367 76.0218 -5.126123 -5.353119 1 1
376
- 368 76.0870 -3.981869 -3.185946 2 2
377
- 369 75.8808 -3.643262 -3.981226 1 2
378
- 370 75.9459 -4.803279 -5.046257 1 1
379
- 371 76.0108 -2.173063 -3.502959 1 1
380
- 372 76.0753 -2.362288 -2.483670 1 1
381
- 373 75.8713 -3.859605 -3.277537 2 1
382
- 374 75.9358 -2.270450 -2.984803 1 1
383
- 375 76.0000 -3.384063 -2.361069 2 2
384
- 376 76.0638 -4.555815 -3.478429 2 2
385
- 377 76.1273 -4.047053 -3.874523 2 2
386
- 378 75.9259 -2.981276 -3.361326 1 2
387
- 379 75.7256 -0.693954 -2.065069 1 2
388
- 380 75.7895 -6.104425 -6.544623 1 1
389
- 381 75.8530 -10.614923 -9.420485 2 2
390
- 382 75.9162 -2.732292 -2.680295 2 2
391
- 383 75.9791 -2.050061 -1.470047 2 2
392
- 384 75.7812 -3.538453 -2.800671 2 1
393
- 385 75.5844 -0.793950 -0.749664 2 1
394
- 386 75.6477 -2.577144 -14.530323 1 1
395
- 387 75.7106 -1.597115 -2.380383 1 1
396
- 388 75.7732 -5.779859 -5.668321 2 2
397
- 389 75.8355 -2.933065 -2.809665 2 2
398
- 390 75.6410 -2.877234 -2.841124 2 1
399
- 391 75.4476 -3.996451 -4.682292 1 2
400
- 392 75.2551 -4.269889 -3.539435 2 1
401
- 393 75.0636 -6.124477 -6.160096 1 2
402
- 394 75.1269 -2.537616 -3.409718 1 1
403
- 395 75.1899 -3.247020 -3.074391 2 2
404
- 396 75.2525 -3.379508 -3.942391 1 1
405
- 397 75.3149 -6.498102 -7.020031 1 1
406
- 398 75.3769 -2.728397 -4.784961 1 1
407
- 399 75.4386 -2.907676 -3.938879 1 1
408
- 400 75.5000 -1.741263 -7.111825 1 1
409
- 401 75.5611 -3.590740 -3.919719 1 1
410
- 402 75.6219 -3.631335 -3.524250 2 2
411
- 403 75.4342 -5.801943 -6.477112 1 2
412
- 404 75.4950 -4.272924 -2.962721 2 2
413
- 405 75.5556 -3.290070 -4.302509 1 1
414
- 406 75.6158 -7.776891 -8.361012 1 1
415
- 407 75.6757 -5.091305 -5.073752 2 2
416
- 408 75.4902 -3.255212 -3.364439 1 2
417
- 409 75.5501 -3.926415 -4.649170 1 1
418
- 410 75.6098 -3.483798 -3.552831 1 1
419
- 411 75.6691 -3.706570 -5.044514 1 1
420
- 412 75.7282 -3.952944 -3.218078 2 2
421
- 413 75.7869 -3.246953 -2.454582 2 2
422
- 414 75.6039 -4.052326 -4.412036 1 2
423
- 415 75.6627 -3.787961 -3.468048 2 2
424
- 416 75.7212 -4.237107 -2.229428 2 2
425
- 417 75.7794 -4.093414 -5.211334 1 1
426
- 418 75.8373 -5.263891 -5.649123 1 1
427
- 419 75.8950 -3.590539 -3.476163 2 2
428
- 420 75.7143 -3.309603 -3.524934 1 2
429
- 421 75.7720 -4.253445 -3.930171 2 2
430
- 422 75.8294 -2.024967 -2.351722 1 1
431
- 423 75.8865 -3.328529 -3.418229 1 1
432
- 424 75.9434 -4.698667 -5.575221 1 1
433
- 425 76.0000 -5.017996 -3.035004 2 2
434
- 426 76.0563 -1.891299 -1.654888 2 2
435
- 427 76.1124 -3.714991 -4.439253 1 1
436
- 428 76.1682 -3.158558 -3.901330 1 1
437
- 429 75.9907 -3.168410 -3.197854 1 2
438
- 430 76.0465 -5.133134 -5.406090 1 1
439
- 431 76.1021 -3.888294 -5.205537 1 1
440
- 432 75.9259 -3.138740 -3.876404 1 2
441
- 433 75.9815 -2.185521 -1.888204 2 2
442
- 434 76.0369 -6.893199 -7.726317 1 1
443
- 435 76.0920 -3.799151 -3.812776 1 1
444
- 436 75.9174 -2.433829 -2.151038 2 1
445
- 437 75.9725 -4.154788 -4.031222 2 2
446
- 438 76.0274 -3.763388 -3.581155 2 2
447
- 439 76.0820 -2.803676 -3.315717 1 1
448
- 440 76.1364 -3.685161 -4.285500 1 1
449
- 441 76.1905 -3.611357 -4.818323 1 1
450
- 442 76.2443 -4.436072 -4.469254 1 1
451
- 443 76.2980 -1.781449 -3.266227 1 1
452
- 444 76.1261 -5.847820 -6.001008 1 2
453
- 445 76.1798 -1.576406 -2.330247 1 1
454
- 446 76.2332 -4.394664 -2.911426 2 2
455
- 447 76.0626 -3.169931 -2.815287 2 1
456
- 448 76.1161 -5.011687 -4.670306 2 2
457
- 449 76.1693 -2.507846 -2.274910 2 2
458
- 450 76.2222 -5.972946 -5.131619 2 2
459
- 451 76.2749 -3.769641 -3.666932 2 2
460
- 452 76.1062 -4.273323 -4.057246 2 1
461
- 453 76.1589 -4.248158 -3.336632 2 2
462
- 454 76.2115 -3.756472 -4.107255 1 1
463
- 455 76.0440 -2.151390 -10.775732 1 2
464
- 456 76.0965 -3.782016 -3.749278 2 2
465
- 457 75.9300 -8.671247 -8.347865 2 1
466
- 458 75.9825 -3.456632 -2.467050 2 2
467
- 459 76.0349 -4.026151 -3.816900 2 2
468
- 460 76.0870 -0.766373 -9.377628 1 1
469
- 461 76.1388 -2.320868 -3.614206 1 1
470
- 462 75.9740 -5.615784 -5.397568 2 1
471
- 463 76.0259 -1.404386 -1.638056 1 1
472
- 464 76.0776 -2.075418 -2.565052 1 1
473
- 465 76.1290 -3.576914 -2.189496 2 2
474
- 466 76.1803 -5.453683 -5.415705 2 2
475
- 467 76.0171 -5.196224 -4.910501 2 1
476
- 468 76.0684 -4.306496 -3.239837 2 2
477
- 469 76.1194 -1.286383 -1.570482 1 1
478
- 470 76.1702 -2.014356 -1.497166 2 2
479
- 471 76.2208 -2.537866 -3.052447 1 1
480
- 472 76.0593 -3.577091 -3.641531 1 2
481
- 473 76.1099 -3.772007 -3.078128 2 2
482
- 474 76.1603 -5.115432 -6.090123 1 1
483
- 475 76.2105 -2.627730 -2.230556 2 2
484
- 476 76.0504 -4.382669 -4.579823 1 2
485
- 477 75.8910 -4.253557 -4.550361 1 2
486
- 478 75.9414 -2.434344 -2.594521 1 1
487
- 479 75.7829 -2.995286 -2.954063 2 1
488
- 480 75.8333 -6.049005 -5.253458 2 2
489
- 481 75.8836 -2.118574 -2.416438 1 1
490
- 482 75.7261 -4.494975 -5.212556 1 2
491
- 483 75.7764 -3.950930 -4.545393 1 1
492
- 484 75.6198 -2.765139 -2.190529 2 1
493
- 485 75.6701 -1.313322 -1.837549 1 1
494
- 486 75.7202 -2.629016 -3.358057 1 1
495
- 487 75.7700 -3.607910 -3.204731 2 2
496
- 488 75.8197 -4.186873 -3.056393 2 2
497
- 489 75.8691 -2.033813 -2.683666 1 1
498
- 490 75.9184 -2.629680 -2.968053 1 1
499
- 491 75.9674 -5.085482 -4.217232 2 2
500
- 492 76.0163 -5.146113 -4.237003 2 2
501
- 493 76.0649 -1.439693 -2.356155 1 1
502
- 494 76.1134 -3.445274 -2.734270 2 2
503
- 495 76.1616 -4.009191 -4.046856 1 1
504
- 496 76.2097 -3.331941 -3.098896 2 2
505
- 497 76.2575 -4.948798 -5.162479 1 1
506
- 498 76.1044 -1.640163 -1.995278 1 2
507
- 499 76.1523 -2.798473 -3.364216 1 1
508
- 500 76.2000 -5.778918 -4.956643 2 2
509
- 501 76.2475 -2.735411 -2.857680 1 1
510
- 502 76.2948 -1.280262 -2.536473 1 1
511
- 503 76.3419 -3.805073 -3.391152 2 2
512
- 504 76.3889 -2.826063 -2.977212 1 1
513
- 505 76.4356 -7.119470 -6.961491 2 2
514
- 506 76.4822 -4.127006 -5.049663 1 1
515
- 507 76.5286 -3.285988 -3.715587 1 1
516
- 508 76.5748 -4.894180 -2.927106 2 2
517
- 509 76.6208 -6.410851 -10.345158 1 1
518
- 510 76.6667 -3.500458 -3.504783 1 1
519
- 511 76.7123 -5.657507 -4.094260 2 2
520
- 512 76.7578 -3.093310 -3.504429 1 1
521
- 513 76.6082 -2.655087 -2.750253 1 2
522
- 514 76.4591 -4.732865 -3.813814 2 1
523
- 515 76.5049 -6.717085 -9.227280 1 1
524
- 516 76.5504 -4.510008 -4.527916 1 1
525
- 517 76.5957 -1.571497 -1.874664 1 1
526
- 518 76.6409 -2.659388 -2.633505 2 2
527
- 519 76.6859 -7.014584 -6.593971 2 2
528
- 520 76.7308 -2.632117 -2.604722 2 2
529
- 521 76.7754 -1.868933 -2.171804 1 1
530
- 522 76.8199 -3.865719 -2.387849 2 2
531
- 523 76.8642 -3.985602 -3.714757 2 2
532
- 524 76.9084 -3.996321 -4.476978 1 1
533
- 525 76.9524 -4.517378 -2.583506 2 2
534
- 526 76.9962 -1.144090 -14.055627 1 1
535
- 527 77.0398 -6.399655 -4.070200 2 2
536
- 528 77.0833 -6.081004 -5.888438 2 2
537
- 529 77.1267 -5.456785 -6.100530 1 1
538
- 530 77.1698 -4.790719 -2.996214 2 2
539
- 531 77.2128 -3.449187 -3.866183 1 1
540
- 532 77.2556 -2.113783 -3.372341 1 1
541
- 533 77.2983 -3.454821 -2.410600 2 2
542
- 534 77.3408 -2.171590 -2.745953 1 1
543
- 535 77.1963 -3.356109 -3.439282 1 2
544
- 536 77.0522 -2.438681 -2.204886 2 1
545
- 537 77.0950 -5.044393 -6.249876 1 1
546
- 538 77.1375 -2.543222 -5.183500 1 1
547
- 539 77.1800 -3.779446 -1.993794 2 2
548
- 540 77.2222 -2.814368 -2.671962 2 2
549
- 541 77.2643 -2.696660 -3.544717 1 1
550
- 542 77.1218 -3.964691 -4.642269 1 2
551
- 543 77.1639 -5.548188 -3.964163 2 2
552
- 544 77.2059 -3.427215 -3.836074 1 1
553
- 545 77.0642 -2.073215 -3.944394 1 2
554
- 546 77.1062 -2.897811 -4.529515 1 1
555
- 547 77.1481 -5.747774 -5.419506 2 2
556
- 548 77.0073 -4.341218 -4.258549 2 1
557
- 549 77.0492 -5.227015 -5.566442 1 1
558
- 550 76.9091 -2.737144 -2.766502 1 2
559
- 551 76.7695 -0.875324 -13.641369 1 2
560
- 552 76.6304 -1.959909 -1.642534 2 1
561
- 553 76.6727 -5.122007 -5.048776 2 2
562
- 554 76.7148 -4.878400 -4.836797 2 2
563
- 555 76.7568 -1.892179 -1.434232 2 2
564
- 556 76.7986 -5.300454 -5.013000 2 2
565
- 557 76.6607 -5.170732 -4.970516 2 1
566
- 558 76.7025 -1.588254 -3.126369 1 1
567
- 559 76.7442 -2.581484 -2.509268 2 2
568
- 560 76.7857 -2.270680 -2.317619 1 1
569
- 561 76.8271 -2.346809 -2.907251 1 1
570
- 562 76.8683 -0.826822 -1.955136 1 1
571
- 563 76.7318 -1.428880 -2.517457 1 2
572
- 564 76.7730 -2.808650 -3.141755 1 1
573
- 565 76.8142 -4.250589 -3.883252 2 2
574
- 566 76.8551 -4.410234 -4.547375 1 1
575
- 567 76.8959 -1.923465 -1.630184 2 2
576
- 568 76.9366 -7.650739 -3.527249 2 2
577
- 569 76.8014 -5.821078 -4.811245 2 1
578
- 570 76.8421 -3.897639 -5.075897 1 1
579
- 571 76.8827 -4.144323 -3.202256 2 2
580
- 572 76.9231 -2.361911 -2.064571 2 2
581
- 573 76.7888 -3.972851 -4.205170 1 2
582
- 574 76.8293 -3.985984 -3.835071 2 2
583
- 575 76.8696 -5.331562 -5.287053 2 2
584
- 576 76.9097 -3.604257 -3.996209 1 1
585
- 577 76.9497 -4.134508 -3.535963 2 2
586
- 578 76.9896 -2.934656 -2.797698 2 2
587
- 579 76.8566 -4.349979 -4.759905 1 2
588
- 580 76.8966 -1.983199 -2.436849 1 1
589
- 581 76.7642 -6.787226 -6.212898 2 1
590
- 582 76.8041 -5.739397 -5.548686 2 2
591
- 583 76.8439 -4.874174 -4.626549 2 2
592
- 584 76.7123 -4.962113 -4.614454 2 1
593
- 585 76.7521 -0.441924 -1.162588 1 1
594
- 586 76.7918 -3.867053 -3.931057 1 1
595
- 587 76.6610 -5.334748 -5.280060 2 1
596
- 588 76.7007 -3.657826 -2.645446 2 2
597
- 589 76.5705 -3.816307 -3.889349 1 2
598
- 590 76.6102 -3.992912 -3.234971 2 2
599
- 591 76.6497 -3.821137 -3.502542 2 2
600
- 592 76.6892 -3.080455 -3.970570 1 1
601
- 593 76.7285 -3.873529 -3.437561 2 2
602
- 594 76.5993 -4.497372 -4.219712 2 1
603
- 595 76.4706 -1.511023 -1.537643 1 2
604
- 596 76.5101 -5.102921 -4.898255 2 2
605
- 597 76.5494 -1.889139 -2.069728 1 1
606
- 598 76.5886 -3.060552 -3.547203 1 1
607
- 599 76.6277 -2.970317 -3.041126 1 1
608
- 600 76.6667 -4.190898 -4.027712 2 2
609
- 601 76.7055 -3.090207 -3.047163 2 2
610
- 602 76.7442 -2.283031 -3.929517 1 1
611
- 603 76.7828 -6.090616 -5.654686 2 2
612
- 604 76.6556 -3.033616 -3.065367 1 2
613
- 605 76.5289 -2.728064 -2.368390 2 1
614
- 606 76.5677 -3.221817 -2.686769 2 2
615
- 607 76.4415 -2.492642 -1.891147 2 1
616
- 608 76.4803 -2.473199 -2.987411 1 1
617
- 609 76.5189 -5.629251 -6.368165 1 1
618
- 610 76.3934 -2.665304 -2.583247 2 1
619
- 611 76.4321 -5.118680 -5.229679 1 1
620
- 612 76.4706 -3.298893 -5.752535 1 1
621
- 613 76.5090 -2.481999 -2.107925 2 2
622
- 614 76.5472 -2.482575 -1.807913 2 2
623
- 615 76.5854 -2.435894 -2.273718 2 2
624
- 616 76.6234 -3.488464 -3.371219 2 2
625
- 617 76.6613 -12.500495 -3.358167 2 2
626
- 618 76.5372 -3.842674 -4.023818 1 2
627
- 619 76.5751 -3.395132 -2.717220 2 2
628
- 620 76.4516 -4.828323 -4.643736 2 1
629
- 621 76.4895 -3.494996 -3.053757 2 2
630
- 622 76.3666 -2.652963 -2.546255 2 1
631
- 623 76.2440 -2.629064 -2.440247 2 1
632
- 624 76.2821 -4.198253 -3.644315 2 2
633
- 625 76.3200 -3.523351 -5.417534 1 1
634
- 626 76.3578 -2.348354 -3.195171 1 1
635
- 627 76.3955 -3.864117 -3.966140 1 1
636
- 628 76.4331 -2.569930 -2.381017 2 2
637
- 629 76.4706 -1.580079 -1.999833 1 1
638
- 630 76.5079 -1.281138 -1.871720 1 1
639
- 631 76.5452 -1.545893 -2.643916 1 1
640
- 632 76.5823 -2.101959 -1.307263 2 2
641
- 633 76.6193 -3.036577 -2.435689 2 2
642
- 634 76.6562 -2.354149 -2.673197 1 1
643
- 635 76.5354 -3.859247 -4.585956 1 2
644
- 636 76.5723 -2.097624 -3.900617 1 1
645
- 637 76.6091 -3.230950 -5.764407 1 1
646
- 638 76.6458 -5.030876 -5.217453 1 1
647
- 639 76.6823 -1.691553 -2.134574 1 1
648
- 640 76.5625 -1.268764 -2.103911 1 2
649
- 641 76.5991 -4.206017 -5.165740 1 1
650
- 642 76.6355 -4.468725 -2.336098 2 2
651
- 643 76.6719 -1.209771 -0.654422 2 2
652
- 644 76.7081 -3.407416 -4.630384 1 1
653
- 645 76.7442 -1.481183 -1.559432 1 1
654
- 646 76.6254 -4.347997 -4.170047 2 1
655
- 647 76.5070 -4.558063 -3.952455 2 1
656
- 648 76.5432 -3.516982 -3.185503 2 2
657
- 649 76.5794 -2.622825 -2.991383 1 1
658
- 650 76.6154 -3.431971 -4.166425 1 1
659
- 651 76.6513 -5.725307 -5.438232 2 2
660
- 652 76.6871 -5.834062 -5.126859 2 2
661
- 653 76.7228 -6.515482 -5.580177 2 2
662
- 654 76.6055 -3.230389 -3.503927 1 2
663
- 655 76.4885 -2.637214 -2.637227 1 2
664
- 656 76.5244 -3.848460 -1.893588 2 2
665
- 657 76.4079 -5.456481 -5.979530 1 2
666
- 658 76.4438 -5.641815 -3.648344 2 2
667
- 659 76.4795 -2.244718 -2.135135 2 2
668
- 660 76.5152 -3.472934 -2.413404 2 2
669
- 661 76.3994 -2.791725 -2.788333 2 1
670
- 662 76.4350 -3.319531 -3.569393 1 1
671
- 663 76.4706 -5.874499 -4.640968 2 2
672
- 664 76.5060 -2.788764 -2.853702 1 1
673
- 665 76.5414 -4.052362 -3.932591 2 2
674
- 666 76.5766 -4.354697 -4.570284 1 1
675
- 667 76.4618 -5.219250 -9.232826 1 2
676
- 668 76.4970 -2.627172 -3.412858 1 1
677
- 669 76.5321 -2.841293 -3.173120 1 1
678
- 670 76.4179 -3.724929 -3.008459 2 1
679
- 671 76.4531 -3.396294 -4.830095 1 1
680
- 672 76.4881 -2.778478 -2.735994 2 2
681
- 673 76.3744 -1.916355 -1.964943 1 2
682
- 674 76.4095 -5.866518 -5.904108 1 1
683
- 675 76.2963 -4.041154 -3.737163 2 1
684
- 676 76.1834 -1.996013 -2.715125 1 2
685
- 677 76.2186 -3.867311 -4.162969 1 1
686
- 678 76.1062 -5.188138 -5.194997 1 2
687
- 679 76.1414 -5.403625 -4.785535 2 2
688
- 680 76.1765 -3.136537 -4.869346 1 1
689
- 681 76.0646 -3.778646 -2.913462 2 1
690
- 682 75.9531 -2.313861 -2.405468 1 2
691
- 683 75.9883 -1.475527 -2.679750 1 1
692
- 684 76.0234 -4.790264 -8.618103 1 1
693
- 685 76.0584 -3.628318 -1.570202 2 2
694
- 686 76.0933 -5.276064 -4.693400 2 2
695
- 687 76.1281 -2.808424 -1.709805 2 2
696
- 688 76.1628 -6.902458 -7.342489 1 1
697
- 689 76.1974 -3.040002 -2.513304 2 2
698
- 690 76.0870 -2.784919 -2.940635 1 2
699
- 691 75.9768 -3.765795 -3.427800 2 1
700
- 692 76.0116 -1.630141 -2.640359 1 1
701
- 693 76.0462 -4.690427 -4.938859 1 1
702
- 694 76.0807 -3.567502 -3.207409 2 2
703
- 695 76.1151 -5.302876 -5.602167 1 1
704
- 696 76.0057 -4.766691 -4.398072 2 1
705
- 697 76.0402 -2.729209 -2.805274 1 1
706
- 698 76.0745 -4.668901 -4.225488 2 2
707
- 699 76.1087 -3.832909 -5.474349 1 1
708
- 700 76.1429 -3.401276 -2.536767 2 2
709
- 701 76.0342 -2.313753 -1.933872 2 1
710
- 702 75.9259 -4.305689 -4.804205 1 2
711
- 703 75.8179 -3.516911 -3.204198 2 1
712
- 704 75.8523 -1.661300 -1.988352 1 1
713
- 705 75.8865 -2.722722 -2.268792 2 2
714
- 706 75.7790 -2.932345 -2.970964 1 2
715
- 707 75.6719 -5.038128 -4.843409 2 1
716
- 708 75.7062 -4.158300 -3.648957 2 2
717
- 709 75.7405 -3.951885 -3.354503 2 2
718
- 710 75.7746 -2.823708 -3.822505 1 1
719
- 711 75.8087 -2.728188 -2.336287 2 2
720
- 712 75.7022 -2.786553 -4.014195 1 2
721
- 713 75.5961 -3.383832 -3.562825 1 2
722
- 714 75.6303 -3.026311 -2.805184 2 2
723
- 715 75.6643 -5.875782 -4.671214 2 2
724
- 716 75.6983 -4.418433 -3.729445 2 2
725
- 717 75.7322 -2.416038 -0.590412 2 2
726
- 718 75.7660 -2.761889 -3.618522 1 1
727
- 719 75.6606 -5.292922 -4.932745 2 1
728
- 720 75.5556 -0.937683 -4.513329 1 2
729
- 721 75.4508 -4.809662 -5.469491 1 2
730
- 722 75.4848 -2.468329 -2.474531 1 1
731
- 723 75.5187 -2.972364 -3.133240 1 1
732
- 724 75.4144 -3.239390 -3.721723 1 2
733
- 725 75.4483 -2.808884 -2.719271 2 2
734
- 726 75.3444 -4.063407 -4.265435 1 2
735
- 727 75.3783 -3.278115 -2.616635 2 2
736
- 728 75.2747 -4.638354 -4.486451 2 1
737
- 729 75.3086 -2.317364 -2.907472 1 1
738
- 730 75.2055 -3.641715 -3.428185 2 1
739
- 731 75.2394 -3.398665 -3.763162 1 1
740
- 732 75.2732 -3.213069 -2.414680 2 2
741
- 733 75.3070 -4.312629 -3.904779 2 2
742
- 734 75.3406 -0.090574 -7.810925 1 1
743
- 735 75.2381 -2.663461 -2.633826 2 1
744
- 736 75.2717 -3.356396 -4.146633 1 1
745
- 737 75.3053 -4.101182 -4.229880 1 1
746
- 738 75.3388 -4.383867 -4.238799 2 2
747
- 739 75.3721 -0.426296 -9.473241 1 1
748
- 740 75.4054 -4.091724 -3.254830 2 2
749
- 741 75.3036 -4.235574 -4.067957 2 1
750
- 742 75.3369 -2.061121 -1.491463 2 2
751
- 743 75.3701 -3.503661 -3.405503 2 2
752
- 744 75.4032 -5.019755 -3.913309 2 2
753
- 745 75.4362 -4.261997 -5.127662 1 1
754
- 746 75.4692 -5.066962 -5.759768 1 1
755
- 747 75.3681 -5.547479 -4.915647 2 1
756
- 748 75.4011 -3.332092 -3.272070 2 2
757
- 749 75.4339 -3.169371 -3.542470 1 1
758
- 750 75.4667 -1.844345 -9.289640 1 1
759
 
760
- Final Winogrande score(750 tasks): 75.4667 +/- 1.5722
 
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q3_K_M.gguf (version GGUF V3 (latest))
4
 
5
+ Final Winogrande score(750 tasks): 72.6667 +/- 1.6284
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
 
7
+ llama_perf_context_print: load time = 331.54 ms
8
+ llama_perf_context_print: prompt eval time = 94938.20 ms / 22104 tokens ( 4.30 ms per token, 232.83 tokens per second)
9
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
10
+ llama_perf_context_print: total time = 96106.12 ms / 22105 tokens
11
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-q3_k_s.arc CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 869 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 869 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 100.00000000
13
- 3 100.00000000
14
- 4 75.00000000
15
- 5 80.00000000
16
- 6 83.33333333
17
- 7 85.71428571
18
- 8 87.50000000
19
- 9 77.77777778
20
- 10 80.00000000
21
- 11 72.72727273
22
- 12 75.00000000
23
- 13 69.23076923
24
- 14 71.42857143
25
- 15 73.33333333
26
- 16 75.00000000
27
- 17 76.47058824
28
- 18 72.22222222
29
- 19 68.42105263
30
- 20 65.00000000
31
- 21 66.66666667
32
- 22 68.18181818
33
- 23 65.21739130
34
- 24 66.66666667
35
- 25 64.00000000
36
- 26 65.38461538
37
- 27 66.66666667
38
- 28 64.28571429
39
- 29 65.51724138
40
- 30 63.33333333
41
- 31 61.29032258
42
- 32 62.50000000
43
- 33 60.60606061
44
- 34 58.82352941
45
- 35 60.00000000
46
- 36 58.33333333
47
- 37 59.45945946
48
- 38 57.89473684
49
- 39 58.97435897
50
- 40 60.00000000
51
- 41 60.97560976
52
- 42 59.52380952
53
- 43 58.13953488
54
- 44 56.81818182
55
- 45 57.77777778
56
- 46 58.69565217
57
- 47 59.57446809
58
- 48 60.41666667
59
- 49 61.22448980
60
- 50 60.00000000
61
- 51 58.82352941
62
- 52 59.61538462
63
- 53 58.49056604
64
- 54 57.40740741
65
- 55 56.36363636
66
- 56 57.14285714
67
- 57 57.89473684
68
- 58 58.62068966
69
- 59 59.32203390
70
- 60 60.00000000
71
- 61 60.65573770
72
- 62 59.67741935
73
- 63 60.31746032
74
- 64 60.93750000
75
- 65 60.00000000
76
- 66 60.60606061
77
- 67 61.19402985
78
- 68 61.76470588
79
- 69 60.86956522
80
- 70 60.00000000
81
- 71 59.15492958
82
- 72 58.33333333
83
- 73 58.90410959
84
- 74 59.45945946
85
- 75 60.00000000
86
- 76 60.52631579
87
- 77 61.03896104
88
- 78 61.53846154
89
- 79 60.75949367
90
- 80 61.25000000
91
- 81 60.49382716
92
- 82 60.97560976
93
- 83 60.24096386
94
- 84 60.71428571
95
- 85 61.17647059
96
- 86 60.46511628
97
- 87 59.77011494
98
- 88 60.22727273
99
- 89 60.67415730
100
- 90 60.00000000
101
- 91 60.43956044
102
- 92 60.86956522
103
- 93 60.21505376
104
- 94 59.57446809
105
- 95 60.00000000
106
- 96 60.41666667
107
- 97 59.79381443
108
- 98 60.20408163
109
- 99 60.60606061
110
- 100 61.00000000
111
- 101 60.39603960
112
- 102 60.78431373
113
- 103 60.19417476
114
- 104 60.57692308
115
- 105 60.95238095
116
- 106 60.37735849
117
- 107 60.74766355
118
- 108 60.18518519
119
- 109 59.63302752
120
- 110 60.00000000
121
- 111 60.36036036
122
- 112 60.71428571
123
- 113 61.06194690
124
- 114 61.40350877
125
- 115 61.73913043
126
- 116 61.20689655
127
- 117 60.68376068
128
- 118 61.01694915
129
- 119 60.50420168
130
- 120 60.83333333
131
- 121 60.33057851
132
- 122 60.65573770
133
- 123 60.97560976
134
- 124 60.48387097
135
- 125 60.80000000
136
- 126 61.11111111
137
- 127 61.41732283
138
- 128 61.71875000
139
- 129 61.24031008
140
- 130 60.76923077
141
- 131 61.06870229
142
- 132 61.36363636
143
- 133 61.65413534
144
- 134 61.19402985
145
- 135 60.74074074
146
- 136 60.29411765
147
- 137 60.58394161
148
- 138 60.86956522
149
- 139 61.15107914
150
- 140 61.42857143
151
- 141 60.99290780
152
- 142 61.26760563
153
- 143 61.53846154
154
- 144 61.11111111
155
- 145 61.37931034
156
- 146 60.95890411
157
- 147 61.22448980
158
- 148 61.48648649
159
- 149 61.74496644
160
- 150 62.00000000
161
- 151 62.25165563
162
- 152 62.50000000
163
- 153 62.74509804
164
- 154 62.33766234
165
- 155 62.58064516
166
- 156 62.82051282
167
- 157 63.05732484
168
- 158 62.65822785
169
- 159 62.89308176
170
- 160 62.50000000
171
- 161 62.73291925
172
- 162 62.96296296
173
- 163 62.57668712
174
- 164 62.19512195
175
- 165 62.42424242
176
- 166 62.65060241
177
- 167 62.27544910
178
- 168 62.50000000
179
- 169 62.13017751
180
- 170 62.35294118
181
- 171 62.57309942
182
- 172 62.20930233
183
- 173 62.42774566
184
- 174 62.64367816
185
- 175 62.28571429
186
- 176 62.50000000
187
- 177 62.71186441
188
- 178 62.92134831
189
- 179 63.12849162
190
- 180 63.33333333
191
- 181 62.98342541
192
- 182 63.18681319
193
- 183 62.84153005
194
- 184 63.04347826
195
- 185 63.24324324
196
- 186 63.44086022
197
- 187 63.63636364
198
- 188 63.29787234
199
- 189 62.96296296
200
- 190 62.63157895
201
- 191 62.82722513
202
- 192 63.02083333
203
- 193 63.21243523
204
- 194 63.40206186
205
- 195 63.58974359
206
- 196 63.77551020
207
- 197 63.95939086
208
- 198 64.14141414
209
- 199 63.81909548
210
- 200 64.00000000
211
- 201 64.17910448
212
- 202 63.86138614
213
- 203 64.03940887
214
- 204 64.21568627
215
- 205 64.39024390
216
- 206 64.07766990
217
- 207 63.76811594
218
- 208 63.94230769
219
- 209 64.11483254
220
- 210 64.28571429
221
- 211 64.45497630
222
- 212 64.15094340
223
- 213 64.31924883
224
- 214 64.48598131
225
- 215 64.65116279
226
- 216 64.35185185
227
- 217 64.05529954
228
- 218 63.76146789
229
- 219 63.92694064
230
- 220 63.63636364
231
- 221 63.80090498
232
- 222 63.51351351
233
- 223 63.22869955
234
- 224 63.39285714
235
- 225 63.11111111
236
- 226 62.83185841
237
- 227 62.99559471
238
- 228 63.15789474
239
- 229 63.31877729
240
- 230 63.47826087
241
- 231 63.20346320
242
- 232 62.93103448
243
- 233 63.09012876
244
- 234 63.24786325
245
- 235 62.97872340
246
- 236 62.71186441
247
- 237 62.44725738
248
- 238 62.60504202
249
- 239 62.76150628
250
- 240 62.91666667
251
- 241 63.07053942
252
- 242 63.22314050
253
- 243 63.37448560
254
- 244 63.11475410
255
- 245 63.26530612
256
- 246 63.41463415
257
- 247 63.56275304
258
- 248 63.70967742
259
- 249 63.45381526
260
- 250 63.60000000
261
- 251 63.74501992
262
- 252 63.88888889
263
- 253 64.03162055
264
- 254 63.77952756
265
- 255 63.92156863
266
- 256 64.06250000
267
- 257 63.81322957
268
- 258 63.95348837
269
- 259 63.70656371
270
- 260 63.84615385
271
- 261 63.98467433
272
- 262 64.12213740
273
- 263 64.25855513
274
- 264 64.39393939
275
- 265 64.15094340
276
- 266 63.90977444
277
- 267 64.04494382
278
- 268 64.17910448
279
- 269 64.31226766
280
- 270 64.44444444
281
- 271 64.57564576
282
- 272 64.33823529
283
- 273 64.10256410
284
- 274 63.86861314
285
- 275 63.63636364
286
- 276 63.76811594
287
- 277 63.89891697
288
- 278 63.66906475
289
- 279 63.79928315
290
- 280 63.92857143
291
- 281 64.05693950
292
- 282 64.18439716
293
- 283 63.95759717
294
- 284 64.08450704
295
- 285 64.21052632
296
- 286 64.33566434
297
- 287 64.45993031
298
- 288 64.58333333
299
- 289 64.35986159
300
- 290 64.48275862
301
- 291 64.60481100
302
- 292 64.72602740
303
- 293 64.84641638
304
- 294 64.62585034
305
- 295 64.74576271
306
- 296 64.86486486
307
- 297 64.98316498
308
- 298 65.10067114
309
- 299 65.21739130
310
- 300 65.00000000
311
- 301 64.78405316
312
- 302 64.90066225
313
- 303 65.01650165
314
- 304 65.13157895
315
- 305 65.24590164
316
- 306 65.03267974
317
- 307 64.82084691
318
- 308 64.93506494
319
- 309 65.04854369
320
- 310 64.83870968
321
- 311 64.95176849
322
- 312 64.74358974
323
- 313 64.85623003
324
- 314 64.64968153
325
- 315 64.76190476
326
- 316 64.87341772
327
- 317 64.98422713
328
- 318 64.77987421
329
- 319 64.57680251
330
- 320 64.37500000
331
- 321 64.48598131
332
- 322 64.59627329
333
- 323 64.70588235
334
- 324 64.81481481
335
- 325 64.92307692
336
- 326 65.03067485
337
- 327 64.83180428
338
- 328 64.93902439
339
- 329 65.04559271
340
- 330 64.84848485
341
- 331 64.95468278
342
- 332 65.06024096
343
- 333 65.16516517
344
- 334 65.26946108
345
- 335 65.07462687
346
- 336 65.17857143
347
- 337 65.28189911
348
- 338 65.08875740
349
- 339 65.19174041
350
- 340 65.29411765
351
- 341 65.39589443
352
- 342 65.49707602
353
- 343 65.59766764
354
- 344 65.69767442
355
- 345 65.50724638
356
- 346 65.60693642
357
- 347 65.41786744
358
- 348 65.22988506
359
- 349 65.32951289
360
- 350 65.42857143
361
- 351 65.52706553
362
- 352 65.62500000
363
- 353 65.72237960
364
- 354 65.53672316
365
- 355 65.35211268
366
- 356 65.44943820
367
- 357 65.54621849
368
- 358 65.64245810
369
- 359 65.45961003
370
- 360 65.55555556
371
- 361 65.65096953
372
- 362 65.74585635
373
- 363 65.84022039
374
- 364 65.65934066
375
- 365 65.75342466
376
- 366 65.57377049
377
- 367 65.39509537
378
- 368 65.48913043
379
- 369 65.58265583
380
- 370 65.67567568
381
- 371 65.76819407
382
- 372 65.86021505
383
- 373 65.95174263
384
- 374 66.04278075
385
- 375 66.13333333
386
- 376 65.95744681
387
- 377 65.78249337
388
- 378 65.87301587
389
- 379 65.69920844
390
- 380 65.78947368
391
- 381 65.61679790
392
- 382 65.44502618
393
- 383 65.27415144
394
- 384 65.36458333
395
- 385 65.19480519
396
- 386 65.28497409
397
- 387 65.37467700
398
- 388 65.20618557
399
- 389 65.03856041
400
- 390 64.87179487
401
- 391 64.96163683
402
- 392 65.05102041
403
- 393 65.13994911
404
- 394 65.22842640
405
- 395 65.31645570
406
- 396 65.15151515
407
- 397 65.23929471
408
- 398 65.07537688
409
- 399 65.16290727
410
- 400 65.25000000
411
- 401 65.08728180
412
- 402 65.17412935
413
- 403 65.26054591
414
- 404 65.34653465
415
- 405 65.43209877
416
- 406 65.51724138
417
- 407 65.60196560
418
- 408 65.68627451
419
- 409 65.52567237
420
- 410 65.60975610
421
- 411 65.69343066
422
- 412 65.77669903
423
- 413 65.61743341
424
- 414 65.70048309
425
- 415 65.54216867
426
- 416 65.62500000
427
- 417 65.46762590
428
- 418 65.31100478
429
- 419 65.15513126
430
- 420 65.23809524
431
- 421 65.08313539
432
- 422 65.16587678
433
- 423 65.24822695
434
- 424 65.09433962
435
- 425 65.17647059
436
- 426 65.02347418
437
- 427 65.10538642
438
- 428 64.95327103
439
- 429 65.03496503
440
- 430 64.88372093
441
- 431 64.96519722
442
- 432 65.04629630
443
- 433 64.89607390
444
- 434 64.97695853
445
- 435 65.05747126
446
- 436 65.13761468
447
- 437 65.21739130
448
- 438 65.06849315
449
- 439 65.14806378
450
- 440 65.00000000
451
- 441 64.85260771
452
- 442 64.70588235
453
- 443 64.55981941
454
- 444 64.41441441
455
- 445 64.49438202
456
- 446 64.57399103
457
- 447 64.65324385
458
- 448 64.73214286
459
- 449 64.81069042
460
- 450 64.66666667
461
- 451 64.74501109
462
- 452 64.82300885
463
- 453 64.67991170
464
- 454 64.75770925
465
- 455 64.83516484
466
- 456 64.91228070
467
- 457 64.98905908
468
- 458 65.06550218
469
- 459 65.14161220
470
- 460 65.00000000
471
- 461 65.07592191
472
- 462 65.15151515
473
- 463 65.01079914
474
- 464 64.87068966
475
- 465 64.94623656
476
- 466 65.02145923
477
- 467 65.09635974
478
- 468 65.17094017
479
- 469 65.03198294
480
- 470 65.10638298
481
- 471 65.18046709
482
- 472 65.25423729
483
- 473 65.11627907
484
- 474 65.18987342
485
- 475 65.26315789
486
- 476 65.33613445
487
- 477 65.40880503
488
- 478 65.48117155
489
- 479 65.55323591
490
- 480 65.62500000
491
- 481 65.69646570
492
- 482 65.76763485
493
- 483 65.83850932
494
- 484 65.90909091
495
- 485 65.97938144
496
- 486 66.04938272
497
- 487 65.91375770
498
- 488 65.98360656
499
- 489 66.05316973
500
- 490 65.91836735
501
- 491 65.78411405
502
- 492 65.85365854
503
- 493 65.72008114
504
- 494 65.78947368
505
- 495 65.65656566
506
- 496 65.72580645
507
- 497 65.79476861
508
- 498 65.86345382
509
- 499 65.93186373
510
- 500 66.00000000
511
- 501 65.86826347
512
- 502 65.93625498
513
- 503 66.00397614
514
- 504 66.07142857
515
- 505 66.13861386
516
- 506 66.20553360
517
- 507 66.07495069
518
- 508 65.94488189
519
- 509 65.81532417
520
- 510 65.68627451
521
- 511 65.55772994
522
- 512 65.62500000
523
- 513 65.49707602
524
- 514 65.56420233
525
- 515 65.63106796
526
- 516 65.69767442
527
- 517 65.76402321
528
- 518 65.83011583
529
- 519 65.70327553
530
- 520 65.57692308
531
- 521 65.64299424
532
- 522 65.70881226
533
- 523 65.58317400
534
- 524 65.45801527
535
- 525 65.33333333
536
- 526 65.20912548
537
- 527 65.08538899
538
- 528 64.96212121
539
- 529 64.83931947
540
- 530 64.90566038
541
- 531 64.97175141
542
- 532 65.03759398
543
- 533 65.10318949
544
- 534 64.98127341
545
- 535 64.85981308
546
- 536 64.92537313
547
- 537 64.99068901
548
- 538 65.05576208
549
- 539 65.12059369
550
- 540 65.00000000
551
- 541 64.87985213
552
- 542 64.94464945
553
- 543 64.82504604
554
- 544 64.70588235
555
- 545 64.77064220
556
- 546 64.65201465
557
- 547 64.53382084
558
- 548 64.59854015
559
- 549 64.66302368
560
- 550 64.72727273
561
- 551 64.60980036
562
- 552 64.49275362
563
- 553 64.55696203
564
- 554 64.62093863
565
- 555 64.68468468
566
- 556 64.56834532
567
- 557 64.63195691
568
- 558 64.51612903
569
- 559 64.57960644
570
- 560 64.64285714
571
- 561 64.70588235
572
- 562 64.76868327
573
- 563 64.83126110
574
- 564 64.89361702
575
- 565 64.77876106
576
- 566 64.84098940
577
- 567 64.72663139
578
- 568 64.78873239
579
- 569 64.67486819
580
- 570 64.56140351
581
- 571 64.62346760
582
- 572 64.51048951
583
- 573 64.57242583
584
- 574 64.63414634
585
- 575 64.69565217
586
- 576 64.75694444
587
- 577 64.81802426
588
- 578 64.87889273
589
- 579 64.93955095
590
- 580 64.82758621
591
- 581 64.71600688
592
- 582 64.77663230
593
- 583 64.83704974
594
- 584 64.89726027
595
- 585 64.95726496
596
- 586 65.01706485
597
- 587 65.07666099
598
- 588 65.13605442
599
- 589 65.19524618
600
- 590 65.25423729
601
- 591 65.31302876
602
- 592 65.20270270
603
- 593 65.09274874
604
- 594 65.15151515
605
- 595 65.04201681
606
- 596 64.93288591
607
- 597 64.82412060
608
- 598 64.88294314
609
- 599 64.94156928
610
- 600 64.83333333
611
- 601 64.89184692
612
- 602 64.78405316
613
- 603 64.67661692
614
- 604 64.73509934
615
- 605 64.79338843
616
- 606 64.85148515
617
- 607 64.90939044
618
- 608 64.96710526
619
- 609 64.86042693
620
- 610 64.91803279
621
- 611 64.81178396
622
- 612 64.86928105
623
- 613 64.76345840
624
- 614 64.82084691
625
- 615 64.71544715
626
- 616 64.61038961
627
- 617 64.66774716
628
- 618 64.56310680
629
- 619 64.45880452
630
- 620 64.51612903
631
- 621 64.57326892
632
- 622 64.46945338
633
- 623 64.36597111
634
- 624 64.26282051
635
- 625 64.16000000
636
- 626 64.05750799
637
- 627 64.11483254
638
- 628 64.17197452
639
- 629 64.06995231
640
- 630 64.12698413
641
- 631 64.18383518
642
- 632 64.08227848
643
- 633 63.98104265
644
- 634 64.03785489
645
- 635 64.09448819
646
- 636 64.15094340
647
- 637 64.05023548
648
- 638 64.10658307
649
- 639 64.16275430
650
- 640 64.06250000
651
- 641 63.96255850
652
- 642 63.86292835
653
- 643 63.76360809
654
- 644 63.81987578
655
- 645 63.72093023
656
- 646 63.77708978
657
- 647 63.83307573
658
- 648 63.73456790
659
- 649 63.63636364
660
- 650 63.69230769
661
- 651 63.74807988
662
- 652 63.65030675
663
- 653 63.70597243
664
- 654 63.76146789
665
- 655 63.66412214
666
- 656 63.56707317
667
- 657 63.47031963
668
- 658 63.37386018
669
- 659 63.27769347
670
- 660 63.33333333
671
- 661 63.23751891
672
- 662 63.29305136
673
- 663 63.34841629
674
- 664 63.40361446
675
- 665 63.30827068
676
- 666 63.21321321
677
- 667 63.26836582
678
- 668 63.17365269
679
- 669 63.07922272
680
- 670 62.98507463
681
- 671 62.89120715
682
- 672 62.94642857
683
- 673 62.85289747
684
- 674 62.90801187
685
- 675 62.81481481
686
- 676 62.86982249
687
- 677 62.92466765
688
- 678 62.97935103
689
- 679 63.03387334
690
- 680 63.08823529
691
- 681 62.99559471
692
- 682 62.90322581
693
- 683 62.81112738
694
- 684 62.86549708
695
- 685 62.91970803
696
- 686 62.82798834
697
- 687 62.73653566
698
- 688 62.79069767
699
- 689 62.69956459
700
- 690 62.75362319
701
- 691 62.80752533
702
- 692 62.86127168
703
- 693 62.91486291
704
- 694 62.96829971
705
- 695 62.87769784
706
- 696 62.78735632
707
- 697 62.69727403
708
- 698 62.75071633
709
- 699 62.80400572
710
- 700 62.85714286
711
- 701 62.91012839
712
- 702 62.96296296
713
- 703 63.01564723
714
- 704 63.06818182
715
- 705 63.12056738
716
- 706 63.17280453
717
- 707 63.22489392
718
- 708 63.27683616
719
- 709 63.32863188
720
- 710 63.38028169
721
- 711 63.43178622
722
- 712 63.48314607
723
- 713 63.53436185
724
- 714 63.44537815
725
- 715 63.35664336
726
- 716 63.40782123
727
- 717 63.31938633
728
- 718 63.23119777
729
- 719 63.14325452
730
- 720 63.19444444
731
- 721 63.24549237
732
- 722 63.29639889
733
- 723 63.34716459
734
- 724 63.25966851
735
- 725 63.31034483
736
- 726 63.22314050
737
- 727 63.13617607
738
- 728 63.18681319
739
- 729 63.23731139
740
- 730 63.28767123
741
- 731 63.33789330
742
- 732 63.38797814
743
- 733 63.43792633
744
- 734 63.48773842
745
- 735 63.53741497
746
- 736 63.58695652
747
- 737 63.50067843
748
- 738 63.55013550
749
- 739 63.46414073
750
- 740 63.37837838
751
- 741 63.42780027
752
- 742 63.34231806
753
- 743 63.39165545
754
- 744 63.30645161
755
- 745 63.35570470
756
- 746 63.40482574
757
- 747 63.31994645
758
- 748 63.23529412
759
- 749 63.15086782
760
- 750 63.20000000
761
 
762
- Final result: 63.2000 +/- 1.7621
763
- Random chance: 25.0083 +/- 1.5824
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q3_K_S.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 61.7135 +/- 1.7797
6
+ Random chance: 25.0251 +/- 1.5859
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 1613.35 ms
10
+ llama_perf_context_print: prompt eval time = 172264.72 ms / 36428 tokens ( 4.73 ms per token, 211.47 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 174482.10 ms / 36429 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-q3_k_s.hsw CHANGED
@@ -1,760 +1,12 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- hellaswag_score : loaded 10042 tasks from prompt.
6
- ================================= is_spm = 0
7
- hellaswag_score : selecting 750 randomized tasks.
8
- hellaswag_score : calculating hellaswag score over selected tasks.
9
 
10
- task acc_norm
11
- 1 0.00000000
12
- 2 50.00000000
13
- 3 33.33333333
14
- 4 50.00000000
15
- 5 60.00000000
16
- 6 66.66666667
17
- 7 57.14285714
18
- 8 62.50000000
19
- 9 55.55555556
20
- 10 60.00000000
21
- 11 63.63636364
22
- 12 58.33333333
23
- 13 61.53846154
24
- 14 64.28571429
25
- 15 66.66666667
26
- 16 68.75000000
27
- 17 64.70588235
28
- 18 66.66666667
29
- 19 63.15789474
30
- 20 65.00000000
31
- 21 66.66666667
32
- 22 68.18181818
33
- 23 69.56521739
34
- 24 66.66666667
35
- 25 64.00000000
36
- 26 61.53846154
37
- 27 59.25925926
38
- 28 60.71428571
39
- 29 62.06896552
40
- 30 60.00000000
41
- 31 61.29032258
42
- 32 59.37500000
43
- 33 60.60606061
44
- 34 61.76470588
45
- 35 62.85714286
46
- 36 61.11111111
47
- 37 59.45945946
48
- 38 60.52631579
49
- 39 61.53846154
50
- 40 60.00000000
51
- 41 60.97560976
52
- 42 59.52380952
53
- 43 60.46511628
54
- 44 61.36363636
55
- 45 60.00000000
56
- 46 60.86956522
57
- 47 59.57446809
58
- 48 60.41666667
59
- 49 61.22448980
60
- 50 60.00000000
61
- 51 60.78431373
62
- 52 59.61538462
63
- 53 58.49056604
64
- 54 57.40740741
65
- 55 56.36363636
66
- 56 55.35714286
67
- 57 56.14035088
68
- 58 56.89655172
69
- 59 57.62711864
70
- 60 58.33333333
71
- 61 57.37704918
72
- 62 56.45161290
73
- 63 57.14285714
74
- 64 57.81250000
75
- 65 58.46153846
76
- 66 57.57575758
77
- 67 58.20895522
78
- 68 58.82352941
79
- 69 57.97101449
80
- 70 58.57142857
81
- 71 57.74647887
82
- 72 58.33333333
83
- 73 58.90410959
84
- 74 59.45945946
85
- 75 58.66666667
86
- 76 59.21052632
87
- 77 59.74025974
88
- 78 60.25641026
89
- 79 60.75949367
90
- 80 61.25000000
91
- 81 61.72839506
92
- 82 62.19512195
93
- 83 62.65060241
94
- 84 63.09523810
95
- 85 63.52941176
96
- 86 63.95348837
97
- 87 64.36781609
98
- 88 64.77272727
99
- 89 64.04494382
100
- 90 64.44444444
101
- 91 64.83516484
102
- 92 64.13043478
103
- 93 64.51612903
104
- 94 64.89361702
105
- 95 65.26315789
106
- 96 65.62500000
107
- 97 65.97938144
108
- 98 66.32653061
109
- 99 66.66666667
110
- 100 66.00000000
111
- 101 66.33663366
112
- 102 65.68627451
113
- 103 65.04854369
114
- 104 65.38461538
115
- 105 65.71428571
116
- 106 66.03773585
117
- 107 66.35514019
118
- 108 66.66666667
119
- 109 66.05504587
120
- 110 66.36363636
121
- 111 66.66666667
122
- 112 66.96428571
123
- 113 67.25663717
124
- 114 67.54385965
125
- 115 67.82608696
126
- 116 67.24137931
127
- 117 66.66666667
128
- 118 66.94915254
129
- 119 67.22689076
130
- 120 66.66666667
131
- 121 66.94214876
132
- 122 67.21311475
133
- 123 67.47967480
134
- 124 67.74193548
135
- 125 67.20000000
136
- 126 67.46031746
137
- 127 67.71653543
138
- 128 67.18750000
139
- 129 67.44186047
140
- 130 66.92307692
141
- 131 67.17557252
142
- 132 66.66666667
143
- 133 66.91729323
144
- 134 67.16417910
145
- 135 67.40740741
146
- 136 67.64705882
147
- 137 67.88321168
148
- 138 68.11594203
149
- 139 67.62589928
150
- 140 67.85714286
151
- 141 68.08510638
152
- 142 68.30985915
153
- 143 68.53146853
154
- 144 68.75000000
155
- 145 68.96551724
156
- 146 69.17808219
157
- 147 69.38775510
158
- 148 69.59459459
159
- 149 69.79865772
160
- 150 70.00000000
161
- 151 69.53642384
162
- 152 69.07894737
163
- 153 69.28104575
164
- 154 69.48051948
165
- 155 69.67741935
166
- 156 69.87179487
167
- 157 69.42675159
168
- 158 69.62025316
169
- 159 69.18238994
170
- 160 69.37500000
171
- 161 69.56521739
172
- 162 69.75308642
173
- 163 69.93865031
174
- 164 70.12195122
175
- 165 70.30303030
176
- 166 70.48192771
177
- 167 70.65868263
178
- 168 70.83333333
179
- 169 71.00591716
180
- 170 71.17647059
181
- 171 71.34502924
182
- 172 71.51162791
183
- 173 71.09826590
184
- 174 71.26436782
185
- 175 71.42857143
186
- 176 71.59090909
187
- 177 71.75141243
188
- 178 71.91011236
189
- 179 71.50837989
190
- 180 71.66666667
191
- 181 71.27071823
192
- 182 71.42857143
193
- 183 71.58469945
194
- 184 71.19565217
195
- 185 71.35135135
196
- 186 70.96774194
197
- 187 70.58823529
198
- 188 70.74468085
199
- 189 70.37037037
200
- 190 70.52631579
201
- 191 70.68062827
202
- 192 70.83333333
203
- 193 70.46632124
204
- 194 70.61855670
205
- 195 70.25641026
206
- 196 69.89795918
207
- 197 69.54314721
208
- 198 69.19191919
209
- 199 69.34673367
210
- 200 69.50000000
211
- 201 69.65174129
212
- 202 69.80198020
213
- 203 69.95073892
214
- 204 70.09803922
215
- 205 70.24390244
216
- 206 70.38834951
217
- 207 70.53140097
218
- 208 70.19230769
219
- 209 70.33492823
220
- 210 70.47619048
221
- 211 70.61611374
222
- 212 70.75471698
223
- 213 70.89201878
224
- 214 71.02803738
225
- 215 70.69767442
226
- 216 70.83333333
227
- 217 70.50691244
228
- 218 70.64220183
229
- 219 70.77625571
230
- 220 70.90909091
231
- 221 70.58823529
232
- 222 70.72072072
233
- 223 70.85201794
234
- 224 70.98214286
235
- 225 71.11111111
236
- 226 71.23893805
237
- 227 70.92511013
238
- 228 71.05263158
239
- 229 71.17903930
240
- 230 71.30434783
241
- 231 70.99567100
242
- 232 70.68965517
243
- 233 70.81545064
244
- 234 70.94017094
245
- 235 70.63829787
246
- 236 70.33898305
247
- 237 70.46413502
248
- 238 70.58823529
249
- 239 70.71129707
250
- 240 70.83333333
251
- 241 70.95435685
252
- 242 71.07438017
253
- 243 71.19341564
254
- 244 70.90163934
255
- 245 70.61224490
256
- 246 70.32520325
257
- 247 70.44534413
258
- 248 70.56451613
259
- 249 70.68273092
260
- 250 70.80000000
261
- 251 70.91633466
262
- 252 71.03174603
263
- 253 70.75098814
264
- 254 70.86614173
265
- 255 70.98039216
266
- 256 71.09375000
267
- 257 70.81712062
268
- 258 70.93023256
269
- 259 71.04247104
270
- 260 71.15384615
271
- 261 71.26436782
272
- 262 70.99236641
273
- 263 71.10266160
274
- 264 71.21212121
275
- 265 71.32075472
276
- 266 71.05263158
277
- 267 71.16104869
278
- 268 71.26865672
279
- 269 71.37546468
280
- 270 71.48148148
281
- 271 71.21771218
282
- 272 71.32352941
283
- 273 71.06227106
284
- 274 71.16788321
285
- 275 70.90909091
286
- 276 71.01449275
287
- 277 71.11913357
288
- 278 71.22302158
289
- 279 71.32616487
290
- 280 71.42857143
291
- 281 71.53024911
292
- 282 71.63120567
293
- 283 71.73144876
294
- 284 71.83098592
295
- 285 71.57894737
296
- 286 71.67832168
297
- 287 71.77700348
298
- 288 71.87500000
299
- 289 71.97231834
300
- 290 72.06896552
301
- 291 72.16494845
302
- 292 71.91780822
303
- 293 72.01365188
304
- 294 72.10884354
305
- 295 72.20338983
306
- 296 72.29729730
307
- 297 72.39057239
308
- 298 72.48322148
309
- 299 72.24080268
310
- 300 72.33333333
311
- 301 72.42524917
312
- 302 72.51655629
313
- 303 72.27722772
314
- 304 72.36842105
315
- 305 72.45901639
316
- 306 72.54901961
317
- 307 72.31270358
318
- 308 72.40259740
319
- 309 72.49190939
320
- 310 72.58064516
321
- 311 72.66881029
322
- 312 72.75641026
323
- 313 72.84345048
324
- 314 72.92993631
325
- 315 73.01587302
326
- 316 73.10126582
327
- 317 73.18611987
328
- 318 72.95597484
329
- 319 73.04075235
330
- 320 73.12500000
331
- 321 72.89719626
332
- 322 72.98136646
333
- 323 73.06501548
334
- 324 73.14814815
335
- 325 73.23076923
336
- 326 73.31288344
337
- 327 73.08868502
338
- 328 73.17073171
339
- 329 73.25227964
340
- 330 73.33333333
341
- 331 73.41389728
342
- 332 73.49397590
343
- 333 73.57357357
344
- 334 73.35329341
345
- 335 73.43283582
346
- 336 73.51190476
347
- 337 73.29376855
348
- 338 73.07692308
349
- 339 73.15634218
350
- 340 73.23529412
351
- 341 73.31378299
352
- 342 73.39181287
353
- 343 73.46938776
354
- 344 73.54651163
355
- 345 73.62318841
356
- 346 73.41040462
357
- 347 73.48703170
358
- 348 73.27586207
359
- 349 73.06590258
360
- 350 73.14285714
361
- 351 73.21937322
362
- 352 73.29545455
363
- 353 73.37110482
364
- 354 73.44632768
365
- 355 73.52112676
366
- 356 73.59550562
367
- 357 73.66946779
368
- 358 73.46368715
369
- 359 73.53760446
370
- 360 73.33333333
371
- 361 73.40720222
372
- 362 73.48066298
373
- 363 73.55371901
374
- 364 73.62637363
375
- 365 73.69863014
376
- 366 73.77049180
377
- 367 73.84196185
378
- 368 73.91304348
379
- 369 73.98373984
380
- 370 74.05405405
381
- 371 73.85444744
382
- 372 73.92473118
383
- 373 73.99463807
384
- 374 74.06417112
385
- 375 74.13333333
386
- 376 74.20212766
387
- 377 74.00530504
388
- 378 74.07407407
389
- 379 74.14248021
390
- 380 74.21052632
391
- 381 74.27821522
392
- 382 74.34554974
393
- 383 74.41253264
394
- 384 74.47916667
395
- 385 74.54545455
396
- 386 74.61139896
397
- 387 74.67700258
398
- 388 74.48453608
399
- 389 74.55012853
400
- 390 74.61538462
401
- 391 74.42455243
402
- 392 74.48979592
403
- 393 74.55470738
404
- 394 74.61928934
405
- 395 74.68354430
406
- 396 74.74747475
407
- 397 74.81108312
408
- 398 74.87437186
409
- 399 74.93734336
410
- 400 75.00000000
411
- 401 75.06234414
412
- 402 75.12437811
413
- 403 74.93796526
414
- 404 75.00000000
415
- 405 74.81481481
416
- 406 74.87684729
417
- 407 74.93857494
418
- 408 75.00000000
419
- 409 75.06112469
420
- 410 75.12195122
421
- 411 74.93917275
422
- 412 75.00000000
423
- 413 75.06053269
424
- 414 75.12077295
425
- 415 74.93975904
426
- 416 75.00000000
427
- 417 75.05995204
428
- 418 75.11961722
429
- 419 75.17899761
430
- 420 75.23809524
431
- 421 75.05938242
432
- 422 75.11848341
433
- 423 75.17730496
434
- 424 75.23584906
435
- 425 75.29411765
436
- 426 75.35211268
437
- 427 75.17564403
438
- 428 75.23364486
439
- 429 75.05827506
440
- 430 75.11627907
441
- 431 75.17401392
442
- 432 75.00000000
443
- 433 75.05773672
444
- 434 75.11520737
445
- 435 74.94252874
446
- 436 75.00000000
447
- 437 74.82837529
448
- 438 74.65753425
449
- 439 74.71526196
450
- 440 74.54545455
451
- 441 74.60317460
452
- 442 74.66063348
453
- 443 74.71783296
454
- 444 74.77477477
455
- 445 74.83146067
456
- 446 74.88789238
457
- 447 74.94407159
458
- 448 75.00000000
459
- 449 75.05567929
460
- 450 75.11111111
461
- 451 74.94456763
462
- 452 74.77876106
463
- 453 74.83443709
464
- 454 74.88986784
465
- 455 74.94505495
466
- 456 75.00000000
467
- 457 75.05470460
468
- 458 75.10917031
469
- 459 75.16339869
470
- 460 75.21739130
471
- 461 75.27114967
472
- 462 75.32467532
473
- 463 75.37796976
474
- 464 75.43103448
475
- 465 75.26881720
476
- 466 75.32188841
477
- 467 75.37473233
478
- 468 75.21367521
479
- 469 75.05330490
480
- 470 74.89361702
481
- 471 74.94692144
482
- 472 75.00000000
483
- 473 75.05285412
484
- 474 75.10548523
485
- 475 75.15789474
486
- 476 75.21008403
487
- 477 75.05241090
488
- 478 75.10460251
489
- 479 75.15657620
490
- 480 75.20833333
491
- 481 75.05197505
492
- 482 74.89626556
493
- 483 74.94824017
494
- 484 74.79338843
495
- 485 74.63917526
496
- 486 74.69135802
497
- 487 74.74332649
498
- 488 74.59016393
499
- 489 74.64212679
500
- 490 74.69387755
501
- 491 74.74541752
502
- 492 74.79674797
503
- 493 74.84787018
504
- 494 74.89878543
505
- 495 74.94949495
506
- 496 75.00000000
507
- 497 75.05030181
508
- 498 75.10040161
509
- 499 74.94989980
510
- 500 75.00000000
511
- 501 74.85029940
512
- 502 74.90039841
513
- 503 74.95029821
514
- 504 75.00000000
515
- 505 75.04950495
516
- 506 75.09881423
517
- 507 75.14792899
518
- 508 75.19685039
519
- 509 75.24557957
520
- 510 75.29411765
521
- 511 75.34246575
522
- 512 75.19531250
523
- 513 75.04873294
524
- 514 75.09727626
525
- 515 75.14563107
526
- 516 75.00000000
527
- 517 74.85493230
528
- 518 74.90347490
529
- 519 74.95183044
530
- 520 75.00000000
531
- 521 74.85604607
532
- 522 74.71264368
533
- 523 74.76099426
534
- 524 74.80916031
535
- 525 74.85714286
536
- 526 74.71482890
537
- 527 74.57305503
538
- 528 74.62121212
539
- 529 74.66918715
540
- 530 74.71698113
541
- 531 74.76459510
542
- 532 74.81203008
543
- 533 74.85928705
544
- 534 74.90636704
545
- 535 74.95327103
546
- 536 75.00000000
547
- 537 75.04655493
548
- 538 75.09293680
549
- 539 74.95361781
550
- 540 75.00000000
551
- 541 75.04621072
552
- 542 75.09225092
553
- 543 74.95395948
554
- 544 75.00000000
555
- 545 75.04587156
556
- 546 75.09157509
557
- 547 75.13711152
558
- 548 75.18248175
559
- 549 75.22768670
560
- 550 75.09090909
561
- 551 75.13611615
562
- 552 75.00000000
563
- 553 75.04520796
564
- 554 75.09025271
565
- 555 74.95495495
566
- 556 75.00000000
567
- 557 74.86535009
568
- 558 74.73118280
569
- 559 74.77638640
570
- 560 74.82142857
571
- 561 74.86631016
572
- 562 74.91103203
573
- 563 74.95559503
574
- 564 75.00000000
575
- 565 74.86725664
576
- 566 74.73498233
577
- 567 74.77954145
578
- 568 74.82394366
579
- 569 74.69244288
580
- 570 74.73684211
581
- 571 74.78108581
582
- 572 74.82517483
583
- 573 74.86910995
584
- 574 74.91289199
585
- 575 74.95652174
586
- 576 75.00000000
587
- 577 75.04332756
588
- 578 75.08650519
589
- 579 75.12953368
590
- 580 75.17241379
591
- 581 75.04302926
592
- 582 75.08591065
593
- 583 75.12864494
594
- 584 75.17123288
595
- 585 75.21367521
596
- 586 75.25597270
597
- 587 75.29812606
598
- 588 75.17006803
599
- 589 75.04244482
600
- 590 74.91525424
601
- 591 74.95769882
602
- 592 75.00000000
603
- 593 74.87352445
604
- 594 74.74747475
605
- 595 74.78991597
606
- 596 74.83221477
607
- 597 74.87437186
608
- 598 74.91638796
609
- 599 74.79131886
610
- 600 74.83333333
611
- 601 74.87520799
612
- 602 74.91694352
613
- 603 74.95854063
614
- 604 75.00000000
615
- 605 75.04132231
616
- 606 74.91749175
617
- 607 74.95881384
618
- 608 74.83552632
619
- 609 74.87684729
620
- 610 74.91803279
621
- 611 74.95908347
622
- 612 74.83660131
623
- 613 74.71451876
624
- 614 74.75570033
625
- 615 74.79674797
626
- 616 74.67532468
627
- 617 74.71636953
628
- 618 74.75728155
629
- 619 74.79806139
630
- 620 74.83870968
631
- 621 74.87922705
632
- 622 74.91961415
633
- 623 74.95987159
634
- 624 75.00000000
635
- 625 75.04000000
636
- 626 75.07987220
637
- 627 75.11961722
638
- 628 75.15923567
639
- 629 75.03974563
640
- 630 74.92063492
641
- 631 74.96038035
642
- 632 74.84177215
643
- 633 74.72353870
644
- 634 74.76340694
645
- 635 74.80314961
646
- 636 74.84276730
647
- 637 74.88226060
648
- 638 74.92163009
649
- 639 74.96087637
650
- 640 74.84375000
651
- 641 74.88299532
652
- 642 74.92211838
653
- 643 74.96111975
654
- 644 75.00000000
655
- 645 74.88372093
656
- 646 74.92260062
657
- 647 74.96136012
658
- 648 75.00000000
659
- 649 74.88443760
660
- 650 74.92307692
661
- 651 74.96159754
662
- 652 75.00000000
663
- 653 75.03828484
664
- 654 75.07645260
665
- 655 75.11450382
666
- 656 75.15243902
667
- 657 75.19025875
668
- 658 75.22796353
669
- 659 75.11380880
670
- 660 75.15151515
671
- 661 75.18910741
672
- 662 75.22658610
673
- 663 75.26395173
674
- 664 75.30120482
675
- 665 75.33834586
676
- 666 75.37537538
677
- 667 75.41229385
678
- 668 75.44910180
679
- 669 75.33632287
680
- 670 75.37313433
681
- 671 75.40983607
682
- 672 75.29761905
683
- 673 75.33432392
684
- 674 75.37091988
685
- 675 75.40740741
686
- 676 75.44378698
687
- 677 75.48005908
688
- 678 75.51622419
689
- 679 75.55228277
690
- 680 75.58823529
691
- 681 75.47723935
692
- 682 75.51319648
693
- 683 75.54904832
694
- 684 75.58479532
695
- 685 75.62043796
696
- 686 75.65597668
697
- 687 75.69141194
698
- 688 75.72674419
699
- 689 75.76197388
700
- 690 75.79710145
701
- 691 75.83212735
702
- 692 75.86705202
703
- 693 75.75757576
704
- 694 75.79250720
705
- 695 75.82733813
706
- 696 75.86206897
707
- 697 75.89670014
708
- 698 75.93123209
709
- 699 75.96566524
710
- 700 76.00000000
711
- 701 76.03423680
712
- 702 76.06837607
713
- 703 76.10241821
714
- 704 76.13636364
715
- 705 76.17021277
716
- 706 76.20396601
717
- 707 76.23762376
718
- 708 76.27118644
719
- 709 76.30465444
720
- 710 76.33802817
721
- 711 76.23066104
722
- 712 76.26404494
723
- 713 76.29733520
724
- 714 76.33053221
725
- 715 76.22377622
726
- 716 76.25698324
727
- 717 76.29009763
728
- 718 76.32311978
729
- 719 76.35605007
730
- 720 76.38888889
731
- 721 76.42163662
732
- 722 76.45429363
733
- 723 76.48686030
734
- 724 76.51933702
735
- 725 76.55172414
736
- 726 76.58402204
737
- 727 76.61623109
738
- 728 76.64835165
739
- 729 76.54320988
740
- 730 76.57534247
741
- 731 76.60738714
742
- 732 76.50273224
743
- 733 76.39836289
744
- 734 76.43051771
745
- 735 76.32653061
746
- 736 76.35869565
747
- 737 76.25508820
748
- 738 76.15176152
749
- 739 76.04871448
750
- 740 76.08108108
751
- 741 76.11336032
752
- 742 76.14555256
753
- 743 76.17765814
754
- 744 76.20967742
755
- 745 76.10738255
756
- 746 76.13941019
757
- 747 76.03748327
758
- 748 76.06951872
759
- 749 76.10146862
760
- 750 76.13333333
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q3_K_S.gguf (version GGUF V3 (latest))
4
 
5
+ 750 74.00000000
 
 
 
 
6
 
7
+
8
+ llama_perf_context_print: load time = 346.89 ms
9
+ llama_perf_context_print: prompt eval time = 471049.03 ms / 125576 tokens ( 3.75 ms per token, 266.59 tokens per second)
10
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
11
+ llama_perf_context_print: total time = 476803.59 ms / 125577 tokens
12
+ ggml_metal_free: deallocating
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scores/Watt-Tool-8B-q3_k_s.mmlu CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 1548 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 1548 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 50.00000000
13
- 3 33.33333333
14
- 4 50.00000000
15
- 5 40.00000000
16
- 6 33.33333333
17
- 7 42.85714286
18
- 8 50.00000000
19
- 9 44.44444444
20
- 10 50.00000000
21
- 11 45.45454545
22
- 12 41.66666667
23
- 13 38.46153846
24
- 14 35.71428571
25
- 15 40.00000000
26
- 16 43.75000000
27
- 17 47.05882353
28
- 18 44.44444444
29
- 19 47.36842105
30
- 20 50.00000000
31
- 21 52.38095238
32
- 22 50.00000000
33
- 23 47.82608696
34
- 24 50.00000000
35
- 25 52.00000000
36
- 26 50.00000000
37
- 27 51.85185185
38
- 28 53.57142857
39
- 29 55.17241379
40
- 30 56.66666667
41
- 31 58.06451613
42
- 32 59.37500000
43
- 33 57.57575758
44
- 34 58.82352941
45
- 35 60.00000000
46
- 36 61.11111111
47
- 37 59.45945946
48
- 38 60.52631579
49
- 39 58.97435897
50
- 40 60.00000000
51
- 41 58.53658537
52
- 42 57.14285714
53
- 43 58.13953488
54
- 44 56.81818182
55
- 45 55.55555556
56
- 46 54.34782609
57
- 47 53.19148936
58
- 48 54.16666667
59
- 49 53.06122449
60
- 50 52.00000000
61
- 51 50.98039216
62
- 52 50.00000000
63
- 53 49.05660377
64
- 54 48.14814815
65
- 55 49.09090909
66
- 56 50.00000000
67
- 57 50.87719298
68
- 58 51.72413793
69
- 59 52.54237288
70
- 60 51.66666667
71
- 61 52.45901639
72
- 62 51.61290323
73
- 63 50.79365079
74
- 64 50.00000000
75
- 65 49.23076923
76
- 66 50.00000000
77
- 67 49.25373134
78
- 68 48.52941176
79
- 69 47.82608696
80
- 70 48.57142857
81
- 71 49.29577465
82
- 72 50.00000000
83
- 73 49.31506849
84
- 74 50.00000000
85
- 75 50.66666667
86
- 76 51.31578947
87
- 77 51.94805195
88
- 78 52.56410256
89
- 79 51.89873418
90
- 80 52.50000000
91
- 81 53.08641975
92
- 82 52.43902439
93
- 83 53.01204819
94
- 84 53.57142857
95
- 85 52.94117647
96
- 86 52.32558140
97
- 87 52.87356322
98
- 88 53.40909091
99
- 89 52.80898876
100
- 90 52.22222222
101
- 91 51.64835165
102
- 92 51.08695652
103
- 93 50.53763441
104
- 94 50.00000000
105
- 95 49.47368421
106
- 96 50.00000000
107
- 97 50.51546392
108
- 98 50.00000000
109
- 99 50.50505051
110
- 100 50.00000000
111
- 101 49.50495050
112
- 102 49.01960784
113
- 103 48.54368932
114
- 104 49.03846154
115
- 105 49.52380952
116
- 106 50.00000000
117
- 107 49.53271028
118
- 108 49.07407407
119
- 109 48.62385321
120
- 110 48.18181818
121
- 111 48.64864865
122
- 112 49.10714286
123
- 113 48.67256637
124
- 114 49.12280702
125
- 115 49.56521739
126
- 116 50.00000000
127
- 117 50.42735043
128
- 118 50.00000000
129
- 119 50.42016807
130
- 120 50.00000000
131
- 121 50.41322314
132
- 122 50.00000000
133
- 123 49.59349593
134
- 124 50.00000000
135
- 125 50.40000000
136
- 126 50.00000000
137
- 127 49.60629921
138
- 128 49.21875000
139
- 129 48.83720930
140
- 130 48.46153846
141
- 131 48.85496183
142
- 132 49.24242424
143
- 133 48.87218045
144
- 134 48.50746269
145
- 135 48.14814815
146
- 136 48.52941176
147
- 137 48.17518248
148
- 138 47.82608696
149
- 139 48.20143885
150
- 140 48.57142857
151
- 141 48.93617021
152
- 142 49.29577465
153
- 143 49.65034965
154
- 144 49.30555556
155
- 145 48.96551724
156
- 146 49.31506849
157
- 147 48.97959184
158
- 148 49.32432432
159
- 149 49.66442953
160
- 150 49.33333333
161
- 151 49.00662252
162
- 152 49.34210526
163
- 153 49.67320261
164
- 154 49.35064935
165
- 155 49.03225806
166
- 156 48.71794872
167
- 157 49.04458599
168
- 158 49.36708861
169
- 159 49.05660377
170
- 160 49.37500000
171
- 161 49.06832298
172
- 162 48.76543210
173
- 163 48.46625767
174
- 164 48.17073171
175
- 165 48.48484848
176
- 166 48.19277108
177
- 167 48.50299401
178
- 168 48.21428571
179
- 169 47.92899408
180
- 170 47.64705882
181
- 171 47.36842105
182
- 172 47.09302326
183
- 173 46.82080925
184
- 174 46.55172414
185
- 175 46.28571429
186
- 176 46.59090909
187
- 177 46.32768362
188
- 178 46.06741573
189
- 179 46.36871508
190
- 180 46.11111111
191
- 181 45.85635359
192
- 182 45.60439560
193
- 183 45.90163934
194
- 184 45.65217391
195
- 185 45.94594595
196
- 186 46.23655914
197
- 187 45.98930481
198
- 188 46.27659574
199
- 189 46.56084656
200
- 190 46.31578947
201
- 191 46.59685864
202
- 192 46.87500000
203
- 193 46.63212435
204
- 194 46.90721649
205
- 195 47.17948718
206
- 196 47.44897959
207
- 197 47.20812183
208
- 198 47.47474747
209
- 199 47.23618090
210
- 200 47.00000000
211
- 201 46.76616915
212
- 202 46.53465347
213
- 203 46.30541872
214
- 204 46.07843137
215
- 205 46.34146341
216
- 206 46.11650485
217
- 207 46.37681159
218
- 208 46.15384615
219
- 209 45.93301435
220
- 210 45.71428571
221
- 211 45.97156398
222
- 212 45.75471698
223
- 213 46.00938967
224
- 214 45.79439252
225
- 215 46.04651163
226
- 216 46.29629630
227
- 217 46.08294931
228
- 218 46.33027523
229
- 219 46.57534247
230
- 220 46.81818182
231
- 221 46.60633484
232
- 222 46.39639640
233
- 223 46.63677130
234
- 224 46.42857143
235
- 225 46.66666667
236
- 226 46.46017699
237
- 227 46.25550661
238
- 228 46.49122807
239
- 229 46.28820961
240
- 230 46.08695652
241
- 231 45.88744589
242
- 232 45.68965517
243
- 233 45.49356223
244
- 234 45.72649573
245
- 235 45.53191489
246
- 236 45.33898305
247
- 237 45.14767932
248
- 238 44.95798319
249
- 239 44.76987448
250
- 240 45.00000000
251
- 241 44.81327801
252
- 242 45.04132231
253
- 243 45.26748971
254
- 244 45.08196721
255
- 245 45.30612245
256
- 246 45.12195122
257
- 247 44.93927126
258
- 248 45.16129032
259
- 249 44.97991968
260
- 250 44.80000000
261
- 251 44.62151394
262
- 252 44.44444444
263
- 253 44.26877470
264
- 254 44.09448819
265
- 255 43.92156863
266
- 256 44.14062500
267
- 257 44.35797665
268
- 258 44.18604651
269
- 259 44.01544402
270
- 260 44.23076923
271
- 261 44.06130268
272
- 262 43.89312977
273
- 263 43.72623574
274
- 264 43.56060606
275
- 265 43.77358491
276
- 266 43.98496241
277
- 267 43.82022472
278
- 268 43.65671642
279
- 269 43.86617100
280
- 270 44.07407407
281
- 271 43.91143911
282
- 272 43.75000000
283
- 273 43.58974359
284
- 274 43.43065693
285
- 275 43.27272727
286
- 276 43.47826087
287
- 277 43.68231047
288
- 278 43.52517986
289
- 279 43.36917563
290
- 280 43.21428571
291
- 281 43.06049822
292
- 282 42.90780142
293
- 283 42.75618375
294
- 284 42.60563380
295
- 285 42.45614035
296
- 286 42.30769231
297
- 287 42.50871080
298
- 288 42.36111111
299
- 289 42.21453287
300
- 290 42.41379310
301
- 291 42.61168385
302
- 292 42.80821918
303
- 293 42.66211604
304
- 294 42.51700680
305
- 295 42.37288136
306
- 296 42.22972973
307
- 297 42.08754209
308
- 298 42.28187919
309
- 299 42.47491639
310
- 300 42.66666667
311
- 301 42.52491694
312
- 302 42.38410596
313
- 303 42.24422442
314
- 304 42.10526316
315
- 305 42.29508197
316
- 306 42.15686275
317
- 307 42.34527687
318
- 308 42.20779221
319
- 309 42.07119741
320
- 310 41.93548387
321
- 311 41.80064309
322
- 312 41.98717949
323
- 313 42.17252396
324
- 314 42.03821656
325
- 315 41.90476190
326
- 316 41.77215190
327
- 317 41.95583596
328
- 318 41.82389937
329
- 319 42.00626959
330
- 320 41.87500000
331
- 321 41.74454829
332
- 322 41.92546584
333
- 323 41.79566563
334
- 324 41.66666667
335
- 325 41.84615385
336
- 326 41.71779141
337
- 327 41.89602446
338
- 328 41.76829268
339
- 329 41.64133739
340
- 330 41.51515152
341
- 331 41.38972810
342
- 332 41.26506024
343
- 333 41.44144144
344
- 334 41.31736527
345
- 335 41.49253731
346
- 336 41.36904762
347
- 337 41.24629080
348
- 338 41.12426036
349
- 339 41.00294985
350
- 340 41.17647059
351
- 341 41.05571848
352
- 342 41.22807018
353
- 343 41.10787172
354
- 344 40.98837209
355
- 345 40.86956522
356
- 346 41.04046243
357
- 347 41.21037464
358
- 348 41.09195402
359
- 349 41.26074499
360
- 350 41.42857143
361
- 351 41.31054131
362
- 352 41.47727273
363
- 353 41.35977337
364
- 354 41.24293785
365
- 355 41.12676056
366
- 356 41.29213483
367
- 357 41.17647059
368
- 358 41.06145251
369
- 359 40.94707521
370
- 360 41.11111111
371
- 361 40.99722992
372
- 362 41.16022099
373
- 363 41.04683196
374
- 364 40.93406593
375
- 365 40.82191781
376
- 366 40.98360656
377
- 367 40.87193460
378
- 368 40.76086957
379
- 369 40.65040650
380
- 370 40.81081081
381
- 371 40.70080863
382
- 372 40.59139785
383
- 373 40.48257373
384
- 374 40.37433155
385
- 375 40.53333333
386
- 376 40.42553191
387
- 377 40.58355438
388
- 378 40.74074074
389
- 379 40.63324538
390
- 380 40.52631579
391
- 381 40.68241470
392
- 382 40.83769634
393
- 383 40.99216710
394
- 384 40.88541667
395
- 385 41.03896104
396
- 386 40.93264249
397
- 387 40.82687339
398
- 388 40.72164948
399
- 389 40.87403599
400
- 390 41.02564103
401
- 391 41.17647059
402
- 392 41.32653061
403
- 393 41.22137405
404
- 394 41.11675127
405
- 395 41.26582278
406
- 396 41.41414141
407
- 397 41.30982368
408
- 398 41.20603015
409
- 399 41.35338346
410
- 400 41.25000000
411
- 401 41.39650873
412
- 402 41.29353234
413
- 403 41.19106700
414
- 404 41.33663366
415
- 405 41.23456790
416
- 406 41.13300493
417
- 407 41.03194103
418
- 408 41.17647059
419
- 409 41.32029340
420
- 410 41.21951220
421
- 411 41.11922141
422
- 412 41.01941748
423
- 413 40.92009685
424
- 414 40.82125604
425
- 415 40.72289157
426
- 416 40.86538462
427
- 417 41.00719424
428
- 418 40.90909091
429
- 419 41.05011933
430
- 420 40.95238095
431
- 421 41.09263658
432
- 422 40.99526066
433
- 423 40.89834515
434
- 424 41.03773585
435
- 425 40.94117647
436
- 426 40.84507042
437
- 427 40.74941452
438
- 428 40.65420561
439
- 429 40.79254079
440
- 430 40.69767442
441
- 431 40.83526682
442
- 432 40.97222222
443
- 433 40.87759815
444
- 434 40.78341014
445
- 435 40.68965517
446
- 436 40.59633028
447
- 437 40.50343249
448
- 438 40.63926941
449
- 439 40.77448747
450
- 440 40.68181818
451
- 441 40.58956916
452
- 442 40.49773756
453
- 443 40.40632054
454
- 444 40.31531532
455
- 445 40.22471910
456
- 446 40.13452915
457
- 447 40.04474273
458
- 448 39.95535714
459
- 449 39.86636971
460
- 450 39.77777778
461
- 451 39.68957871
462
- 452 39.60176991
463
- 453 39.73509934
464
- 454 39.64757709
465
- 455 39.78021978
466
- 456 39.91228070
467
- 457 40.04376368
468
- 458 39.95633188
469
- 459 39.86928105
470
- 460 40.00000000
471
- 461 39.91323210
472
- 462 39.82683983
473
- 463 39.95680346
474
- 464 40.08620690
475
- 465 40.00000000
476
- 466 39.91416309
477
- 467 40.04282655
478
- 468 40.17094017
479
- 469 40.29850746
480
- 470 40.42553191
481
- 471 40.33970276
482
- 472 40.25423729
483
- 473 40.38054968
484
- 474 40.29535865
485
- 475 40.21052632
486
- 476 40.33613445
487
- 477 40.25157233
488
- 478 40.37656904
489
- 479 40.50104384
490
- 480 40.62500000
491
- 481 40.54054054
492
- 482 40.45643154
493
- 483 40.37267081
494
- 484 40.28925620
495
- 485 40.20618557
496
- 486 40.12345679
497
- 487 40.24640657
498
- 488 40.16393443
499
- 489 40.08179959
500
- 490 40.20408163
501
- 491 40.12219959
502
- 492 40.04065041
503
- 493 39.95943205
504
- 494 40.08097166
505
- 495 40.20202020
506
- 496 40.32258065
507
- 497 40.24144869
508
- 498 40.16064257
509
- 499 40.08016032
510
- 500 40.00000000
511
- 501 39.92015968
512
- 502 39.84063745
513
- 503 39.76143141
514
- 504 39.68253968
515
- 505 39.60396040
516
- 506 39.72332016
517
- 507 39.64497041
518
- 508 39.76377953
519
- 509 39.88212181
520
- 510 40.00000000
521
- 511 39.92172211
522
- 512 39.84375000
523
- 513 39.76608187
524
- 514 39.68871595
525
- 515 39.61165049
526
- 516 39.72868217
527
- 517 39.84526112
528
- 518 39.96138996
529
- 519 39.88439306
530
- 520 39.80769231
531
- 521 39.73128599
532
- 522 39.84674330
533
- 523 39.77055449
534
- 524 39.69465649
535
- 525 39.61904762
536
- 526 39.54372624
537
- 527 39.46869070
538
- 528 39.39393939
539
- 529 39.50850662
540
- 530 39.62264151
541
- 531 39.54802260
542
- 532 39.47368421
543
- 533 39.39962477
544
- 534 39.51310861
545
- 535 39.62616822
546
- 536 39.55223881
547
- 537 39.66480447
548
- 538 39.77695167
549
- 539 39.70315399
550
- 540 39.62962963
551
- 541 39.55637708
552
- 542 39.66789668
553
- 543 39.59484346
554
- 544 39.70588235
555
- 545 39.63302752
556
- 546 39.56043956
557
- 547 39.48811700
558
- 548 39.59854015
559
- 549 39.52641166
560
- 550 39.45454545
561
- 551 39.56442831
562
- 552 39.49275362
563
- 553 39.42133816
564
- 554 39.35018051
565
- 555 39.27927928
566
- 556 39.38848921
567
- 557 39.49730700
568
- 558 39.60573477
569
- 559 39.53488372
570
- 560 39.46428571
571
- 561 39.39393939
572
- 562 39.50177936
573
- 563 39.60923623
574
- 564 39.53900709
575
- 565 39.64601770
576
- 566 39.57597173
577
- 567 39.50617284
578
- 568 39.61267606
579
- 569 39.54305800
580
- 570 39.47368421
581
- 571 39.40455342
582
- 572 39.51048951
583
- 573 39.44153578
584
- 574 39.37282230
585
- 575 39.30434783
586
- 576 39.40972222
587
- 577 39.51473137
588
- 578 39.44636678
589
- 579 39.37823834
590
- 580 39.48275862
591
- 581 39.41480207
592
- 582 39.34707904
593
- 583 39.45111492
594
- 584 39.55479452
595
- 585 39.65811966
596
- 586 39.76109215
597
- 587 39.69335605
598
- 588 39.62585034
599
- 589 39.55857385
600
- 590 39.49152542
601
- 591 39.42470389
602
- 592 39.35810811
603
- 593 39.46037099
604
- 594 39.56228956
605
- 595 39.49579832
606
- 596 39.59731544
607
- 597 39.53098827
608
- 598 39.46488294
609
- 599 39.56594324
610
- 600 39.50000000
611
- 601 39.43427621
612
- 602 39.36877076
613
- 603 39.46932007
614
- 604 39.40397351
615
- 605 39.33884298
616
- 606 39.27392739
617
- 607 39.37397035
618
- 608 39.30921053
619
- 609 39.24466338
620
- 610 39.18032787
621
- 611 39.27986907
622
- 612 39.21568627
623
- 613 39.31484502
624
- 614 39.25081433
625
- 615 39.18699187
626
- 616 39.12337662
627
- 617 39.22204214
628
- 618 39.15857605
629
- 619 39.09531502
630
- 620 39.03225806
631
- 621 38.96940419
632
- 622 38.90675241
633
- 623 38.84430177
634
- 624 38.78205128
635
- 625 38.72000000
636
- 626 38.81789137
637
- 627 38.75598086
638
- 628 38.69426752
639
- 629 38.79173291
640
- 630 38.88888889
641
- 631 38.98573693
642
- 632 39.08227848
643
- 633 39.02053712
644
- 634 38.95899054
645
- 635 38.89763780
646
- 636 38.83647799
647
- 637 38.77551020
648
- 638 38.71473354
649
- 639 38.81064163
650
- 640 38.90625000
651
- 641 38.84555382
652
- 642 38.78504673
653
- 643 38.72472784
654
- 644 38.81987578
655
- 645 38.75968992
656
- 646 38.69969040
657
- 647 38.63987635
658
- 648 38.58024691
659
- 649 38.52080123
660
- 650 38.46153846
661
- 651 38.40245776
662
- 652 38.49693252
663
- 653 38.43797856
664
- 654 38.53211009
665
- 655 38.47328244
666
- 656 38.41463415
667
- 657 38.35616438
668
- 658 38.44984802
669
- 659 38.39150228
670
- 660 38.33333333
671
- 661 38.27534039
672
- 662 38.21752266
673
- 663 38.15987934
674
- 664 38.10240964
675
- 665 38.04511278
676
- 666 37.98798799
677
- 667 38.08095952
678
- 668 38.17365269
679
- 669 38.26606876
680
- 670 38.35820896
681
- 671 38.30104322
682
- 672 38.24404762
683
- 673 38.18722140
684
- 674 38.13056380
685
- 675 38.07407407
686
- 676 38.16568047
687
- 677 38.10930576
688
- 678 38.20058997
689
- 679 38.14432990
690
- 680 38.23529412
691
- 681 38.32599119
692
- 682 38.26979472
693
- 683 38.21376281
694
- 684 38.30409357
695
- 685 38.24817518
696
- 686 38.33819242
697
- 687 38.28238719
698
- 688 38.22674419
699
- 689 38.17126270
700
- 690 38.11594203
701
- 691 38.20549928
702
- 692 38.15028902
703
- 693 38.23953824
704
- 694 38.18443804
705
- 695 38.12949640
706
- 696 38.21839080
707
- 697 38.16355811
708
- 698 38.25214900
709
- 699 38.34048641
710
- 700 38.28571429
711
- 701 38.37375178
712
- 702 38.46153846
713
- 703 38.54907539
714
- 704 38.49431818
715
- 705 38.43971631
716
- 706 38.38526912
717
- 707 38.33097595
718
- 708 38.27683616
719
- 709 38.22284908
720
- 710 38.16901408
721
- 711 38.25597750
722
- 712 38.34269663
723
- 713 38.42917251
724
- 714 38.51540616
725
- 715 38.46153846
726
- 716 38.40782123
727
- 717 38.35425384
728
- 718 38.30083565
729
- 719 38.38664812
730
- 720 38.33333333
731
- 721 38.28016644
732
- 722 38.22714681
733
- 723 38.31258645
734
- 724 38.25966851
735
- 725 38.34482759
736
- 726 38.29201102
737
- 727 38.23933975
738
- 728 38.18681319
739
- 729 38.13443073
740
- 730 38.08219178
741
- 731 38.03009576
742
- 732 37.97814208
743
- 733 38.06275580
744
- 734 38.01089918
745
- 735 37.95918367
746
- 736 37.90760870
747
- 737 37.99185889
748
- 738 38.07588076
749
- 739 38.15967524
750
- 740 38.10810811
751
- 741 38.05668016
752
- 742 38.14016173
753
- 743 38.22341857
754
- 744 38.30645161
755
- 745 38.25503356
756
- 746 38.20375335
757
- 747 38.15261044
758
- 748 38.10160428
759
- 749 38.05073431
760
- 750 38.13333333
761
 
762
- Final result: 38.1333 +/- 1.7748
763
- Random chance: 25.0000 +/- 1.5822
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q3_K_S.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 37.2000 +/- 1.7661
6
+ Random chance: 25.0000 +/- 1.5822
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 312.54 ms
10
+ llama_perf_context_print: prompt eval time = 279002.34 ms / 69611 tokens ( 4.01 ms per token, 249.50 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 281794.77 ms / 69612 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-q3_k_s.ppx CHANGED
@@ -1,37 +1,37 @@
1
  ====== Perplexity statistics ======
2
- Mean PPL(Q) : 10.242604 ± 0.066586
3
- Mean PPL(base) : 7.534124 ± 0.048206
4
- Cor(ln(PPL(Q)), ln(PPL(base))): 94.81%
5
- Mean ln(PPL(Q)/PPL(base)) : 0.307113 ± 0.002080
6
- Mean PPL(Q)/PPL(base) : 1.359495 ± 0.002827
7
- Mean PPL(Q)-PPL(base) : 2.708480 ± 0.025898
8
 
9
  ====== KL divergence statistics ======
10
- Mean KLD: 0.276176 ± 0.000860
11
- Maximum KLD: 8.462747
12
- 99.9% KLD: 3.304338
13
- 99.0% KLD: 1.622670
14
- 99.0% KLD: 1.622670
15
- Median KLD: 0.211115
16
- 10.0% KLD: 0.017779
17
- 5.0% KLD: 0.004912
18
- 1.0% KLD: 0.000565
19
- Minimum KLD: 0.000002
20
 
21
  ====== Token probability statistics ======
22
- Mean Δp: -7.071 ± 0.041 %
23
- Maximum Δp: 93.414%
24
- 99.9% Δp: 45.356%
25
- 99.0% Δp: 24.292%
26
- 95.0% Δp: 9.695%
27
- 90.0% Δp: 4.049%
28
- 75.0% Δp: 0.041%
29
- Median Δp: -1.661%
30
- 25.0% Δp: -11.538%
31
- 10.0% Δp: -26.617%
32
- 5.0% Δp: -38.289%
33
- 1.0% Δp: -66.295%
34
- 0.1% Δp: -86.075%
35
- Minimum Δp: -99.772%
36
- RMS Δp : 17.133 ± 0.061 %
37
- Same top p: 73.659 ± 0.116 %
 
1
  ====== Perplexity statistics ======
2
+ Mean PPL(Q) : 9.798719 ± 0.061509
3
+ Mean PPL(base) : 7.237090 ± 0.045539
4
+ Cor(ln(PPL(Q)), ln(PPL(base))): 94.51%
5
+ Mean ln(PPL(Q)/PPL(base)) : 0.303032 ± 0.002083
6
+ Mean PPL(Q)/PPL(base) : 1.353958 ± 0.002821
7
+ Mean PPL(Q)-PPL(base) : 2.561629 ± 0.023725
8
 
9
  ====== KL divergence statistics ======
10
+ Mean KLD: 0.285340 ± 0.000933
11
+ Maximum KLD: 8.015823
12
+ 99.9% KLD: 3.395762
13
+ 99.0% KLD: 1.885579
14
+ 99.0% KLD: 1.885579
15
+ Median KLD: 0.207489
16
+ 10.0% KLD: 0.018084
17
+ 5.0% KLD: 0.005005
18
+ 1.0% KLD: 0.000535
19
+ Minimum KLD: 0.000000
20
 
21
  ====== Token probability statistics ======
22
+ Mean Δp: -7.606 ± 0.042 %
23
+ Maximum Δp: 79.040%
24
+ 99.9% Δp: 41.873%
25
+ 99.0% Δp: 22.236%
26
+ 95.0% Δp: 8.510%
27
+ 90.0% Δp: 3.272%
28
+ 75.0% Δp: 0.019%
29
+ Median Δp: -1.956%
30
+ 25.0% Δp: -11.831%
31
+ 10.0% Δp: -26.778%
32
+ 5.0% Δp: -39.549%
33
+ 1.0% Δp: -72.463%
34
+ 0.1% Δp: -89.273%
35
+ Minimum Δp: -99.255%
36
+ RMS Δp : 17.751 ± 0.067 %
37
+ Same top p: 74.688 ± 0.115 %
scores/Watt-Tool-8B-q3_k_s.tqa CHANGED
@@ -1,763 +1,13 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 817 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 817 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
 
10
- task acc_norm
11
- 1 100.00000000
12
- 2 50.00000000
13
- 3 33.33333333
14
- 4 25.00000000
15
- 5 40.00000000
16
- 6 33.33333333
17
- 7 28.57142857
18
- 8 37.50000000
19
- 9 33.33333333
20
- 10 30.00000000
21
- 11 27.27272727
22
- 12 33.33333333
23
- 13 38.46153846
24
- 14 35.71428571
25
- 15 33.33333333
26
- 16 37.50000000
27
- 17 35.29411765
28
- 18 33.33333333
29
- 19 31.57894737
30
- 20 30.00000000
31
- 21 28.57142857
32
- 22 27.27272727
33
- 23 26.08695652
34
- 24 29.16666667
35
- 25 28.00000000
36
- 26 26.92307692
37
- 27 25.92592593
38
- 28 25.00000000
39
- 29 27.58620690
40
- 30 26.66666667
41
- 31 25.80645161
42
- 32 28.12500000
43
- 33 30.30303030
44
- 34 29.41176471
45
- 35 28.57142857
46
- 36 27.77777778
47
- 37 29.72972973
48
- 38 28.94736842
49
- 39 30.76923077
50
- 40 30.00000000
51
- 41 29.26829268
52
- 42 28.57142857
53
- 43 30.23255814
54
- 44 31.81818182
55
- 45 33.33333333
56
- 46 32.60869565
57
- 47 34.04255319
58
- 48 35.41666667
59
- 49 34.69387755
60
- 50 36.00000000
61
- 51 35.29411765
62
- 52 34.61538462
63
- 53 33.96226415
64
- 54 33.33333333
65
- 55 34.54545455
66
- 56 33.92857143
67
- 57 33.33333333
68
- 58 32.75862069
69
- 59 33.89830508
70
- 60 35.00000000
71
- 61 34.42622951
72
- 62 35.48387097
73
- 63 34.92063492
74
- 64 34.37500000
75
- 65 33.84615385
76
- 66 34.84848485
77
- 67 34.32835821
78
- 68 33.82352941
79
- 69 34.78260870
80
- 70 34.28571429
81
- 71 33.80281690
82
- 72 34.72222222
83
- 73 34.24657534
84
- 74 33.78378378
85
- 75 33.33333333
86
- 76 32.89473684
87
- 77 32.46753247
88
- 78 32.05128205
89
- 79 31.64556962
90
- 80 32.50000000
91
- 81 32.09876543
92
- 82 32.92682927
93
- 83 32.53012048
94
- 84 32.14285714
95
- 85 31.76470588
96
- 86 31.39534884
97
- 87 31.03448276
98
- 88 30.68181818
99
- 89 31.46067416
100
- 90 31.11111111
101
- 91 31.86813187
102
- 92 31.52173913
103
- 93 31.18279570
104
- 94 30.85106383
105
- 95 31.57894737
106
- 96 32.29166667
107
- 97 32.98969072
108
- 98 32.65306122
109
- 99 33.33333333
110
- 100 34.00000000
111
- 101 34.65346535
112
- 102 34.31372549
113
- 103 33.98058252
114
- 104 33.65384615
115
- 105 33.33333333
116
- 106 33.01886792
117
- 107 33.64485981
118
- 108 34.25925926
119
- 109 33.94495413
120
- 110 34.54545455
121
- 111 34.23423423
122
- 112 34.82142857
123
- 113 35.39823009
124
- 114 35.96491228
125
- 115 35.65217391
126
- 116 36.20689655
127
- 117 35.89743590
128
- 118 35.59322034
129
- 119 35.29411765
130
- 120 35.83333333
131
- 121 36.36363636
132
- 122 36.06557377
133
- 123 35.77235772
134
- 124 36.29032258
135
- 125 36.80000000
136
- 126 37.30158730
137
- 127 37.00787402
138
- 128 36.71875000
139
- 129 37.20930233
140
- 130 36.92307692
141
- 131 36.64122137
142
- 132 36.36363636
143
- 133 36.09022556
144
- 134 35.82089552
145
- 135 36.29629630
146
- 136 36.02941176
147
- 137 36.49635036
148
- 138 36.95652174
149
- 139 37.41007194
150
- 140 37.14285714
151
- 141 36.87943262
152
- 142 36.61971831
153
- 143 36.36363636
154
- 144 36.80555556
155
- 145 36.55172414
156
- 146 36.30136986
157
- 147 36.05442177
158
- 148 35.81081081
159
- 149 35.57046980
160
- 150 35.33333333
161
- 151 35.76158940
162
- 152 35.52631579
163
- 153 35.29411765
164
- 154 35.06493506
165
- 155 35.48387097
166
- 156 35.89743590
167
- 157 35.66878981
168
- 158 35.44303797
169
- 159 35.22012579
170
- 160 35.00000000
171
- 161 35.40372671
172
- 162 35.80246914
173
- 163 35.58282209
174
- 164 35.36585366
175
- 165 35.75757576
176
- 166 36.14457831
177
- 167 35.92814371
178
- 168 36.30952381
179
- 169 36.09467456
180
- 170 35.88235294
181
- 171 35.67251462
182
- 172 35.46511628
183
- 173 35.26011561
184
- 174 35.63218391
185
- 175 35.42857143
186
- 176 35.22727273
187
- 177 35.02824859
188
- 178 34.83146067
189
- 179 35.19553073
190
- 180 35.00000000
191
- 181 34.80662983
192
- 182 34.61538462
193
- 183 34.42622951
194
- 184 34.23913043
195
- 185 34.05405405
196
- 186 34.40860215
197
- 187 34.22459893
198
- 188 34.04255319
199
- 189 33.86243386
200
- 190 33.68421053
201
- 191 33.50785340
202
- 192 33.33333333
203
- 193 33.16062176
204
- 194 33.50515464
205
- 195 33.33333333
206
- 196 33.16326531
207
- 197 33.50253807
208
- 198 33.83838384
209
- 199 33.66834171
210
- 200 34.00000000
211
- 201 33.83084577
212
- 202 34.15841584
213
- 203 33.99014778
214
- 204 33.82352941
215
- 205 33.65853659
216
- 206 33.49514563
217
- 207 33.81642512
218
- 208 33.65384615
219
- 209 33.49282297
220
- 210 33.33333333
221
- 211 33.17535545
222
- 212 33.01886792
223
- 213 32.86384977
224
- 214 32.71028037
225
- 215 32.55813953
226
- 216 32.87037037
227
- 217 33.17972350
228
- 218 33.48623853
229
- 219 33.33333333
230
- 220 33.18181818
231
- 221 33.03167421
232
- 222 32.88288288
233
- 223 32.73542601
234
- 224 33.03571429
235
- 225 32.88888889
236
- 226 32.74336283
237
- 227 33.03964758
238
- 228 32.89473684
239
- 229 32.75109170
240
- 230 32.60869565
241
- 231 32.46753247
242
- 232 32.75862069
243
- 233 33.04721030
244
- 234 33.33333333
245
- 235 33.19148936
246
- 236 33.47457627
247
- 237 33.75527426
248
- 238 34.03361345
249
- 239 33.89121339
250
- 240 33.75000000
251
- 241 33.60995851
252
- 242 33.47107438
253
- 243 33.33333333
254
- 244 33.19672131
255
- 245 33.06122449
256
- 246 32.92682927
257
- 247 33.19838057
258
- 248 33.06451613
259
- 249 33.33333333
260
- 250 33.20000000
261
- 251 33.46613546
262
- 252 33.73015873
263
- 253 33.59683794
264
- 254 33.46456693
265
- 255 33.33333333
266
- 256 33.20312500
267
- 257 33.07392996
268
- 258 32.94573643
269
- 259 32.81853282
270
- 260 32.69230769
271
- 261 32.56704981
272
- 262 32.44274809
273
- 263 32.31939163
274
- 264 32.57575758
275
- 265 32.83018868
276
- 266 32.70676692
277
- 267 32.58426966
278
- 268 32.46268657
279
- 269 32.34200743
280
- 270 32.22222222
281
- 271 32.10332103
282
- 272 31.98529412
283
- 273 32.23443223
284
- 274 32.11678832
285
- 275 32.00000000
286
- 276 31.88405797
287
- 277 31.76895307
288
- 278 31.65467626
289
- 279 31.54121864
290
- 280 31.42857143
291
- 281 31.67259786
292
- 282 31.56028369
293
- 283 31.80212014
294
- 284 31.69014085
295
- 285 31.57894737
296
- 286 31.81818182
297
- 287 31.70731707
298
- 288 31.59722222
299
- 289 31.48788927
300
- 290 31.37931034
301
- 291 31.61512027
302
- 292 31.50684932
303
- 293 31.74061433
304
- 294 31.63265306
305
- 295 31.86440678
306
- 296 32.09459459
307
- 297 32.32323232
308
- 298 32.21476510
309
- 299 32.10702341
310
- 300 32.00000000
311
- 301 31.89368771
312
- 302 31.78807947
313
- 303 31.68316832
314
- 304 31.57894737
315
- 305 31.47540984
316
- 306 31.37254902
317
- 307 31.27035831
318
- 308 31.49350649
319
- 309 31.39158576
320
- 310 31.29032258
321
- 311 31.18971061
322
- 312 31.41025641
323
- 313 31.30990415
324
- 314 31.21019108
325
- 315 31.42857143
326
- 316 31.32911392
327
- 317 31.23028391
328
- 318 31.13207547
329
- 319 31.03448276
330
- 320 30.93750000
331
- 321 30.84112150
332
- 322 31.05590062
333
- 323 30.95975232
334
- 324 30.86419753
335
- 325 30.76923077
336
- 326 30.67484663
337
- 327 30.88685015
338
- 328 31.09756098
339
- 329 31.00303951
340
- 330 31.21212121
341
- 331 31.11782477
342
- 332 31.32530120
343
- 333 31.53153153
344
- 334 31.43712575
345
- 335 31.64179104
346
- 336 31.84523810
347
- 337 31.75074184
348
- 338 31.65680473
349
- 339 31.56342183
350
- 340 31.47058824
351
- 341 31.67155425
352
- 342 31.87134503
353
- 343 31.77842566
354
- 344 31.97674419
355
- 345 32.17391304
356
- 346 32.08092486
357
- 347 32.27665706
358
- 348 32.18390805
359
- 349 32.09169054
360
- 350 32.00000000
361
- 351 31.90883191
362
- 352 32.10227273
363
- 353 32.01133144
364
- 354 31.92090395
365
- 355 32.11267606
366
- 356 32.02247191
367
- 357 31.93277311
368
- 358 31.84357542
369
- 359 31.75487465
370
- 360 31.66666667
371
- 361 31.85595568
372
- 362 31.76795580
373
- 363 31.95592287
374
- 364 31.86813187
375
- 365 31.78082192
376
- 366 31.69398907
377
- 367 31.88010899
378
- 368 31.79347826
379
- 369 31.70731707
380
- 370 31.62162162
381
- 371 31.53638814
382
- 372 31.72043011
383
- 373 31.63538874
384
- 374 31.81818182
385
- 375 31.73333333
386
- 376 31.64893617
387
- 377 31.83023873
388
- 378 31.74603175
389
- 379 31.66226913
390
- 380 31.57894737
391
- 381 31.49606299
392
- 382 31.41361257
393
- 383 31.33159269
394
- 384 31.25000000
395
- 385 31.16883117
396
- 386 31.08808290
397
- 387 31.00775194
398
- 388 31.18556701
399
- 389 31.10539846
400
- 390 31.02564103
401
- 391 31.20204604
402
- 392 31.12244898
403
- 393 31.29770992
404
- 394 31.47208122
405
- 395 31.39240506
406
- 396 31.56565657
407
- 397 31.73803526
408
- 398 31.65829146
409
- 399 31.57894737
410
- 400 31.50000000
411
- 401 31.42144638
412
- 402 31.59203980
413
- 403 31.51364764
414
- 404 31.43564356
415
- 405 31.35802469
416
- 406 31.28078818
417
- 407 31.20393120
418
- 408 31.12745098
419
- 409 31.05134474
420
- 410 30.97560976
421
- 411 30.90024331
422
- 412 31.06796117
423
- 413 30.99273608
424
- 414 31.15942029
425
- 415 31.08433735
426
- 416 31.00961538
427
- 417 30.93525180
428
- 418 30.86124402
429
- 419 30.78758950
430
- 420 30.95238095
431
- 421 31.11638955
432
- 422 31.04265403
433
- 423 30.96926714
434
- 424 30.89622642
435
- 425 30.82352941
436
- 426 30.75117371
437
- 427 30.67915691
438
- 428 30.60747664
439
- 429 30.76923077
440
- 430 30.69767442
441
- 431 30.62645012
442
- 432 30.78703704
443
- 433 30.94688222
444
- 434 31.10599078
445
- 435 31.03448276
446
- 436 30.96330275
447
- 437 30.89244851
448
- 438 31.05022831
449
- 439 31.20728929
450
- 440 31.36363636
451
- 441 31.29251701
452
- 442 31.22171946
453
- 443 31.15124153
454
- 444 31.30630631
455
- 445 31.23595506
456
- 446 31.16591928
457
- 447 31.09619687
458
- 448 31.02678571
459
- 449 31.18040089
460
- 450 31.33333333
461
- 451 31.26385809
462
- 452 31.19469027
463
- 453 31.34657837
464
- 454 31.49779736
465
- 455 31.42857143
466
- 456 31.35964912
467
- 457 31.29102845
468
- 458 31.22270742
469
- 459 31.15468410
470
- 460 31.08695652
471
- 461 31.01952278
472
- 462 31.16883117
473
- 463 31.31749460
474
- 464 31.46551724
475
- 465 31.61290323
476
- 466 31.54506438
477
- 467 31.47751606
478
- 468 31.62393162
479
- 469 31.55650320
480
- 470 31.48936170
481
- 471 31.63481953
482
- 472 31.77966102
483
- 473 31.71247357
484
- 474 31.64556962
485
- 475 31.78947368
486
- 476 31.72268908
487
- 477 31.86582809
488
- 478 31.79916318
489
- 479 31.73277662
490
- 480 31.66666667
491
- 481 31.60083160
492
- 482 31.74273859
493
- 483 31.67701863
494
- 484 31.61157025
495
- 485 31.75257732
496
- 486 31.68724280
497
- 487 31.82751540
498
- 488 31.76229508
499
- 489 31.90184049
500
- 490 32.04081633
501
- 491 32.17922607
502
- 492 32.31707317
503
- 493 32.25152130
504
- 494 32.38866397
505
- 495 32.52525253
506
- 496 32.45967742
507
- 497 32.39436620
508
- 498 32.32931727
509
- 499 32.26452906
510
- 500 32.20000000
511
- 501 32.33532934
512
- 502 32.27091633
513
- 503 32.20675944
514
- 504 32.34126984
515
- 505 32.47524752
516
- 506 32.41106719
517
- 507 32.54437870
518
- 508 32.67716535
519
- 509 32.80943026
520
- 510 32.94117647
521
- 511 32.87671233
522
- 512 32.81250000
523
- 513 32.74853801
524
- 514 32.68482490
525
- 515 32.81553398
526
- 516 32.75193798
527
- 517 32.88201161
528
- 518 32.81853282
529
- 519 32.75529865
530
- 520 32.69230769
531
- 521 32.62955854
532
- 522 32.75862069
533
- 523 32.69598470
534
- 524 32.63358779
535
- 525 32.57142857
536
- 526 32.50950570
537
- 527 32.63757116
538
- 528 32.57575758
539
- 529 32.70321361
540
- 530 32.64150943
541
- 531 32.58003766
542
- 532 32.51879699
543
- 533 32.64540338
544
- 534 32.58426966
545
- 535 32.52336449
546
- 536 32.46268657
547
- 537 32.58845438
548
- 538 32.52788104
549
- 539 32.46753247
550
- 540 32.59259259
551
- 541 32.53234750
552
- 542 32.47232472
553
- 543 32.41252302
554
- 544 32.35294118
555
- 545 32.29357798
556
- 546 32.23443223
557
- 547 32.35831810
558
- 548 32.48175182
559
- 549 32.42258652
560
- 550 32.36363636
561
- 551 32.30490018
562
- 552 32.42753623
563
- 553 32.54972875
564
- 554 32.49097473
565
- 555 32.43243243
566
- 556 32.37410072
567
- 557 32.31597846
568
- 558 32.25806452
569
- 559 32.20035778
570
- 560 32.32142857
571
- 561 32.26381462
572
- 562 32.20640569
573
- 563 32.14920071
574
- 564 32.26950355
575
- 565 32.38938053
576
- 566 32.50883392
577
- 567 32.45149912
578
- 568 32.57042254
579
- 569 32.68892794
580
- 570 32.63157895
581
- 571 32.74956217
582
- 572 32.69230769
583
- 573 32.63525305
584
- 574 32.75261324
585
- 575 32.86956522
586
- 576 32.98611111
587
- 577 33.10225303
588
- 578 33.21799308
589
- 579 33.16062176
590
- 580 33.27586207
591
- 581 33.21858864
592
- 582 33.33333333
593
- 583 33.27615780
594
- 584 33.39041096
595
- 585 33.33333333
596
- 586 33.44709898
597
- 587 33.39011925
598
- 588 33.33333333
599
- 589 33.27674024
600
- 590 33.38983051
601
- 591 33.33333333
602
- 592 33.27702703
603
- 593 33.22091062
604
- 594 33.33333333
605
- 595 33.27731092
606
- 596 33.22147651
607
- 597 33.16582915
608
- 598 33.11036789
609
- 599 33.05509182
610
- 600 33.00000000
611
- 601 33.11148087
612
- 602 33.05647841
613
- 603 33.16749585
614
- 604 33.11258278
615
- 605 33.05785124
616
- 606 33.16831683
617
- 607 33.11367381
618
- 608 33.05921053
619
- 609 33.00492611
620
- 610 32.95081967
621
- 611 32.89689034
622
- 612 33.00653595
623
- 613 32.95269168
624
- 614 32.89902280
625
- 615 32.84552846
626
- 616 32.95454545
627
- 617 33.06320908
628
- 618 33.00970874
629
- 619 33.11793215
630
- 620 33.06451613
631
- 621 33.01127214
632
- 622 33.11897106
633
- 623 33.06581059
634
- 624 33.01282051
635
- 625 32.96000000
636
- 626 32.90734824
637
- 627 32.85486443
638
- 628 32.96178344
639
- 629 32.90937997
640
- 630 32.85714286
641
- 631 32.96354992
642
- 632 32.91139241
643
- 633 33.01737757
644
- 634 32.96529968
645
- 635 32.91338583
646
- 636 32.86163522
647
- 637 32.81004710
648
- 638 32.75862069
649
- 639 32.70735524
650
- 640 32.65625000
651
- 641 32.60530421
652
- 642 32.71028037
653
- 643 32.65940902
654
- 644 32.60869565
655
- 645 32.71317829
656
- 646 32.66253870
657
- 647 32.61205564
658
- 648 32.71604938
659
- 649 32.66563945
660
- 650 32.76923077
661
- 651 32.87250384
662
- 652 32.82208589
663
- 653 32.92496172
664
- 654 32.87461774
665
- 655 32.82442748
666
- 656 32.92682927
667
- 657 33.02891933
668
- 658 32.97872340
669
- 659 33.08042489
670
- 660 33.03030303
671
- 661 32.98033283
672
- 662 32.93051360
673
- 663 32.88084465
674
- 664 32.83132530
675
- 665 32.93233083
676
- 666 33.03303303
677
- 667 32.98350825
678
- 668 32.93413174
679
- 669 32.88490284
680
- 670 32.98507463
681
- 671 32.93591654
682
- 672 32.88690476
683
- 673 32.83803863
684
- 674 32.78931751
685
- 675 32.88888889
686
- 676 32.98816568
687
- 677 32.93943870
688
- 678 32.89085546
689
- 679 32.84241532
690
- 680 32.79411765
691
- 681 32.74596182
692
- 682 32.69794721
693
- 683 32.65007321
694
- 684 32.74853801
695
- 685 32.70072993
696
- 686 32.65306122
697
- 687 32.60553130
698
- 688 32.55813953
699
- 689 32.65602322
700
- 690 32.60869565
701
- 691 32.70622287
702
- 692 32.80346821
703
- 693 32.90043290
704
- 694 32.85302594
705
- 695 32.80575540
706
- 696 32.90229885
707
- 697 32.85509326
708
- 698 32.80802292
709
- 699 32.76108727
710
- 700 32.85714286
711
- 701 32.81027104
712
- 702 32.76353276
713
- 703 32.71692745
714
- 704 32.67045455
715
- 705 32.62411348
716
- 706 32.71954674
717
- 707 32.81471004
718
- 708 32.76836158
719
- 709 32.72214386
720
- 710 32.81690141
721
- 711 32.77074543
722
- 712 32.86516854
723
- 713 32.81907433
724
- 714 32.77310924
725
- 715 32.86713287
726
- 716 32.82122905
727
- 717 32.77545328
728
- 718 32.86908078
729
- 719 32.96244784
730
- 720 32.91666667
731
- 721 32.87101248
732
- 722 32.82548476
733
- 723 32.78008299
734
- 724 32.87292818
735
- 725 32.82758621
736
- 726 32.92011019
737
- 727 32.87482806
738
- 728 32.82967033
739
- 729 32.92181070
740
- 730 32.87671233
741
- 731 32.83173735
742
- 732 32.92349727
743
- 733 32.87858117
744
- 734 32.83378747
745
- 735 32.78911565
746
- 736 32.74456522
747
- 737 32.70013569
748
- 738 32.65582656
749
- 739 32.61163735
750
- 740 32.56756757
751
- 741 32.65856950
752
- 742 32.61455526
753
- 743 32.57065949
754
- 744 32.66129032
755
- 745 32.61744966
756
- 746 32.70777480
757
- 747 32.79785810
758
- 748 32.75401070
759
- 749 32.84379172
760
- 750 32.93333333
761
 
762
- Final result: 32.9333 +/- 1.7172
763
- Random chance: 19.8992 +/- 1.4588
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q3_K_S.gguf (version GGUF V3 (latest))
4
 
5
+ Final result: 35.6707 +/- 2.6490
6
+ Random chance: 28.6213 +/- 2.4995
 
 
 
7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
+ llama_perf_context_print: load time = 315.60 ms
10
+ llama_perf_context_print: prompt eval time = 83571.96 ms / 17852 tokens ( 4.68 ms per token, 213.61 tokens per second)
11
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
12
+ llama_perf_context_print: total time = 84826.79 ms / 17853 tokens
13
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-q3_k_s.wng CHANGED
@@ -1,760 +1,11 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
 
3
 
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- winogrande_score : loaded 1267 tasks from prompt.
6
- winogrande_score : selecting 750 random tasks
7
- winogrande_score : tokenizing selected tasks
8
- winogrande_score : calculating winogrande score over selected tasks.
9
- 1 100.0000 -3.153120 -4.076086 1 1
10
- 2 100.0000 -1.831665 -1.985842 1 1
11
- 3 100.0000 -2.855140 -2.601004 2 2
12
- 4 100.0000 -4.608931 -4.738229 1 1
13
- 5 80.0000 -2.688293 -3.217363 1 2
14
- 6 83.3333 -4.746247 -4.847733 1 1
15
- 7 85.7143 -4.456703 -3.199037 2 2
16
- 8 87.5000 -3.354345 -2.614721 2 2
17
- 9 88.8889 -3.856076 -5.679955 1 1
18
- 10 90.0000 -1.754814 -1.557218 2 2
19
- 11 90.9091 -9.263533 -4.266823 2 2
20
- 12 91.6667 -2.793560 -2.137211 2 2
21
- 13 92.3077 -3.460777 -4.629338 1 1
22
- 14 92.8571 -3.314338 -4.039227 1 1
23
- 15 86.6667 -4.979626 -6.693942 1 2
24
- 16 87.5000 -4.541503 -4.315021 2 2
25
- 17 82.3529 -6.936707 -7.047203 1 2
26
- 18 83.3333 -1.314946 -3.084000 1 1
27
- 19 84.2105 -3.995245 -5.270151 1 1
28
- 20 85.0000 -2.676099 -3.193691 1 1
29
- 21 85.7143 -1.974171 -3.122176 1 1
30
- 22 86.3636 -2.499765 -2.497687 2 2
31
- 23 86.9565 -4.980042 -5.129396 1 1
32
- 24 83.3333 -2.872607 -2.459736 2 1
33
- 25 84.0000 -2.057020 -1.961337 2 2
34
- 26 84.6154 -3.331482 -4.307706 1 1
35
- 27 85.1852 -5.309682 -4.938490 2 2
36
- 28 85.7143 -4.831486 -5.566896 1 1
37
- 29 82.7586 -4.373794 -4.328263 2 1
38
- 30 83.3333 -2.131480 -1.973988 2 2
39
- 31 83.8710 -4.726704 -5.128890 1 1
40
- 32 84.3750 -4.564570 -6.147904 1 1
41
- 33 84.8485 -4.447222 -5.684215 1 1
42
- 34 85.2941 -2.342756 -2.493780 1 1
43
- 35 85.7143 -4.933187 -4.774284 2 2
44
- 36 86.1111 -4.292021 -3.935918 2 2
45
- 37 86.4865 -2.969247 -3.102524 1 1
46
- 38 84.2105 -4.394711 -4.331513 2 1
47
- 39 84.6154 -5.291846 -4.674432 2 2
48
- 40 85.0000 -6.345221 -6.224534 2 2
49
- 41 85.3659 -2.099088 -2.605396 1 1
50
- 42 85.7143 -2.543126 -3.433228 1 1
51
- 43 86.0465 -3.411368 -3.300426 2 2
52
- 44 86.3636 -2.382605 -1.950659 2 2
53
- 45 86.6667 -0.072639 -11.633352 1 1
54
- 46 86.9565 -2.475190 -2.532517 1 1
55
- 47 85.1064 -3.780380 -4.296318 1 2
56
- 48 83.3333 -3.866758 -3.931443 1 2
57
- 49 81.6327 -2.579759 -1.730986 2 1
58
- 50 80.0000 -2.870125 -2.900514 1 2
59
- 51 80.3922 -2.614712 -3.143509 1 1
60
- 52 80.7692 -3.146950 -2.269897 2 2
61
- 53 81.1321 -3.848458 -4.377901 1 1
62
- 54 81.4815 -2.846338 -2.501442 2 2
63
- 55 81.8182 -3.021010 -2.717772 2 2
64
- 56 80.3571 -3.910343 -4.773771 1 2
65
- 57 78.9474 -2.500751 -2.632373 1 2
66
- 58 79.3103 -3.858269 -4.845475 1 1
67
- 59 79.6610 -3.588984 -3.582208 2 2
68
- 60 78.3333 -4.219373 -4.255674 1 2
69
- 61 78.6885 -2.978368 -3.284405 1 1
70
- 62 79.0323 -5.803391 -6.037661 1 1
71
- 63 79.3651 -4.896131 -4.734250 2 2
72
- 64 78.1250 -3.827759 -3.796322 2 1
73
- 65 78.4615 -5.626138 -4.962242 2 2
74
- 66 78.7879 -3.091298 -3.384047 1 1
75
- 67 79.1045 -3.404868 -2.675802 2 2
76
- 68 79.4118 -5.370503 -5.192732 2 2
77
- 69 79.7101 -4.732292 -6.661155 1 1
78
- 70 80.0000 -3.920010 -3.388202 2 2
79
- 71 78.8732 -3.051769 -2.615203 2 1
80
- 72 79.1667 -3.963734 -3.762297 2 2
81
- 73 79.4521 -2.504124 -2.609797 1 1
82
- 74 79.7297 -2.557906 -2.836032 1 1
83
- 75 80.0000 -1.671938 -2.711881 1 1
84
- 76 80.2632 -2.414078 -4.749568 1 1
85
- 77 80.5195 -3.020953 -2.880693 2 2
86
- 78 80.7692 -0.984942 -1.694134 1 1
87
- 79 79.7468 -2.386042 -2.632522 1 2
88
- 80 80.0000 -2.443093 -3.673740 1 1
89
- 81 80.2469 -6.087738 -4.611579 2 2
90
- 82 79.2683 -3.667204 -3.952948 1 2
91
- 83 79.5181 -6.050276 -5.404021 2 2
92
- 84 79.7619 -2.332475 -2.313838 2 2
93
- 85 80.0000 -4.883725 -5.234994 1 1
94
- 86 80.2326 -5.792475 -5.759270 2 2
95
- 87 79.3103 -1.792545 -1.646764 2 1
96
- 88 78.4091 -5.097758 -4.559705 2 1
97
- 89 78.6517 -3.495170 -3.523257 1 1
98
- 90 78.8889 -4.746659 -4.079937 2 2
99
- 91 78.0220 -4.788940 -4.629738 2 1
100
- 92 78.2609 -4.903769 -4.380101 2 2
101
- 93 78.4946 -2.778426 -2.668484 2 2
102
- 94 78.7234 -5.065083 -4.071744 2 2
103
- 95 78.9474 -3.466854 -4.012455 1 1
104
- 96 79.1667 -1.945234 -3.052349 1 1
105
- 97 79.3814 -4.778632 -5.059439 1 1
106
- 98 79.5918 -4.029372 -3.704009 2 2
107
- 99 78.7879 -2.697536 -2.636393 2 1
108
- 100 79.0000 -5.542161 -4.590293 2 2
109
- 101 79.2079 -5.016406 -4.782845 2 2
110
- 102 78.4314 -3.906967 -3.891825 2 1
111
- 103 78.6408 -3.226637 -2.826427 2 2
112
- 104 78.8462 -7.447130 -7.127861 2 2
113
- 105 79.0476 -1.074855 -2.447202 1 1
114
- 106 79.2453 -2.879722 -2.740552 2 2
115
- 107 79.4393 -2.112792 -1.846187 2 2
116
- 108 79.6296 -4.157679 -3.321515 2 2
117
- 109 78.8991 -4.713466 -5.177304 1 2
118
- 110 79.0909 -4.685050 -3.924612 2 2
119
- 111 79.2793 -3.225951 -2.846309 2 2
120
- 112 78.5714 -4.150638 -3.147015 2 1
121
- 113 77.8761 -5.093537 -6.388408 1 2
122
- 114 78.0702 -3.938864 -5.032339 1 1
123
- 115 78.2609 -1.682003 -2.548842 1 1
124
- 116 78.4483 -3.239937 -3.035410 2 2
125
- 117 77.7778 -4.233876 -4.456881 1 2
126
- 118 77.9661 -1.439299 -2.389720 1 1
127
- 119 78.1513 -2.166229 -3.025556 1 1
128
- 120 77.5000 -2.747257 -2.554438 2 1
129
- 121 77.6860 -1.183946 -0.903156 2 2
130
- 122 77.8689 -6.275000 -5.763568 2 2
131
- 123 78.0488 -3.417439 -3.970876 1 1
132
- 124 78.2258 -2.889547 -2.600402 2 2
133
- 125 78.4000 -3.160100 -2.692845 2 2
134
- 126 77.7778 -6.657937 -6.708669 1 2
135
- 127 77.9528 -2.846436 -2.337550 2 2
136
- 128 78.1250 -1.727650 -2.115449 1 1
137
- 129 78.2946 -2.409674 -1.961675 2 2
138
- 130 78.4615 -6.425175 -5.045963 2 2
139
- 131 78.6260 -3.504076 -3.439669 2 2
140
- 132 78.7879 -2.563538 -2.024985 2 2
141
- 133 78.9474 -2.323369 -4.511004 1 1
142
- 134 79.1045 -5.067760 -4.541788 2 2
143
- 135 79.2593 -4.633971 -5.466318 1 1
144
- 136 79.4118 -3.516727 -2.655858 2 2
145
- 137 79.5620 -5.497421 -5.786833 1 1
146
- 138 78.9855 -3.410704 -4.462431 1 2
147
- 139 79.1367 -4.383582 -4.859771 1 1
148
- 140 79.2857 -1.550308 -1.000735 2 2
149
- 141 79.4326 -5.668096 -4.218701 2 2
150
- 142 79.5775 -3.503760 -2.772868 2 2
151
- 143 79.7203 -5.141438 -7.750103 1 1
152
- 144 79.1667 -2.381864 -2.344312 2 1
153
- 145 78.6207 -1.749851 -14.673623 1 2
154
- 146 78.0822 -3.859275 -4.118437 1 2
155
- 147 78.2313 -4.793463 -4.221262 2 2
156
- 148 78.3784 -1.906996 -2.496172 1 1
157
- 149 77.8523 -4.227023 -4.310668 1 2
158
- 150 78.0000 -4.258734 -3.312028 2 2
159
- 151 77.4834 -1.986105 -1.786683 2 1
160
- 152 76.9737 -4.256532 -4.464708 1 2
161
- 153 77.1242 -2.787270 -3.355256 1 1
162
- 154 77.2727 -3.797201 -3.970850 1 1
163
- 155 76.7742 -2.717653 -2.921398 1 2
164
- 156 76.9231 -2.039435 -2.353037 1 1
165
- 157 77.0701 -3.054537 -4.051275 1 1
166
- 158 77.2152 -4.684143 -4.170726 2 2
167
- 159 77.3585 -3.786540 -5.116251 1 1
168
- 160 77.5000 -3.790545 -3.419920 2 2
169
- 161 77.0186 -2.415569 -2.221454 2 1
170
- 162 76.5432 -0.090572 -13.912781 1 2
171
- 163 76.6871 -2.292582 -2.487130 1 1
172
- 164 76.8293 -4.741757 -3.535435 2 2
173
- 165 76.3636 -3.838899 -3.875466 1 2
174
- 166 76.5060 -5.052245 -4.244695 2 2
175
- 167 76.0479 -4.772940 -4.297321 2 1
176
- 168 76.1905 -3.980207 -3.663364 2 2
177
- 169 75.7396 -4.779693 -4.887860 1 2
178
- 170 75.8824 -2.443726 -2.820488 1 1
179
- 171 76.0234 -3.912050 -4.606647 1 1
180
- 172 76.1628 -4.200082 -5.143010 1 1
181
- 173 76.3006 -3.013068 -2.912589 2 2
182
- 174 76.4368 -5.716280 -5.432034 2 2
183
- 175 76.5714 -4.085990 -4.304922 1 1
184
- 176 76.7045 -2.412792 -4.491724 1 1
185
- 177 76.8362 -5.990150 -5.807810 2 2
186
- 178 76.4045 -2.684796 -2.413805 2 1
187
- 179 76.5363 -2.316249 -2.749653 1 1
188
- 180 76.6667 -5.796285 -5.599479 2 2
189
- 181 76.7956 -3.643715 -2.956035 2 2
190
- 182 76.9231 -4.698472 -3.642824 2 2
191
- 183 77.0492 -5.130662 -4.942244 2 2
192
- 184 77.1739 -1.575093 -2.553305 1 1
193
- 185 76.7568 -4.237680 -4.750720 1 2
194
- 186 76.8817 -3.196752 -3.817965 1 1
195
- 187 77.0053 -5.145463 -4.520022 2 2
196
- 188 76.5957 -5.057869 -3.767010 2 1
197
- 189 76.1905 -4.159149 -4.747592 1 2
198
- 190 76.3158 -4.654613 -4.429179 2 2
199
- 191 76.4398 -3.283936 -3.836780 1 1
200
- 192 76.5625 -5.338461 -4.128390 2 2
201
- 193 76.6839 -4.048517 -3.851170 2 2
202
- 194 76.8041 -2.001366 -1.649494 2 2
203
- 195 76.9231 -3.325938 -3.343131 1 1
204
- 196 76.5306 -3.237241 -3.708116 1 2
205
- 197 76.6497 -0.879010 -3.815177 1 1
206
- 198 76.7677 -4.259933 -6.825393 1 1
207
- 199 76.8844 -5.181133 -4.399740 2 2
208
- 200 77.0000 -3.203557 -2.906229 2 2
209
- 201 77.1144 -2.902179 -3.121619 1 1
210
- 202 77.2277 -8.521894 -8.047955 2 2
211
- 203 77.3399 -3.387094 -3.017890 2 2
212
- 204 77.4510 -2.199346 -2.878901 1 1
213
- 205 77.5610 -3.757236 -2.469443 2 2
214
- 206 77.1845 -4.231997 -4.294882 1 2
215
- 207 77.2947 -2.995968 -4.024071 1 1
216
- 208 77.4038 -3.195082 -2.559515 2 2
217
- 209 77.5120 -3.836545 -3.437815 2 2
218
- 210 77.6190 -6.548021 -6.417971 2 2
219
- 211 77.7251 -5.316227 -5.317550 1 1
220
- 212 77.8302 -5.028068 -4.285153 2 2
221
- 213 77.4648 -3.507702 -4.939651 1 2
222
- 214 77.1028 -3.285250 -3.275391 2 1
223
- 215 77.2093 -4.143869 -3.804569 2 2
224
- 216 76.8519 -3.295678 -3.243605 2 1
225
- 217 76.9585 -1.224553 -1.745479 1 1
226
- 218 77.0642 -2.949106 -2.615003 2 2
227
- 219 76.7123 -3.432811 -3.388935 2 1
228
- 220 76.3636 -3.994738 -4.411699 1 2
229
- 221 76.4706 -4.593042 -9.195989 1 1
230
- 222 76.1261 -0.894264 -9.452758 1 2
231
- 223 76.2332 -4.026409 -3.354530 2 2
232
- 224 76.3393 -5.321247 -4.415811 2 2
233
- 225 76.0000 -2.763164 -2.983335 1 2
234
- 226 76.1062 -3.208190 -3.774325 1 1
235
- 227 76.2115 -3.045439 -3.239566 1 1
236
- 228 76.3158 -1.935781 -2.988036 1 1
237
- 229 76.4192 -3.071861 -2.968124 2 2
238
- 230 76.5217 -3.473850 -3.753772 1 1
239
- 231 76.6234 -2.472583 -2.915879 1 1
240
- 232 76.7241 -2.019247 -1.850077 2 2
241
- 233 76.8240 -1.600011 -2.065943 1 1
242
- 234 76.9231 -3.964922 -3.530167 2 2
243
- 235 77.0213 -3.602591 -2.887588 2 2
244
- 236 76.6949 -1.891313 -1.497324 2 1
245
- 237 76.7932 -2.617981 -2.829435 1 1
246
- 238 76.8908 -2.863314 -2.332844 2 2
247
- 239 76.9874 -2.967953 -2.949558 2 2
248
- 240 77.0833 -6.779839 -6.712330 2 2
249
- 241 76.7635 -3.232212 -3.846330 1 2
250
- 242 76.8595 -5.863957 -5.507361 2 2
251
- 243 76.5432 -3.434559 -3.301018 2 1
252
- 244 76.6393 -2.101839 -4.051513 1 1
253
- 245 76.7347 -4.120459 -4.773267 1 1
254
- 246 76.8293 -2.704448 -3.274157 1 1
255
- 247 76.5182 -3.794226 -3.837690 1 2
256
- 248 76.6129 -3.366507 -3.194370 2 2
257
- 249 76.7068 -3.060161 -2.197101 2 2
258
- 250 76.8000 -2.370301 -2.866730 1 1
259
- 251 76.8924 -1.779728 -2.163604 1 1
260
- 252 76.9841 -7.199697 -6.504354 2 2
261
- 253 76.6798 -5.449084 -5.146064 2 1
262
- 254 76.3780 -4.448741 -4.492815 1 2
263
- 255 76.4706 -3.688205 -5.364802 1 1
264
- 256 76.1719 -3.242841 -2.789254 2 1
265
- 257 75.8755 -2.534852 -1.914775 2 1
266
- 258 75.5814 -3.865494 -4.020277 1 2
267
- 259 75.6757 -5.235586 -5.560316 1 1
268
- 260 75.3846 -5.829216 -5.536685 2 1
269
- 261 75.0958 -3.764001 -3.784941 1 2
270
- 262 75.1908 -4.573822 -4.353333 2 2
271
- 263 75.2852 -3.451048 -4.204721 1 1
272
- 264 75.3788 -3.992896 -3.267728 2 2
273
- 265 75.4717 -1.067863 -2.089497 1 1
274
- 266 75.5639 -5.189153 -6.661784 1 1
275
- 267 75.2809 -3.866616 -3.522151 2 1
276
- 268 75.0000 -2.861069 -2.761713 2 1
277
- 269 75.0929 -3.195728 -2.844297 2 2
278
- 270 75.1852 -2.671055 -3.176687 1 1
279
- 271 75.2768 -3.150494 -2.760012 2 2
280
- 272 75.3676 -3.914567 -3.519361 2 2
281
- 273 75.0916 -3.552462 -3.615776 1 2
282
- 274 74.8175 -3.339587 -3.470930 1 2
283
- 275 74.9091 -2.782999 -3.054036 1 1
284
- 276 74.6377 -2.703522 -2.549080 2 1
285
- 277 74.7292 -2.433211 -3.162786 1 1
286
- 278 74.8201 -3.207367 -3.251059 1 1
287
- 279 74.9104 -4.468188 -2.743534 2 2
288
- 280 75.0000 -3.848885 -3.442270 2 2
289
- 281 75.0890 -2.541542 -1.793045 2 2
290
- 282 75.1773 -4.275180 -3.537890 2 2
291
- 283 75.2650 -3.182873 -3.082112 2 2
292
- 284 75.3521 -2.651119 -3.490188 1 1
293
- 285 75.4386 -3.779054 -5.585062 1 1
294
- 286 75.5245 -3.592534 -3.794689 1 1
295
- 287 75.6098 -3.368379 -4.960323 1 1
296
- 288 75.6944 -3.798680 -2.951107 2 2
297
- 289 75.7785 -2.494219 -3.250443 1 1
298
- 290 75.8621 -4.725720 -4.928634 1 1
299
- 291 75.9450 -2.439718 -2.861429 1 1
300
- 292 76.0274 -2.844973 -2.283305 2 2
301
- 293 76.1092 -3.744239 -3.160640 2 2
302
- 294 76.1905 -3.271623 -3.531162 1 1
303
- 295 75.9322 -2.551880 -2.441417 2 1
304
- 296 76.0135 -5.110882 -4.894553 2 2
305
- 297 75.7576 -1.409816 -1.739626 1 2
306
- 298 75.8389 -5.338703 -5.005430 2 2
307
- 299 75.9197 -2.440721 -2.262475 2 2
308
- 300 76.0000 -1.999090 -2.370612 1 1
309
- 301 76.0797 -3.964341 -4.490774 1 1
310
- 302 76.1589 -4.130625 -4.018433 2 2
311
- 303 76.2376 -8.186714 -8.340643 1 1
312
- 304 76.3158 -4.735212 -3.490800 2 2
313
- 305 76.3934 -3.741217 -3.504135 2 2
314
- 306 76.4706 -2.186754 -1.814700 2 2
315
- 307 76.5472 -2.656456 -3.164755 1 1
316
- 308 76.6234 -5.473002 -4.381585 2 2
317
- 309 76.6990 -4.477587 -5.089441 1 1
318
- 310 76.4516 -5.690542 -5.770220 1 2
319
- 311 76.5273 -0.661393 -4.241687 1 1
320
- 312 76.6026 -4.198477 -4.268304 1 1
321
- 313 76.6773 -5.040315 -4.964295 2 2
322
- 314 76.7516 -5.230682 -4.677516 2 2
323
- 315 76.8254 -2.002319 -3.655196 1 1
324
- 316 76.8987 -7.410125 -4.765240 2 2
325
- 317 76.6562 -4.675953 -4.632349 2 1
326
- 318 76.7296 -5.121322 -4.602418 2 2
327
- 319 76.8025 -1.649690 -1.861507 1 1
328
- 320 76.8750 -3.679719 -4.615596 1 1
329
- 321 76.6355 -2.955515 -3.035355 1 2
330
- 322 76.3975 -2.498726 -2.947951 1 2
331
- 323 76.4706 -3.433668 -3.875072 1 1
332
- 324 76.2346 -4.672136 -4.368118 2 1
333
- 325 76.0000 -5.400143 -5.403687 1 2
334
- 326 76.0736 -2.447175 -1.926339 2 2
335
- 327 76.1468 -2.423428 -2.927730 1 1
336
- 328 76.2195 -4.191886 -4.491265 1 1
337
- 329 76.2918 -3.113300 -7.485933 1 1
338
- 330 76.3636 -1.375623 -2.011121 1 1
339
- 331 76.1329 -3.992572 -3.172000 2 1
340
- 332 76.2048 -1.646617 -2.658879 1 1
341
- 333 76.2763 -3.983303 -1.289821 2 2
342
- 334 76.3473 -2.813006 -5.562449 1 1
343
- 335 76.1194 -5.684294 -5.337053 2 1
344
- 336 75.8929 -3.446034 -4.032972 1 2
345
- 337 75.9644 -1.956340 -1.569860 2 2
346
- 338 75.7396 -3.548353 -3.041705 2 1
347
- 339 75.8112 -6.032905 -5.991912 2 2
348
- 340 75.8824 -4.900002 -5.656678 1 1
349
- 341 75.9531 -4.062330 -4.879829 1 1
350
- 342 76.0234 -5.525647 -4.941439 2 2
351
- 343 76.0933 -3.460521 -4.155955 1 1
352
- 344 76.1628 -2.258570 -3.314703 1 1
353
- 345 76.2319 -0.783208 -3.189191 1 1
354
- 346 76.3006 -5.473655 -5.336968 2 2
355
- 347 76.3689 -3.271599 -2.802353 2 2
356
- 348 76.4368 -1.061048 -2.040404 1 1
357
- 349 76.2178 -3.310234 -4.411252 1 2
358
- 350 76.0000 -2.579735 -4.750209 1 2
359
- 351 76.0684 -3.223269 -4.030144 1 1
360
- 352 76.1364 -3.897161 -3.018411 2 2
361
- 353 75.9207 -2.836279 -2.708077 2 1
362
- 354 75.9887 -5.382059 -4.642745 2 2
363
- 355 76.0563 -2.876620 -2.318043 2 2
364
- 356 75.8427 -3.574898 -3.781713 1 2
365
- 357 75.9104 -4.347421 -3.996215 2 2
366
- 358 75.9777 -2.048956 -1.863463 2 2
367
- 359 76.0446 -5.589315 -3.639884 2 2
368
- 360 75.8333 -3.722123 -3.840726 1 2
369
- 361 75.9003 -2.975312 -1.889599 2 2
370
- 362 75.9669 -6.287781 -4.955991 2 2
371
- 363 75.7576 -2.825849 -2.609949 2 1
372
- 364 75.8242 -6.234842 -6.988809 1 1
373
- 365 75.8904 -5.515721 -5.584534 1 1
374
- 366 75.6831 -2.276235 -7.604947 1 2
375
- 367 75.7493 -5.079682 -5.353662 1 1
376
- 368 75.8152 -3.674519 -2.788866 2 2
377
- 369 75.6098 -3.449991 -3.790493 1 2
378
- 370 75.4054 -6.487534 -5.003440 2 1
379
- 371 75.4717 -2.431289 -3.382524 1 1
380
- 372 75.5376 -2.560569 -2.739982 1 1
381
- 373 75.3351 -4.654605 -3.713296 2 1
382
- 374 75.4011 -2.549920 -3.191272 1 1
383
- 375 75.4667 -3.476110 -2.513606 2 2
384
- 376 75.5319 -4.349367 -3.760199 2 2
385
- 377 75.5968 -4.103452 -3.884920 2 2
386
- 378 75.3968 -3.067181 -3.429934 1 2
387
- 379 75.1979 -0.842239 -1.922143 1 2
388
- 380 75.2632 -6.232121 -6.475739 1 1
389
- 381 75.3281 -10.535327 -8.939424 2 2
390
- 382 75.3927 -2.771046 -2.623841 2 2
391
- 383 75.4569 -2.171459 -1.210128 2 2
392
- 384 75.2604 -3.341716 -2.521049 2 1
393
- 385 75.3247 -0.643506 -0.723347 1 1
394
- 386 75.3886 -3.008347 -14.149488 1 1
395
- 387 75.4522 -1.773700 -2.503436 1 1
396
- 388 75.5155 -5.348296 -5.249991 2 2
397
- 389 75.5784 -3.096191 -3.086448 2 2
398
- 390 75.6410 -3.132273 -3.150845 1 1
399
- 391 75.4476 -3.936080 -4.229895 1 2
400
- 392 75.2551 -4.501307 -3.412280 2 1
401
- 393 75.3181 -5.599928 -5.108171 2 2
402
- 394 75.3807 -2.294498 -2.903459 1 1
403
- 395 75.1899 -3.260395 -3.276457 1 2
404
- 396 75.2525 -3.501292 -4.197696 1 1
405
- 397 75.3149 -6.788257 -7.143467 1 1
406
- 398 75.3769 -2.902089 -5.375241 1 1
407
- 399 75.4386 -3.500133 -4.324882 1 1
408
- 400 75.5000 -2.129413 -6.846359 1 1
409
- 401 75.5611 -3.558550 -3.717386 1 1
410
- 402 75.3731 -3.464216 -3.569263 1 2
411
- 403 75.1861 -5.751697 -6.308459 1 2
412
- 404 75.2475 -4.521032 -3.423212 2 2
413
- 405 75.3086 -3.097537 -4.486981 1 1
414
- 406 75.3695 -8.205668 -8.870927 1 1
415
- 407 75.1843 -5.137034 -5.167531 1 2
416
- 408 75.0000 -3.378816 -3.634895 1 2
417
- 409 75.0611 -4.025488 -4.690878 1 1
418
- 410 74.8780 -3.926253 -3.924037 2 1
419
- 411 74.9392 -3.864911 -4.893489 1 1
420
- 412 75.0000 -3.959647 -3.396416 2 2
421
- 413 75.0605 -3.333512 -2.769698 2 2
422
- 414 74.8792 -3.968904 -4.484334 1 2
423
- 415 74.9398 -4.121704 -3.341224 2 2
424
- 416 75.0000 -4.366259 -2.864569 2 2
425
- 417 75.0600 -4.234683 -5.093482 1 1
426
- 418 75.1196 -5.624996 -6.079271 1 1
427
- 419 75.1790 -4.037092 -3.509315 2 2
428
- 420 75.0000 -3.126703 -3.140785 1 2
429
- 421 75.0594 -4.295188 -3.889868 2 2
430
- 422 75.1185 -1.951848 -2.213968 1 1
431
- 423 74.9409 -3.299278 -3.239721 2 1
432
- 424 75.0000 -4.610185 -5.727749 1 1
433
- 425 75.0588 -5.208189 -3.380292 2 2
434
- 426 75.1174 -2.234741 -1.923795 2 2
435
- 427 75.1756 -3.898456 -4.190275 1 1
436
- 428 75.2336 -3.849512 -4.278590 1 1
437
- 429 75.2914 -3.292652 -3.173839 2 2
438
- 430 75.3488 -5.191588 -5.467621 1 1
439
- 431 75.4060 -3.856440 -5.124656 1 1
440
- 432 75.2315 -3.065131 -3.787311 1 2
441
- 433 75.2887 -2.270048 -1.915121 2 2
442
- 434 75.3456 -6.695189 -7.708446 1 1
443
- 435 75.1724 -3.858520 -3.667486 2 1
444
- 436 75.0000 -2.585082 -2.015982 2 1
445
- 437 75.0572 -4.170781 -4.085763 2 2
446
- 438 75.1142 -4.188624 -3.842886 2 2
447
- 439 75.1708 -3.117637 -3.234314 1 1
448
- 440 75.2273 -3.359273 -3.447190 1 1
449
- 441 75.2834 -3.812526 -4.704022 1 1
450
- 442 75.3394 -4.375708 -4.580903 1 1
451
- 443 75.3950 -1.852353 -3.132455 1 1
452
- 444 75.2252 -5.770766 -6.277817 1 2
453
- 445 75.2809 -1.729618 -2.437814 1 1
454
- 446 75.3363 -4.371078 -3.247448 2 2
455
- 447 75.1678 -3.008746 -2.834310 2 1
456
- 448 75.2232 -5.145736 -4.666968 2 2
457
- 449 75.2784 -2.513560 -2.137208 2 2
458
- 450 75.3333 -5.413717 -4.363818 2 2
459
- 451 75.1663 -3.920115 -3.937495 1 2
460
- 452 75.0000 -4.556690 -3.977286 2 1
461
- 453 75.0552 -4.003839 -3.500368 2 2
462
- 454 74.8899 -3.887330 -3.880656 2 1
463
- 455 74.7253 -2.286048 -10.494519 1 2
464
- 456 74.5614 -3.713793 -3.867856 1 2
465
- 457 74.3982 -9.347245 -8.626439 2 1
466
- 458 74.4541 -3.697419 -2.830104 2 2
467
- 459 74.5098 -3.916895 -3.826642 2 2
468
- 460 74.5652 -1.439062 -9.364925 1 1
469
- 461 74.6204 -2.474742 -3.962622 1 1
470
- 462 74.4589 -5.914469 -5.688967 2 1
471
- 463 74.5140 -1.091609 -1.344088 1 1
472
- 464 74.5690 -2.085408 -2.508941 1 1
473
- 465 74.6237 -3.780431 -2.692734 2 2
474
- 466 74.6781 -5.396760 -5.359945 2 2
475
- 467 74.5182 -5.320886 -5.013037 2 1
476
- 468 74.5726 -4.373384 -3.097897 2 2
477
- 469 74.6269 -1.189494 -1.609284 1 1
478
- 470 74.6809 -1.955861 -1.428945 2 2
479
- 471 74.7346 -2.556780 -2.926999 1 1
480
- 472 74.5763 -3.383330 -3.437979 1 2
481
- 473 74.6300 -4.223370 -3.692806 2 2
482
- 474 74.6835 -4.749713 -5.821321 1 1
483
- 475 74.7368 -2.552552 -2.227653 2 2
484
- 476 74.5798 -4.295245 -4.806842 1 2
485
- 477 74.4235 -4.078160 -4.632263 1 2
486
- 478 74.4770 -2.509742 -2.686108 1 1
487
- 479 74.3215 -3.206753 -3.077258 2 1
488
- 480 74.3750 -5.583317 -5.275991 2 2
489
- 481 74.4283 -2.165321 -2.484724 1 1
490
- 482 74.2739 -4.418467 -4.990839 1 2
491
- 483 74.3271 -4.218618 -4.708273 1 1
492
- 484 74.1736 -3.102189 -2.281201 2 1
493
- 485 74.2268 -1.398563 -1.979566 1 1
494
- 486 74.2798 -2.688539 -3.133311 1 1
495
- 487 74.3326 -3.852966 -3.384095 2 2
496
- 488 74.3852 -3.956349 -3.041618 2 2
497
- 489 74.4376 -2.387548 -3.196393 1 1
498
- 490 74.4898 -2.714462 -3.029183 1 1
499
- 491 74.5418 -5.241804 -4.197929 2 2
500
- 492 74.5935 -5.217912 -4.185747 2 2
501
- 493 74.6450 -1.269408 -2.235014 1 1
502
- 494 74.6964 -3.368152 -2.774116 2 2
503
- 495 74.5455 -3.964761 -3.938147 2 1
504
- 496 74.5968 -3.173600 -2.831292 2 2
505
- 497 74.6479 -4.692360 -4.993606 1 1
506
- 498 74.4980 -1.774088 -2.211962 1 2
507
- 499 74.5491 -3.085772 -3.588308 1 1
508
- 500 74.6000 -6.052997 -5.219882 2 2
509
- 501 74.6507 -2.196733 -2.791374 1 1
510
- 502 74.7012 -0.984931 -2.257588 1 1
511
- 503 74.7515 -3.455171 -3.152770 2 2
512
- 504 74.8016 -2.860287 -2.881672 1 1
513
- 505 74.8515 -7.238441 -7.146571 2 2
514
- 506 74.9012 -3.971247 -5.101849 1 1
515
- 507 74.9507 -3.513725 -3.731356 1 1
516
- 508 75.0000 -5.440109 -3.678008 2 2
517
- 509 75.0491 -6.030756 -10.856277 1 1
518
- 510 75.0980 -3.118112 -3.444312 1 1
519
- 511 75.1468 -5.194215 -3.843143 2 2
520
- 512 75.1953 -3.253176 -3.691781 1 1
521
- 513 75.0487 -2.628629 -2.925695 1 2
522
- 514 74.9027 -5.089205 -4.156975 2 1
523
- 515 74.9515 -5.628146 -7.627123 1 1
524
- 516 75.0000 -4.457075 -4.469458 1 1
525
- 517 75.0484 -1.670615 -1.884140 1 1
526
- 518 74.9035 -2.396024 -2.414083 1 2
527
- 519 74.7592 -6.966568 -6.968682 1 2
528
- 520 74.6154 -2.581139 -2.590892 1 2
529
- 521 74.6641 -1.856858 -2.091242 1 1
530
- 522 74.7126 -3.758548 -2.597640 2 2
531
- 523 74.7610 -3.534909 -3.265962 2 2
532
- 524 74.6183 -4.671712 -4.666166 2 1
533
- 525 74.6667 -4.680920 -2.642850 2 2
534
- 526 74.7148 -1.699402 -12.973889 1 1
535
- 527 74.7628 -6.244634 -3.798081 2 2
536
- 528 74.6212 -5.618906 -6.094369 1 2
537
- 529 74.6692 -5.372440 -6.198888 1 1
538
- 530 74.7170 -4.576940 -3.071590 2 2
539
- 531 74.7646 -3.498921 -3.903360 1 1
540
- 532 74.8120 -2.008700 -3.533193 1 1
541
- 533 74.8593 -3.413457 -2.707085 2 2
542
- 534 74.9064 -2.482046 -3.125805 1 1
543
- 535 74.9533 -3.409975 -3.389211 2 2
544
- 536 74.8134 -2.343477 -1.991227 2 1
545
- 537 74.8603 -5.134516 -5.983048 1 1
546
- 538 74.9071 -2.249300 -5.308599 1 1
547
- 539 74.9536 -3.825196 -2.104226 2 2
548
- 540 75.0000 -2.747113 -2.667776 2 2
549
- 541 75.0462 -2.708105 -3.301016 1 1
550
- 542 74.9077 -3.991769 -4.620331 1 2
551
- 543 74.9540 -5.853641 -4.285472 2 2
552
- 544 75.0000 -3.643494 -4.185267 1 1
553
- 545 74.8624 -1.946658 -3.958046 1 2
554
- 546 74.9084 -3.123706 -4.533720 1 1
555
- 547 74.9543 -5.454079 -4.690523 2 2
556
- 548 74.8175 -4.606745 -4.207100 2 1
557
- 549 74.8634 -5.178109 -5.521366 1 1
558
- 550 74.9091 -2.884686 -2.852227 2 2
559
- 551 74.7731 -1.230807 -14.145962 1 2
560
- 552 74.6377 -1.918641 -1.743652 2 1
561
- 553 74.6835 -5.309980 -5.059883 2 2
562
- 554 74.7292 -4.626562 -4.607808 2 2
563
- 555 74.7748 -1.655379 -1.392300 2 2
564
- 556 74.8201 -5.207582 -4.875259 2 2
565
- 557 74.8654 -5.105472 -5.111954 1 1
566
- 558 74.9104 -1.648886 -3.208308 1 1
567
- 559 74.9553 -2.369207 -2.295145 2 2
568
- 560 75.0000 -2.147079 -2.352000 1 1
569
- 561 75.0446 -2.531145 -2.848607 1 1
570
- 562 75.0890 -0.918763 -1.762093 1 1
571
- 563 74.9556 -1.355228 -2.710117 1 2
572
- 564 75.0000 -2.887733 -3.115344 1 1
573
- 565 75.0442 -4.334890 -3.953423 2 2
574
- 566 75.0883 -3.987635 -4.749141 1 1
575
- 567 75.1323 -1.847698 -1.679241 2 2
576
- 568 75.1761 -7.525989 -2.889049 2 2
577
- 569 75.0439 -5.537669 -4.476378 2 1
578
- 570 75.0877 -4.826198 -5.325478 1 1
579
- 571 75.1313 -4.128893 -3.441449 2 2
580
- 572 75.1748 -2.327386 -2.167058 2 2
581
- 573 75.2182 -4.162209 -4.150951 2 2
582
- 574 75.0871 -3.775156 -3.807681 1 2
583
- 575 75.1304 -5.188933 -5.127740 2 2
584
- 576 75.1736 -3.756513 -4.120339 1 1
585
- 577 75.2166 -4.104163 -3.506364 2 2
586
- 578 75.2595 -2.969283 -2.921067 2 2
587
- 579 75.1295 -4.515483 -4.830085 1 2
588
- 580 75.1724 -2.140030 -2.551824 1 1
589
- 581 75.0430 -5.838675 -5.810240 2 1
590
- 582 74.9141 -6.015281 -6.176453 1 2
591
- 583 74.9571 -4.919650 -4.552823 2 2
592
- 584 74.8288 -4.720296 -4.569730 2 1
593
- 585 74.8718 -0.557624 -1.676053 1 1
594
- 586 74.9147 -3.641736 -3.744077 1 1
595
- 587 74.7871 -5.127168 -5.048219 2 1
596
- 588 74.8299 -3.556053 -3.165168 2 2
597
- 589 74.8727 -3.964789 -3.889975 2 2
598
- 590 74.9153 -3.873292 -3.165917 2 2
599
- 591 74.9577 -3.764847 -3.630316 2 2
600
- 592 75.0000 -2.911044 -3.535620 1 1
601
- 593 75.0422 -3.785763 -3.396825 2 2
602
- 594 74.9158 -4.855898 -4.364533 2 1
603
- 595 74.9580 -1.641873 -1.584285 2 2
604
- 596 74.8322 -4.800048 -4.855697 1 2
605
- 597 74.8744 -1.991868 -2.090064 1 1
606
- 598 74.9164 -2.941852 -3.922611 1 1
607
- 599 74.9583 -2.970597 -3.189365 1 1
608
- 600 75.0000 -4.134793 -3.994568 2 2
609
- 601 74.8752 -3.107208 -3.137663 1 2
610
- 602 74.9169 -2.160943 -4.441257 1 1
611
- 603 74.9585 -5.961274 -5.606278 2 2
612
- 604 74.8344 -2.995724 -3.049393 1 2
613
- 605 74.7107 -2.460331 -2.131717 2 1
614
- 606 74.7525 -3.080399 -2.632306 2 2
615
- 607 74.6293 -2.465931 -2.064029 2 1
616
- 608 74.6711 -2.654870 -2.908435 1 1
617
- 609 74.7126 -5.639697 -6.179128 1 1
618
- 610 74.5902 -2.631252 -2.541841 2 1
619
- 611 74.6318 -5.204927 -5.282381 1 1
620
- 612 74.6732 -4.006695 -5.309594 1 1
621
- 613 74.7145 -2.519858 -2.257215 2 2
622
- 614 74.7557 -2.775467 -2.003975 2 2
623
- 615 74.7967 -2.416856 -2.233912 2 2
624
- 616 74.8377 -3.524757 -3.477837 2 2
625
- 617 74.8784 -11.676094 -3.705719 2 2
626
- 618 74.7573 -3.678833 -3.877129 1 2
627
- 619 74.7981 -3.446922 -2.869983 2 2
628
- 620 74.6774 -5.281838 -4.726645 2 1
629
- 621 74.7182 -3.508409 -3.296519 2 2
630
- 622 74.5981 -2.711367 -2.676354 2 1
631
- 623 74.4783 -2.673899 -2.559443 2 1
632
- 624 74.5192 -4.542767 -3.710020 2 2
633
- 625 74.5600 -3.813804 -6.014807 1 1
634
- 626 74.6006 -2.466375 -3.468952 1 1
635
- 627 74.4817 -3.703218 -3.592656 2 1
636
- 628 74.5223 -2.884774 -2.819365 2 2
637
- 629 74.5628 -1.528837 -2.114351 1 1
638
- 630 74.6032 -1.242994 -1.916726 1 1
639
- 631 74.6434 -1.680958 -2.803157 1 1
640
- 632 74.6835 -2.055074 -1.774820 2 2
641
- 633 74.7235 -2.732798 -2.047294 2 2
642
- 634 74.7634 -2.564450 -3.393250 1 1
643
- 635 74.6457 -3.968273 -4.624722 1 2
644
- 636 74.6855 -1.773139 -3.557484 1 1
645
- 637 74.7253 -3.502622 -5.640187 1 1
646
- 638 74.7649 -5.134840 -5.617863 1 1
647
- 639 74.6479 -1.805352 -1.577120 2 1
648
- 640 74.5312 -1.154884 -2.088550 1 2
649
- 641 74.5710 -4.183061 -5.051393 1 1
650
- 642 74.6106 -4.426652 -2.540145 2 2
651
- 643 74.6501 -1.210693 -0.751933 2 2
652
- 644 74.6894 -3.908940 -4.840205 1 1
653
- 645 74.7287 -1.486852 -2.567492 1 1
654
- 646 74.7678 -3.925992 -3.976377 1 1
655
- 647 74.6522 -4.568136 -3.847041 2 1
656
- 648 74.6914 -3.727147 -3.178708 2 2
657
- 649 74.7304 -2.824107 -3.073107 1 1
658
- 650 74.7692 -3.565836 -4.391446 1 1
659
- 651 74.8080 -6.150738 -5.614686 2 2
660
- 652 74.8466 -5.992276 -5.052381 2 2
661
- 653 74.8851 -6.379559 -5.447601 2 2
662
- 654 74.7706 -3.415870 -3.601653 1 2
663
- 655 74.6565 -2.650920 -2.749276 1 2
664
- 656 74.6951 -4.007736 -1.879215 2 2
665
- 657 74.5814 -5.270806 -6.087546 1 2
666
- 658 74.6201 -5.485003 -3.275775 2 2
667
- 659 74.6586 -2.218271 -1.960469 2 2
668
- 660 74.6970 -3.527734 -2.808167 2 2
669
- 661 74.7352 -3.027728 -3.839217 1 1
670
- 662 74.7734 -3.505566 -3.885340 1 1
671
- 663 74.8115 -5.279043 -4.413099 2 2
672
- 664 74.8494 -2.981101 -3.045409 1 1
673
- 665 74.7368 -3.749001 -3.933349 1 2
674
- 666 74.7748 -4.461184 -4.805644 1 1
675
- 667 74.6627 -3.974249 -9.322887 1 2
676
- 668 74.7006 -2.760426 -3.850307 1 1
677
- 669 74.7384 -3.127745 -3.480544 1 1
678
- 670 74.6269 -3.606989 -2.951543 2 1
679
- 671 74.6647 -3.490654 -4.579444 1 1
680
- 672 74.5536 -3.010016 -3.017925 1 2
681
- 673 74.5914 -1.995282 -1.863809 2 2
682
- 674 74.4807 -6.496851 -6.042524 2 1
683
- 675 74.3704 -4.059921 -3.547071 2 1
684
- 676 74.2604 -2.056766 -2.712604 1 2
685
- 677 74.2984 -3.925927 -4.111434 1 1
686
- 678 74.1888 -5.075198 -5.243089 1 2
687
- 679 74.2268 -5.483885 -5.092773 2 2
688
- 680 74.2647 -3.278163 -4.638767 1 1
689
- 681 74.1557 -3.722889 -2.728870 2 1
690
- 682 74.0469 -2.570930 -2.608518 1 2
691
- 683 74.0849 -1.890826 -2.913587 1 1
692
- 684 74.1228 -4.129029 -7.643153 1 1
693
- 685 74.1606 -3.761161 -1.574313 2 2
694
- 686 74.1983 -4.825546 -4.762340 2 2
695
- 687 74.2358 -2.795627 -1.716356 2 2
696
- 688 74.2733 -7.165375 -7.306653 1 1
697
- 689 74.3106 -3.009892 -2.326871 2 2
698
- 690 74.2029 -3.077703 -3.281266 1 2
699
- 691 74.0955 -3.399326 -3.270385 2 1
700
- 692 74.1329 -1.411978 -2.682453 1 1
701
- 693 74.1703 -4.799410 -5.072008 1 1
702
- 694 74.2075 -3.558893 -3.269964 2 2
703
- 695 74.2446 -5.365056 -5.555428 1 1
704
- 696 74.1379 -4.188228 -4.106233 2 1
705
- 697 74.0316 -2.823997 -2.773184 2 1
706
- 698 74.0688 -5.139372 -4.586755 2 2
707
- 699 74.1059 -3.826447 -5.523142 1 1
708
- 700 74.1429 -3.372852 -2.712671 2 2
709
- 701 74.0371 -2.350682 -1.980577 2 1
710
- 702 73.9316 -4.509574 -4.797882 1 2
711
- 703 73.9687 -3.518957 -3.649048 1 1
712
- 704 74.0057 -1.547209 -1.920434 1 1
713
- 705 74.0426 -2.453001 -2.143006 2 2
714
- 706 74.0793 -2.974723 -2.918966 2 2
715
- 707 73.9745 -4.656524 -4.440279 2 1
716
- 708 74.0113 -3.933974 -3.484860 2 2
717
- 709 74.0480 -3.637997 -3.000235 2 2
718
- 710 74.0845 -2.678396 -3.976168 1 1
719
- 711 74.1210 -3.141127 -2.993035 2 2
720
- 712 74.0169 -3.383513 -3.975215 1 2
721
- 713 73.9130 -3.400831 -3.498604 1 2
722
- 714 73.9496 -3.175123 -3.051949 2 2
723
- 715 73.9860 -5.350029 -3.946753 2 2
724
- 716 74.0223 -4.430111 -3.682391 2 2
725
- 717 74.0586 -2.837976 -0.629023 2 2
726
- 718 74.0947 -2.353843 -2.912901 1 1
727
- 719 73.9917 -5.227757 -4.744323 2 1
728
- 720 73.8889 -1.132259 -4.046937 1 2
729
- 721 73.7864 -5.215458 -5.964162 1 2
730
- 722 73.6842 -2.503834 -2.447289 2 1
731
- 723 73.7206 -2.564506 -3.191728 1 1
732
- 724 73.6188 -3.394349 -3.773858 1 2
733
- 725 73.6552 -2.715708 -2.635864 2 2
734
- 726 73.5537 -4.182534 -4.386123 1 2
735
- 727 73.5901 -3.355596 -2.512698 2 2
736
- 728 73.4890 -4.606160 -4.573176 2 1
737
- 729 73.5254 -2.246548 -2.582956 1 1
738
- 730 73.4247 -3.597568 -3.305080 2 1
739
- 731 73.4610 -3.596335 -4.077417 1 1
740
- 732 73.4973 -3.313269 -2.639974 2 2
741
- 733 73.5334 -4.857446 -3.997855 2 2
742
- 734 73.5695 -0.108777 -8.477270 1 1
743
- 735 73.6054 -2.873634 -2.933996 1 1
744
- 736 73.6413 -3.209076 -4.421182 1 1
745
- 737 73.5414 -4.236897 -4.022048 2 1
746
- 738 73.5772 -4.456596 -4.301561 2 2
747
- 739 73.6130 -1.082343 -9.446684 1 1
748
- 740 73.6486 -4.062174 -3.257500 2 2
749
- 741 73.5493 -4.122938 -4.091984 2 1
750
- 742 73.5849 -2.161349 -1.759975 2 2
751
- 743 73.6205 -3.549686 -3.383518 2 2
752
- 744 73.6559 -4.703460 -3.791936 2 2
753
- 745 73.6913 -4.275476 -5.174931 1 1
754
- 746 73.5925 -5.435682 -5.362732 2 1
755
- 747 73.4940 -5.837842 -4.517450 2 1
756
- 748 73.5294 -3.287476 -3.248356 2 2
757
- 749 73.5648 -3.187271 -3.555015 1 1
758
- 750 73.6000 -1.182483 -9.244070 1 1
759
 
760
- Final Winogrande score(750 tasks): 73.6000 +/- 1.6106
 
 
 
 
 
1
+ build: 4945 (e354bc3b) with Apple clang version 16.0.0 (clang-1600.0.26.6) for arm64-apple-darwin24.3.0
2
+ llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
3
+ llama_model_loader: loaded meta data with 40 key-value pairs and 292 tensors from ./Watt-Tool-8B-Q3_K_S.gguf (version GGUF V3 (latest))
4
 
5
+ Final Winogrande score(750 tasks): 71.8667 +/- 1.6430
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
 
7
+ llama_perf_context_print: load time = 314.44 ms
8
+ llama_perf_context_print: prompt eval time = 96371.78 ms / 22317 tokens ( 4.32 ms per token, 231.57 tokens per second)
9
+ llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
10
+ llama_perf_context_print: total time = 97549.79 ms / 22318 tokens
11
+ ggml_metal_free: deallocating
scores/Watt-Tool-8B-q4_k_m-naive.arc DELETED
@@ -1,763 +0,0 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
3
-
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 869 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 869 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
-
10
- task acc_norm
11
- 1 100.00000000
12
- 2 100.00000000
13
- 3 100.00000000
14
- 4 75.00000000
15
- 5 80.00000000
16
- 6 83.33333333
17
- 7 85.71428571
18
- 8 87.50000000
19
- 9 77.77777778
20
- 10 80.00000000
21
- 11 72.72727273
22
- 12 75.00000000
23
- 13 76.92307692
24
- 14 78.57142857
25
- 15 80.00000000
26
- 16 81.25000000
27
- 17 82.35294118
28
- 18 77.77777778
29
- 19 73.68421053
30
- 20 70.00000000
31
- 21 71.42857143
32
- 22 72.72727273
33
- 23 69.56521739
34
- 24 70.83333333
35
- 25 68.00000000
36
- 26 65.38461538
37
- 27 66.66666667
38
- 28 64.28571429
39
- 29 65.51724138
40
- 30 66.66666667
41
- 31 64.51612903
42
- 32 65.62500000
43
- 33 63.63636364
44
- 34 61.76470588
45
- 35 62.85714286
46
- 36 61.11111111
47
- 37 62.16216216
48
- 38 60.52631579
49
- 39 61.53846154
50
- 40 62.50000000
51
- 41 63.41463415
52
- 42 61.90476190
53
- 43 60.46511628
54
- 44 59.09090909
55
- 45 60.00000000
56
- 46 60.86956522
57
- 47 61.70212766
58
- 48 62.50000000
59
- 49 63.26530612
60
- 50 62.00000000
61
- 51 60.78431373
62
- 52 61.53846154
63
- 53 60.37735849
64
- 54 59.25925926
65
- 55 58.18181818
66
- 56 58.92857143
67
- 57 59.64912281
68
- 58 60.34482759
69
- 59 61.01694915
70
- 60 61.66666667
71
- 61 62.29508197
72
- 62 61.29032258
73
- 63 61.90476190
74
- 64 62.50000000
75
- 65 61.53846154
76
- 66 62.12121212
77
- 67 62.68656716
78
- 68 63.23529412
79
- 69 63.76811594
80
- 70 62.85714286
81
- 71 61.97183099
82
- 72 61.11111111
83
- 73 61.64383562
84
- 74 62.16216216
85
- 75 62.66666667
86
- 76 61.84210526
87
- 77 62.33766234
88
- 78 62.82051282
89
- 79 62.02531646
90
- 80 62.50000000
91
- 81 61.72839506
92
- 82 62.19512195
93
- 83 61.44578313
94
- 84 61.90476190
95
- 85 62.35294118
96
- 86 62.79069767
97
- 87 63.21839080
98
- 88 63.63636364
99
- 89 64.04494382
100
- 90 64.44444444
101
- 91 64.83516484
102
- 92 65.21739130
103
- 93 64.51612903
104
- 94 63.82978723
105
- 95 64.21052632
106
- 96 64.58333333
107
- 97 63.91752577
108
- 98 64.28571429
109
- 99 64.64646465
110
- 100 65.00000000
111
- 101 64.35643564
112
- 102 64.70588235
113
- 103 64.07766990
114
- 104 63.46153846
115
- 105 63.80952381
116
- 106 63.20754717
117
- 107 63.55140187
118
- 108 62.96296296
119
- 109 62.38532110
120
- 110 62.72727273
121
- 111 63.06306306
122
- 112 63.39285714
123
- 113 63.71681416
124
- 114 64.03508772
125
- 115 64.34782609
126
- 116 64.65517241
127
- 117 64.10256410
128
- 118 64.40677966
129
- 119 63.86554622
130
- 120 64.16666667
131
- 121 63.63636364
132
- 122 63.93442623
133
- 123 64.22764228
134
- 124 63.70967742
135
- 125 64.00000000
136
- 126 64.28571429
137
- 127 64.56692913
138
- 128 64.84375000
139
- 129 64.34108527
140
- 130 64.61538462
141
- 131 64.88549618
142
- 132 65.15151515
143
- 133 65.41353383
144
- 134 65.67164179
145
- 135 65.18518519
146
- 136 64.70588235
147
- 137 64.96350365
148
- 138 65.21739130
149
- 139 65.46762590
150
- 140 65.71428571
151
- 141 65.24822695
152
- 142 65.49295775
153
- 143 65.73426573
154
- 144 65.97222222
155
- 145 66.20689655
156
- 146 65.75342466
157
- 147 65.98639456
158
- 148 66.21621622
159
- 149 66.44295302
160
- 150 66.66666667
161
- 151 66.22516556
162
- 152 66.44736842
163
- 153 66.66666667
164
- 154 66.23376623
165
- 155 66.45161290
166
- 156 66.66666667
167
- 157 66.87898089
168
- 158 66.45569620
169
- 159 66.66666667
170
- 160 66.25000000
171
- 161 66.45962733
172
- 162 66.66666667
173
- 163 66.25766871
174
- 164 66.46341463
175
- 165 66.06060606
176
- 166 66.26506024
177
- 167 65.86826347
178
- 168 66.07142857
179
- 169 65.68047337
180
- 170 65.29411765
181
- 171 65.49707602
182
- 172 65.69767442
183
- 173 65.89595376
184
- 174 66.09195402
185
- 175 65.71428571
186
- 176 65.90909091
187
- 177 66.10169492
188
- 178 66.29213483
189
- 179 66.48044693
190
- 180 66.66666667
191
- 181 66.29834254
192
- 182 66.48351648
193
- 183 66.12021858
194
- 184 66.30434783
195
- 185 66.48648649
196
- 186 66.66666667
197
- 187 66.31016043
198
- 188 65.95744681
199
- 189 65.60846561
200
- 190 65.26315789
201
- 191 65.44502618
202
- 192 65.62500000
203
- 193 65.28497409
204
- 194 65.46391753
205
- 195 65.64102564
206
- 196 65.81632653
207
- 197 65.98984772
208
- 198 65.65656566
209
- 199 65.32663317
210
- 200 65.50000000
211
- 201 65.67164179
212
- 202 65.84158416
213
- 203 66.00985222
214
- 204 66.17647059
215
- 205 65.85365854
216
- 206 65.53398058
217
- 207 65.70048309
218
- 208 65.86538462
219
- 209 66.02870813
220
- 210 66.19047619
221
- 211 66.35071090
222
- 212 66.03773585
223
- 213 66.19718310
224
- 214 66.35514019
225
- 215 66.51162791
226
- 216 66.20370370
227
- 217 66.35944700
228
- 218 66.05504587
229
- 219 66.21004566
230
- 220 65.90909091
231
- 221 66.06334842
232
- 222 66.21621622
233
- 223 65.91928251
234
- 224 66.07142857
235
- 225 65.77777778
236
- 226 65.92920354
237
- 227 66.07929515
238
- 228 66.22807018
239
- 229 66.37554585
240
- 230 66.52173913
241
- 231 66.23376623
242
- 232 65.94827586
243
- 233 66.09442060
244
- 234 65.81196581
245
- 235 65.53191489
246
- 236 65.25423729
247
- 237 64.97890295
248
- 238 65.12605042
249
- 239 65.27196653
250
- 240 65.41666667
251
- 241 65.56016598
252
- 242 65.70247934
253
- 243 65.43209877
254
- 244 65.16393443
255
- 245 65.30612245
256
- 246 65.44715447
257
- 247 65.58704453
258
- 248 65.72580645
259
- 249 65.46184739
260
- 250 65.60000000
261
- 251 65.73705179
262
- 252 65.87301587
263
- 253 66.00790514
264
- 254 65.74803150
265
- 255 65.88235294
266
- 256 65.62500000
267
- 257 65.75875486
268
- 258 65.89147287
269
- 259 65.63706564
270
- 260 65.76923077
271
- 261 65.90038314
272
- 262 66.03053435
273
- 263 66.15969582
274
- 264 66.28787879
275
- 265 66.41509434
276
- 266 66.16541353
277
- 267 66.29213483
278
- 268 66.41791045
279
- 269 66.54275093
280
- 270 66.66666667
281
- 271 66.78966790
282
- 272 66.54411765
283
- 273 66.30036630
284
- 274 66.05839416
285
- 275 65.81818182
286
- 276 65.94202899
287
- 277 66.06498195
288
- 278 65.82733813
289
- 279 65.94982079
290
- 280 66.07142857
291
- 281 66.19217082
292
- 282 66.31205674
293
- 283 66.07773852
294
- 284 66.19718310
295
- 285 66.31578947
296
- 286 66.43356643
297
- 287 66.55052265
298
- 288 66.66666667
299
- 289 66.43598616
300
- 290 66.55172414
301
- 291 66.66666667
302
- 292 66.78082192
303
- 293 66.89419795
304
- 294 66.66666667
305
- 295 66.77966102
306
- 296 66.89189189
307
- 297 67.00336700
308
- 298 67.11409396
309
- 299 67.22408027
310
- 300 67.33333333
311
- 301 67.10963455
312
- 302 67.21854305
313
- 303 67.32673267
314
- 304 67.43421053
315
- 305 67.54098361
316
- 306 67.32026144
317
- 307 67.42671010
318
- 308 67.53246753
319
- 309 67.63754045
320
- 310 67.74193548
321
- 311 67.84565916
322
- 312 67.62820513
323
- 313 67.73162939
324
- 314 67.51592357
325
- 315 67.61904762
326
- 316 67.72151899
327
- 317 67.82334385
328
- 318 67.61006289
329
- 319 67.39811912
330
- 320 67.18750000
331
- 321 67.28971963
332
- 322 67.39130435
333
- 323 67.49226006
334
- 324 67.28395062
335
- 325 67.38461538
336
- 326 67.48466258
337
- 327 67.27828746
338
- 328 67.37804878
339
- 329 67.47720365
340
- 330 67.27272727
341
- 331 67.37160121
342
- 332 67.46987952
343
- 333 67.56756757
344
- 334 67.66467066
345
- 335 67.46268657
346
- 336 67.55952381
347
- 337 67.65578635
348
- 338 67.45562130
349
- 339 67.55162242
350
- 340 67.64705882
351
- 341 67.74193548
352
- 342 67.83625731
353
- 343 67.93002915
354
- 344 68.02325581
355
- 345 67.82608696
356
- 346 67.91907514
357
- 347 67.72334294
358
- 348 67.52873563
359
- 349 67.62177650
360
- 350 67.42857143
361
- 351 67.52136752
362
- 352 67.61363636
363
- 353 67.70538244
364
- 354 67.51412429
365
- 355 67.32394366
366
- 356 67.41573034
367
- 357 67.22689076
368
- 358 67.31843575
369
- 359 67.13091922
370
- 360 67.22222222
371
- 361 67.31301939
372
- 362 67.40331492
373
- 363 67.21763085
374
- 364 67.30769231
375
- 365 67.39726027
376
- 366 67.21311475
377
- 367 67.02997275
378
- 368 67.11956522
379
- 369 67.20867209
380
- 370 67.29729730
381
- 371 67.38544474
382
- 372 67.47311828
383
- 373 67.56032172
384
- 374 67.64705882
385
- 375 67.46666667
386
- 376 67.55319149
387
- 377 67.37400531
388
- 378 67.46031746
389
- 379 67.28232190
390
- 380 67.36842105
391
- 381 67.19160105
392
- 382 67.01570681
393
- 383 66.84073107
394
- 384 66.92708333
395
- 385 66.75324675
396
- 386 66.83937824
397
- 387 66.92506460
398
- 388 66.75257732
399
- 389 66.83804627
400
- 390 66.92307692
401
- 391 67.00767263
402
- 392 67.09183673
403
- 393 67.17557252
404
- 394 67.25888325
405
- 395 67.34177215
406
- 396 67.17171717
407
- 397 67.25440806
408
- 398 67.08542714
409
- 399 67.16791980
410
- 400 67.00000000
411
- 401 67.08229426
412
- 402 66.91542289
413
- 403 66.99751861
414
- 404 67.07920792
415
- 405 67.16049383
416
- 406 67.24137931
417
- 407 67.32186732
418
- 408 67.40196078
419
- 409 67.23716381
420
- 410 67.31707317
421
- 411 67.39659367
422
- 412 67.47572816
423
- 413 67.55447942
424
- 414 67.63285024
425
- 415 67.46987952
426
- 416 67.54807692
427
- 417 67.38609113
428
- 418 67.46411483
429
- 419 67.30310263
430
- 420 67.38095238
431
- 421 67.22090261
432
- 422 67.29857820
433
- 423 67.37588652
434
- 424 67.21698113
435
- 425 67.29411765
436
- 426 67.13615023
437
- 427 67.21311475
438
- 428 67.05607477
439
- 429 67.13286713
440
- 430 66.97674419
441
- 431 67.05336427
442
- 432 67.12962963
443
- 433 66.97459584
444
- 434 66.82027650
445
- 435 66.89655172
446
- 436 66.97247706
447
- 437 67.04805492
448
- 438 66.89497717
449
- 439 66.97038724
450
- 440 66.81818182
451
- 441 66.89342404
452
- 442 66.74208145
453
- 443 66.59142212
454
- 444 66.66666667
455
- 445 66.74157303
456
- 446 66.81614350
457
- 447 66.89038031
458
- 448 66.96428571
459
- 449 67.03786192
460
- 450 66.88888889
461
- 451 66.96230599
462
- 452 67.03539823
463
- 453 67.10816777
464
- 454 67.18061674
465
- 455 67.03296703
466
- 456 67.10526316
467
- 457 67.17724289
468
- 458 67.24890830
469
- 459 67.32026144
470
- 460 67.17391304
471
- 461 67.24511931
472
- 462 67.31601732
473
- 463 67.17062635
474
- 464 67.02586207
475
- 465 67.09677419
476
- 466 67.16738197
477
- 467 67.23768737
478
- 468 67.30769231
479
- 469 67.16417910
480
- 470 67.23404255
481
- 471 67.30360934
482
- 472 67.37288136
483
- 473 67.23044397
484
- 474 67.29957806
485
- 475 67.15789474
486
- 476 67.22689076
487
- 477 67.29559748
488
- 478 67.36401674
489
- 479 67.43215031
490
- 480 67.50000000
491
- 481 67.56756757
492
- 482 67.63485477
493
- 483 67.70186335
494
- 484 67.76859504
495
- 485 67.83505155
496
- 486 67.90123457
497
- 487 67.76180698
498
- 488 67.82786885
499
- 489 67.89366053
500
- 490 67.75510204
501
- 491 67.61710794
502
- 492 67.68292683
503
- 493 67.54563895
504
- 494 67.61133603
505
- 495 67.47474747
506
- 496 67.54032258
507
- 497 67.60563380
508
- 498 67.67068273
509
- 499 67.73547094
510
- 500 67.80000000
511
- 501 67.66467066
512
- 502 67.72908367
513
- 503 67.79324056
514
- 504 67.65873016
515
- 505 67.72277228
516
- 506 67.58893281
517
- 507 67.45562130
518
- 508 67.51968504
519
- 509 67.38703340
520
- 510 67.45098039
521
- 511 67.31898239
522
- 512 67.38281250
523
- 513 67.25146199
524
- 514 67.31517510
525
- 515 67.37864078
526
- 516 67.44186047
527
- 517 67.50483559
528
- 518 67.56756757
529
- 519 67.43737958
530
- 520 67.30769231
531
- 521 67.37044146
532
- 522 67.43295019
533
- 523 67.30401530
534
- 524 67.17557252
535
- 525 67.04761905
536
- 526 66.92015209
537
- 527 66.79316888
538
- 528 66.66666667
539
- 529 66.54064272
540
- 530 66.60377358
541
- 531 66.47834275
542
- 532 66.54135338
543
- 533 66.60412758
544
- 534 66.47940075
545
- 535 66.35514019
546
- 536 66.41791045
547
- 537 66.48044693
548
- 538 66.54275093
549
- 539 66.60482375
550
- 540 66.48148148
551
- 541 66.54343808
552
- 542 66.60516605
553
- 543 66.48250460
554
- 544 66.54411765
555
- 545 66.60550459
556
- 546 66.48351648
557
- 547 66.36197441
558
- 548 66.42335766
559
- 549 66.48451730
560
- 550 66.54545455
561
- 551 66.42468240
562
- 552 66.30434783
563
- 553 66.36528029
564
- 554 66.42599278
565
- 555 66.48648649
566
- 556 66.36690647
567
- 557 66.42728905
568
- 558 66.30824373
569
- 559 66.36851521
570
- 560 66.42857143
571
- 561 66.48841355
572
- 562 66.54804270
573
- 563 66.60746004
574
- 564 66.66666667
575
- 565 66.54867257
576
- 566 66.60777385
577
- 567 66.49029982
578
- 568 66.54929577
579
- 569 66.43233743
580
- 570 66.31578947
581
- 571 66.37478109
582
- 572 66.25874126
583
- 573 66.31762653
584
- 574 66.37630662
585
- 575 66.43478261
586
- 576 66.49305556
587
- 577 66.55112652
588
- 578 66.60899654
589
- 579 66.66666667
590
- 580 66.72413793
591
- 581 66.60929432
592
- 582 66.66666667
593
- 583 66.72384220
594
- 584 66.78082192
595
- 585 66.83760684
596
- 586 66.72354949
597
- 587 66.78023850
598
- 588 66.83673469
599
- 589 66.89303905
600
- 590 66.94915254
601
- 591 67.00507614
602
- 592 66.89189189
603
- 593 66.77908938
604
- 594 66.83501684
605
- 595 66.89075630
606
- 596 66.77852349
607
- 597 66.66666667
608
- 598 66.55518395
609
- 599 66.61101836
610
- 600 66.66666667
611
- 601 66.72212978
612
- 602 66.61129568
613
- 603 66.50082919
614
- 604 66.55629139
615
- 605 66.44628099
616
- 606 66.50165017
617
- 607 66.39209226
618
- 608 66.44736842
619
- 609 66.33825944
620
- 610 66.39344262
621
- 611 66.28477905
622
- 612 66.33986928
623
- 613 66.23164763
624
- 614 66.28664495
625
- 615 66.17886179
626
- 616 66.07142857
627
- 617 66.12641815
628
- 618 66.18122977
629
- 619 66.07431341
630
- 620 66.12903226
631
- 621 66.18357488
632
- 622 66.07717042
633
- 623 66.13162119
634
- 624 66.18589744
635
- 625 66.08000000
636
- 626 65.97444089
637
- 627 66.02870813
638
- 628 66.08280255
639
- 629 65.97774245
640
- 630 66.03174603
641
- 631 66.08557845
642
- 632 65.98101266
643
- 633 65.87677725
644
- 634 65.93059937
645
- 635 65.98425197
646
- 636 66.03773585
647
- 637 65.93406593
648
- 638 65.98746082
649
- 639 66.04068858
650
- 640 66.09375000
651
- 641 66.14664587
652
- 642 66.04361371
653
- 643 65.94090202
654
- 644 65.99378882
655
- 645 65.89147287
656
- 646 65.94427245
657
- 647 65.99690881
658
- 648 65.89506173
659
- 649 65.79352851
660
- 650 65.84615385
661
- 651 65.89861751
662
- 652 65.95092025
663
- 653 65.84992343
664
- 654 65.74923547
665
- 655 65.64885496
666
- 656 65.54878049
667
- 657 65.44901065
668
- 658 65.50151976
669
- 659 65.40212443
670
- 660 65.45454545
671
- 661 65.35552194
672
- 662 65.40785498
673
- 663 65.46003017
674
- 664 65.51204819
675
- 665 65.56390977
676
- 666 65.46546547
677
- 667 65.51724138
678
- 668 65.41916168
679
- 669 65.32137519
680
- 670 65.37313433
681
- 671 65.27570790
682
- 672 65.32738095
683
- 673 65.23031204
684
- 674 65.28189911
685
- 675 65.18518519
686
- 676 65.08875740
687
- 677 65.14032496
688
- 678 65.19174041
689
- 679 65.24300442
690
- 680 65.29411765
691
- 681 65.19823789
692
- 682 65.10263930
693
- 683 65.00732064
694
- 684 65.05847953
695
- 685 65.10948905
696
- 686 65.01457726
697
- 687 64.91994178
698
- 688 64.97093023
699
- 689 65.02177068
700
- 690 65.07246377
701
- 691 65.12301013
702
- 692 65.17341040
703
- 693 65.22366522
704
- 694 65.27377522
705
- 695 65.17985612
706
- 696 65.08620690
707
- 697 64.99282640
708
- 698 64.89971347
709
- 699 64.94992847
710
- 700 65.00000000
711
- 701 65.04992867
712
- 702 65.09971510
713
- 703 65.14935989
714
- 704 65.19886364
715
- 705 65.24822695
716
- 706 65.29745042
717
- 707 65.34653465
718
- 708 65.39548023
719
- 709 65.44428773
720
- 710 65.49295775
721
- 711 65.54149086
722
- 712 65.58988764
723
- 713 65.63814867
724
- 714 65.54621849
725
- 715 65.45454545
726
- 716 65.50279330
727
- 717 65.55090656
728
- 718 65.45961003
729
- 719 65.36856745
730
- 720 65.41666667
731
- 721 65.32593620
732
- 722 65.37396122
733
- 723 65.42185339
734
- 724 65.33149171
735
- 725 65.37931034
736
- 726 65.28925620
737
- 727 65.19944979
738
- 728 65.24725275
739
- 729 65.29492455
740
- 730 65.34246575
741
- 731 65.38987688
742
- 732 65.43715847
743
- 733 65.48431105
744
- 734 65.53133515
745
- 735 65.57823129
746
- 736 65.62500000
747
- 737 65.53595658
748
- 738 65.58265583
749
- 739 65.62922869
750
- 740 65.54054054
751
- 741 65.58704453
752
- 742 65.49865229
753
- 743 65.54508748
754
- 744 65.45698925
755
- 745 65.50335570
756
- 746 65.54959786
757
- 747 65.46184739
758
- 748 65.37433155
759
- 749 65.28704940
760
- 750 65.33333333
761
-
762
- Final result: 65.3333 +/- 1.7389
763
- Random chance: 25.0083 +/- 1.5824
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scores/Watt-Tool-8B-q4_k_m-naive.hsw DELETED
@@ -1,760 +0,0 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
3
-
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- hellaswag_score : loaded 10042 tasks from prompt.
6
- ================================= is_spm = 0
7
- hellaswag_score : selecting 750 randomized tasks.
8
- hellaswag_score : calculating hellaswag score over selected tasks.
9
-
10
- task acc_norm
11
- 1 0.00000000
12
- 2 50.00000000
13
- 3 66.66666667
14
- 4 75.00000000
15
- 5 80.00000000
16
- 6 83.33333333
17
- 7 85.71428571
18
- 8 87.50000000
19
- 9 88.88888889
20
- 10 90.00000000
21
- 11 90.90909091
22
- 12 83.33333333
23
- 13 84.61538462
24
- 14 85.71428571
25
- 15 86.66666667
26
- 16 87.50000000
27
- 17 82.35294118
28
- 18 83.33333333
29
- 19 78.94736842
30
- 20 80.00000000
31
- 21 80.95238095
32
- 22 81.81818182
33
- 23 82.60869565
34
- 24 79.16666667
35
- 25 76.00000000
36
- 26 73.07692308
37
- 27 70.37037037
38
- 28 71.42857143
39
- 29 72.41379310
40
- 30 70.00000000
41
- 31 70.96774194
42
- 32 68.75000000
43
- 33 69.69696970
44
- 34 70.58823529
45
- 35 71.42857143
46
- 36 72.22222222
47
- 37 70.27027027
48
- 38 71.05263158
49
- 39 71.79487179
50
- 40 70.00000000
51
- 41 70.73170732
52
- 42 69.04761905
53
- 43 69.76744186
54
- 44 70.45454545
55
- 45 68.88888889
56
- 46 69.56521739
57
- 47 70.21276596
58
- 48 70.83333333
59
- 49 71.42857143
60
- 50 72.00000000
61
- 51 72.54901961
62
- 52 71.15384615
63
- 53 69.81132075
64
- 54 68.51851852
65
- 55 69.09090909
66
- 56 67.85714286
67
- 57 68.42105263
68
- 58 68.96551724
69
- 59 69.49152542
70
- 60 70.00000000
71
- 61 70.49180328
72
- 62 69.35483871
73
- 63 69.84126984
74
- 64 70.31250000
75
- 65 70.76923077
76
- 66 69.69696970
77
- 67 70.14925373
78
- 68 70.58823529
79
- 69 71.01449275
80
- 70 71.42857143
81
- 71 70.42253521
82
- 72 70.83333333
83
- 73 71.23287671
84
- 74 71.62162162
85
- 75 70.66666667
86
- 76 71.05263158
87
- 77 71.42857143
88
- 78 71.79487179
89
- 79 72.15189873
90
- 80 72.50000000
91
- 81 72.83950617
92
- 82 73.17073171
93
- 83 73.49397590
94
- 84 73.80952381
95
- 85 74.11764706
96
- 86 74.41860465
97
- 87 74.71264368
98
- 88 75.00000000
99
- 89 74.15730337
100
- 90 74.44444444
101
- 91 74.72527473
102
- 92 73.91304348
103
- 93 74.19354839
104
- 94 74.46808511
105
- 95 74.73684211
106
- 96 75.00000000
107
- 97 75.25773196
108
- 98 75.51020408
109
- 99 75.75757576
110
- 100 75.00000000
111
- 101 75.24752475
112
- 102 74.50980392
113
- 103 73.78640777
114
- 104 74.03846154
115
- 105 74.28571429
116
- 106 74.52830189
117
- 107 74.76635514
118
- 108 75.00000000
119
- 109 74.31192661
120
- 110 74.54545455
121
- 111 74.77477477
122
- 112 75.00000000
123
- 113 75.22123894
124
- 114 75.43859649
125
- 115 75.65217391
126
- 116 75.00000000
127
- 117 74.35897436
128
- 118 74.57627119
129
- 119 74.78991597
130
- 120 74.16666667
131
- 121 74.38016529
132
- 122 74.59016393
133
- 123 74.79674797
134
- 124 75.00000000
135
- 125 75.20000000
136
- 126 75.39682540
137
- 127 75.59055118
138
- 128 75.00000000
139
- 129 75.19379845
140
- 130 74.61538462
141
- 131 74.80916031
142
- 132 75.00000000
143
- 133 75.18796992
144
- 134 75.37313433
145
- 135 75.55555556
146
- 136 75.73529412
147
- 137 75.91240876
148
- 138 76.08695652
149
- 139 75.53956835
150
- 140 75.71428571
151
- 141 75.88652482
152
- 142 76.05633803
153
- 143 76.22377622
154
- 144 76.38888889
155
- 145 76.55172414
156
- 146 76.71232877
157
- 147 76.87074830
158
- 148 77.02702703
159
- 149 77.18120805
160
- 150 77.33333333
161
- 151 76.82119205
162
- 152 76.31578947
163
- 153 76.47058824
164
- 154 76.62337662
165
- 155 76.77419355
166
- 156 76.28205128
167
- 157 75.79617834
168
- 158 75.94936709
169
- 159 75.47169811
170
- 160 75.62500000
171
- 161 75.77639752
172
- 162 75.92592593
173
- 163 76.07361963
174
- 164 76.21951220
175
- 165 76.36363636
176
- 166 76.50602410
177
- 167 76.64670659
178
- 168 76.78571429
179
- 169 76.33136095
180
- 170 76.47058824
181
- 171 76.60818713
182
- 172 76.74418605
183
- 173 76.87861272
184
- 174 77.01149425
185
- 175 76.57142857
186
- 176 76.70454545
187
- 177 76.83615819
188
- 178 76.96629213
189
- 179 76.53631285
190
- 180 76.66666667
191
- 181 76.24309392
192
- 182 76.37362637
193
- 183 76.50273224
194
- 184 76.63043478
195
- 185 76.75675676
196
- 186 76.34408602
197
- 187 75.93582888
198
- 188 75.53191489
199
- 189 75.13227513
200
- 190 75.26315789
201
- 191 75.39267016
202
- 192 75.52083333
203
- 193 75.12953368
204
- 194 75.25773196
205
- 195 74.87179487
206
- 196 75.00000000
207
- 197 74.61928934
208
- 198 74.24242424
209
- 199 74.37185930
210
- 200 74.50000000
211
- 201 74.62686567
212
- 202 74.75247525
213
- 203 74.38423645
214
- 204 74.50980392
215
- 205 74.63414634
216
- 206 74.75728155
217
- 207 74.87922705
218
- 208 74.51923077
219
- 209 74.64114833
220
- 210 74.76190476
221
- 211 74.88151659
222
- 212 75.00000000
223
- 213 75.11737089
224
- 214 75.23364486
225
- 215 74.88372093
226
- 216 75.00000000
227
- 217 74.65437788
228
- 218 74.77064220
229
- 219 74.88584475
230
- 220 75.00000000
231
- 221 74.66063348
232
- 222 74.77477477
233
- 223 74.88789238
234
- 224 75.00000000
235
- 225 75.11111111
236
- 226 75.22123894
237
- 227 74.88986784
238
- 228 75.00000000
239
- 229 75.10917031
240
- 230 74.78260870
241
- 231 74.45887446
242
- 232 74.13793103
243
- 233 74.24892704
244
- 234 74.35897436
245
- 235 74.04255319
246
- 236 73.72881356
247
- 237 73.83966245
248
- 238 73.94957983
249
- 239 74.05857741
250
- 240 74.16666667
251
- 241 74.27385892
252
- 242 74.38016529
253
- 243 74.48559671
254
- 244 74.59016393
255
- 245 74.28571429
256
- 246 74.39024390
257
- 247 74.49392713
258
- 248 74.59677419
259
- 249 74.69879518
260
- 250 74.80000000
261
- 251 74.90039841
262
- 252 75.00000000
263
- 253 74.70355731
264
- 254 74.80314961
265
- 255 74.90196078
266
- 256 75.00000000
267
- 257 74.70817121
268
- 258 74.80620155
269
- 259 74.90347490
270
- 260 75.00000000
271
- 261 75.09578544
272
- 262 74.80916031
273
- 263 74.90494297
274
- 264 75.00000000
275
- 265 75.09433962
276
- 266 75.18796992
277
- 267 75.28089888
278
- 268 75.37313433
279
- 269 75.46468401
280
- 270 75.55555556
281
- 271 75.27675277
282
- 272 75.36764706
283
- 273 75.09157509
284
- 274 75.18248175
285
- 275 74.90909091
286
- 276 75.00000000
287
- 277 75.09025271
288
- 278 75.17985612
289
- 279 75.26881720
290
- 280 75.35714286
291
- 281 75.44483986
292
- 282 75.53191489
293
- 283 75.61837456
294
- 284 75.70422535
295
- 285 75.43859649
296
- 286 75.52447552
297
- 287 75.60975610
298
- 288 75.69444444
299
- 289 75.77854671
300
- 290 75.86206897
301
- 291 75.94501718
302
- 292 75.68493151
303
- 293 75.76791809
304
- 294 75.85034014
305
- 295 75.93220339
306
- 296 76.01351351
307
- 297 76.09427609
308
- 298 76.17449664
309
- 299 75.91973244
310
- 300 75.66666667
311
- 301 75.74750831
312
- 302 75.82781457
313
- 303 75.57755776
314
- 304 75.32894737
315
- 305 75.40983607
316
- 306 75.49019608
317
- 307 75.24429967
318
- 308 75.32467532
319
- 309 75.40453074
320
- 310 75.48387097
321
- 311 75.56270096
322
- 312 75.64102564
323
- 313 75.71884984
324
- 314 75.79617834
325
- 315 75.87301587
326
- 316 75.94936709
327
- 317 76.02523659
328
- 318 75.78616352
329
- 319 75.54858934
330
- 320 75.62500000
331
- 321 75.38940810
332
- 322 75.15527950
333
- 323 75.23219814
334
- 324 75.30864198
335
- 325 75.38461538
336
- 326 75.46012270
337
- 327 75.22935780
338
- 328 75.30487805
339
- 329 75.37993921
340
- 330 75.45454545
341
- 331 75.52870091
342
- 332 75.60240964
343
- 333 75.67567568
344
- 334 75.44910180
345
- 335 75.52238806
346
- 336 75.59523810
347
- 337 75.37091988
348
- 338 75.14792899
349
- 339 75.22123894
350
- 340 75.29411765
351
- 341 75.36656891
352
- 342 75.43859649
353
- 343 75.51020408
354
- 344 75.29069767
355
- 345 75.36231884
356
- 346 75.14450867
357
- 347 75.21613833
358
- 348 75.00000000
359
- 349 74.78510029
360
- 350 74.85714286
361
- 351 74.92877493
362
- 352 75.00000000
363
- 353 75.07082153
364
- 354 75.14124294
365
- 355 75.21126761
366
- 356 75.28089888
367
- 357 75.35014006
368
- 358 75.41899441
369
- 359 75.48746518
370
- 360 75.27777778
371
- 361 75.06925208
372
- 362 75.13812155
373
- 363 75.20661157
374
- 364 75.27472527
375
- 365 75.34246575
376
- 366 75.40983607
377
- 367 75.47683924
378
- 368 75.54347826
379
- 369 75.60975610
380
- 370 75.67567568
381
- 371 75.74123989
382
- 372 75.80645161
383
- 373 75.60321716
384
- 374 75.66844920
385
- 375 75.73333333
386
- 376 75.79787234
387
- 377 75.59681698
388
- 378 75.66137566
389
- 379 75.72559367
390
- 380 75.78947368
391
- 381 75.85301837
392
- 382 75.91623037
393
- 383 75.97911227
394
- 384 76.04166667
395
- 385 76.10389610
396
- 386 76.16580311
397
- 387 76.22739018
398
- 388 76.28865979
399
- 389 76.34961440
400
- 390 76.41025641
401
- 391 76.21483376
402
- 392 76.27551020
403
- 393 76.33587786
404
- 394 76.39593909
405
- 395 76.45569620
406
- 396 76.51515152
407
- 397 76.57430730
408
- 398 76.63316583
409
- 399 76.69172932
410
- 400 76.75000000
411
- 401 76.80798005
412
- 402 76.86567164
413
- 403 76.67493797
414
- 404 76.73267327
415
- 405 76.54320988
416
- 406 76.60098522
417
- 407 76.65847666
418
- 408 76.71568627
419
- 409 76.77261614
420
- 410 76.58536585
421
- 411 76.39902676
422
- 412 76.45631068
423
- 413 76.27118644
424
- 414 76.32850242
425
- 415 76.14457831
426
- 416 76.20192308
427
- 417 76.25899281
428
- 418 76.31578947
429
- 419 76.37231504
430
- 420 76.42857143
431
- 421 76.48456057
432
- 422 76.54028436
433
- 423 76.59574468
434
- 424 76.65094340
435
- 425 76.70588235
436
- 426 76.76056338
437
- 427 76.58079625
438
- 428 76.63551402
439
- 429 76.45687646
440
- 430 76.51162791
441
- 431 76.33410673
442
- 432 76.15740741
443
- 433 76.21247113
444
- 434 76.26728111
445
- 435 76.09195402
446
- 436 76.14678899
447
- 437 75.97254005
448
- 438 76.02739726
449
- 439 76.08200456
450
- 440 76.13636364
451
- 441 76.19047619
452
- 442 76.24434389
453
- 443 76.29796840
454
- 444 76.35135135
455
- 445 76.40449438
456
- 446 76.45739910
457
- 447 76.51006711
458
- 448 76.56250000
459
- 449 76.61469933
460
- 450 76.66666667
461
- 451 76.49667406
462
- 452 76.54867257
463
- 453 76.60044150
464
- 454 76.65198238
465
- 455 76.70329670
466
- 456 76.75438596
467
- 457 76.80525164
468
- 458 76.85589520
469
- 459 76.90631808
470
- 460 76.95652174
471
- 461 77.00650759
472
- 462 77.05627706
473
- 463 77.10583153
474
- 464 77.15517241
475
- 465 76.98924731
476
- 466 77.03862661
477
- 467 77.08779443
478
- 468 76.92307692
479
- 469 76.75906183
480
- 470 76.59574468
481
- 471 76.64543524
482
- 472 76.69491525
483
- 473 76.74418605
484
- 474 76.79324895
485
- 475 76.84210526
486
- 476 76.89075630
487
- 477 76.72955975
488
- 478 76.77824268
489
- 479 76.82672234
490
- 480 76.87500000
491
- 481 76.92307692
492
- 482 76.76348548
493
- 483 76.81159420
494
- 484 76.65289256
495
- 485 76.49484536
496
- 486 76.54320988
497
- 487 76.59137577
498
- 488 76.43442623
499
- 489 76.48261759
500
- 490 76.53061224
501
- 491 76.57841141
502
- 492 76.62601626
503
- 493 76.67342799
504
- 494 76.72064777
505
- 495 76.76767677
506
- 496 76.81451613
507
- 497 76.86116700
508
- 498 76.90763052
509
- 499 76.75350701
510
- 500 76.80000000
511
- 501 76.84630739
512
- 502 76.89243028
513
- 503 76.93836978
514
- 504 76.98412698
515
- 505 77.02970297
516
- 506 77.07509881
517
- 507 77.12031558
518
- 508 77.16535433
519
- 509 77.21021611
520
- 510 77.25490196
521
- 511 77.29941292
522
- 512 77.14843750
523
- 513 77.19298246
524
- 514 77.23735409
525
- 515 77.28155340
526
- 516 77.13178295
527
- 517 77.17601547
528
- 518 77.22007722
529
- 519 77.26396917
530
- 520 77.30769231
531
- 521 77.15930902
532
- 522 77.20306513
533
- 523 77.24665392
534
- 524 77.29007634
535
- 525 77.33333333
536
- 526 77.18631179
537
- 527 77.03984820
538
- 528 77.08333333
539
- 529 77.12665406
540
- 530 77.16981132
541
- 531 77.21280603
542
- 532 77.25563910
543
- 533 77.29831144
544
- 534 77.34082397
545
- 535 77.38317757
546
- 536 77.42537313
547
- 537 77.46741155
548
- 538 77.50929368
549
- 539 77.36549165
550
- 540 77.40740741
551
- 541 77.44916821
552
- 542 77.49077491
553
- 543 77.53222836
554
- 544 77.57352941
555
- 545 77.61467890
556
- 546 77.65567766
557
- 547 77.69652651
558
- 548 77.73722628
559
- 549 77.77777778
560
- 550 77.63636364
561
- 551 77.67695100
562
- 552 77.53623188
563
- 553 77.57685353
564
- 554 77.61732852
565
- 555 77.47747748
566
- 556 77.51798561
567
- 557 77.37881508
568
- 558 77.24014337
569
- 559 77.28085868
570
- 560 77.32142857
571
- 561 77.36185383
572
- 562 77.22419929
573
- 563 77.26465364
574
- 564 77.30496454
575
- 565 77.16814159
576
- 566 77.03180212
577
- 567 77.07231041
578
- 568 77.11267606
579
- 569 76.97715290
580
- 570 77.01754386
581
- 571 77.05779335
582
- 572 77.09790210
583
- 573 77.13787086
584
- 574 77.17770035
585
- 575 77.21739130
586
- 576 77.25694444
587
- 577 77.12305026
588
- 578 77.16262976
589
- 579 77.20207254
590
- 580 77.24137931
591
- 581 77.10843373
592
- 582 77.14776632
593
- 583 77.18696398
594
- 584 77.22602740
595
- 585 77.26495726
596
- 586 77.30375427
597
- 587 77.34241908
598
- 588 77.21088435
599
- 589 77.07979626
600
- 590 77.11864407
601
- 591 77.15736041
602
- 592 77.19594595
603
- 593 77.06576728
604
- 594 76.93602694
605
- 595 76.97478992
606
- 596 77.01342282
607
- 597 77.05192630
608
- 598 77.09030100
609
- 599 76.96160267
610
- 600 77.00000000
611
- 601 77.03826955
612
- 602 77.07641196
613
- 603 77.11442786
614
- 604 77.15231788
615
- 605 77.19008264
616
- 606 77.06270627
617
- 607 77.10049423
618
- 608 76.97368421
619
- 609 77.01149425
620
- 610 77.04918033
621
- 611 77.08674304
622
- 612 76.96078431
623
- 613 76.99836868
624
- 614 77.03583062
625
- 615 77.07317073
626
- 616 76.94805195
627
- 617 76.98541329
628
- 618 77.02265372
629
- 619 77.05977383
630
- 620 77.09677419
631
- 621 77.13365539
632
- 622 77.17041801
633
- 623 77.20706260
634
- 624 77.24358974
635
- 625 77.28000000
636
- 626 77.31629393
637
- 627 77.35247209
638
- 628 77.38853503
639
- 629 77.26550079
640
- 630 77.30158730
641
- 631 77.33755943
642
- 632 77.21518987
643
- 633 77.25118483
644
- 634 77.28706625
645
- 635 77.32283465
646
- 636 77.35849057
647
- 637 77.39403454
648
- 638 77.42946708
649
- 639 77.46478873
650
- 640 77.34375000
651
- 641 77.37909516
652
- 642 77.41433022
653
- 643 77.44945568
654
- 644 77.48447205
655
- 645 77.36434109
656
- 646 77.39938080
657
- 647 77.43431221
658
- 648 77.46913580
659
- 649 77.34976888
660
- 650 77.38461538
661
- 651 77.41935484
662
- 652 77.45398773
663
- 653 77.48851455
664
- 654 77.52293578
665
- 655 77.55725191
666
- 656 77.59146341
667
- 657 77.62557078
668
- 658 77.65957447
669
- 659 77.54172989
670
- 660 77.42424242
671
- 661 77.45839637
672
- 662 77.49244713
673
- 663 77.52639517
674
- 664 77.56024096
675
- 665 77.59398496
676
- 666 77.62762763
677
- 667 77.66116942
678
- 668 77.69461078
679
- 669 77.72795217
680
- 670 77.76119403
681
- 671 77.79433681
682
- 672 77.82738095
683
- 673 77.86032689
684
- 674 77.89317507
685
- 675 77.92592593
686
- 676 77.95857988
687
- 677 77.99113737
688
- 678 78.02359882
689
- 679 78.05596465
690
- 680 78.08823529
691
- 681 77.97356828
692
- 682 78.00586510
693
- 683 78.03806735
694
- 684 78.07017544
695
- 685 78.10218978
696
- 686 78.13411079
697
- 687 78.16593886
698
- 688 78.19767442
699
- 689 78.22931785
700
- 690 78.26086957
701
- 691 78.29232996
702
- 692 78.32369942
703
- 693 78.21067821
704
- 694 78.24207493
705
- 695 78.27338129
706
- 696 78.30459770
707
- 697 78.33572453
708
- 698 78.36676218
709
- 699 78.39771102
710
- 700 78.42857143
711
- 701 78.45934379
712
- 702 78.49002849
713
- 703 78.52062589
714
- 704 78.55113636
715
- 705 78.58156028
716
- 706 78.61189802
717
- 707 78.64214993
718
- 708 78.67231638
719
- 709 78.70239774
720
- 710 78.73239437
721
- 711 78.76230661
722
- 712 78.79213483
723
- 713 78.82187938
724
- 714 78.85154062
725
- 715 78.74125874
726
- 716 78.77094972
727
- 717 78.80055788
728
- 718 78.83008357
729
- 719 78.85952712
730
- 720 78.88888889
731
- 721 78.77947295
732
- 722 78.80886427
733
- 723 78.83817427
734
- 724 78.86740331
735
- 725 78.89655172
736
- 726 78.92561983
737
- 727 78.95460798
738
- 728 78.98351648
739
- 729 79.01234568
740
- 730 79.04109589
741
- 731 79.06976744
742
- 732 78.96174863
743
- 733 78.85402456
744
- 734 78.88283379
745
- 735 78.77551020
746
- 736 78.80434783
747
- 737 78.69742198
748
- 738 78.59078591
749
- 739 78.48443843
750
- 740 78.51351351
751
- 741 78.54251012
752
- 742 78.57142857
753
- 743 78.60026918
754
- 744 78.62903226
755
- 745 78.65771812
756
- 746 78.68632708
757
- 747 78.58099063
758
- 748 78.60962567
759
- 749 78.63818425
760
- 750 78.66666667
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scores/Watt-Tool-8B-q4_k_m-naive.mmlu DELETED
@@ -1,763 +0,0 @@
1
- common_init_from_params: setting dry_penalty_last_n to ctx_size = 768
2
- common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
3
-
4
- system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |
5
- multiple_choice_score: there are 1548 tasks in prompt
6
- multiple_choice_score: selecting 750 random tasks from 1548 tasks available
7
- multiple_choice_score: preparing task data...done
8
- multiple_choice_score : calculating TruthfulQA score over 750 tasks.
9
-
10
- task acc_norm
11
- 1 100.00000000
12
- 2 50.00000000
13
- 3 33.33333333
14
- 4 50.00000000
15
- 5 40.00000000
16
- 6 33.33333333
17
- 7 42.85714286
18
- 8 50.00000000
19
- 9 44.44444444
20
- 10 50.00000000
21
- 11 45.45454545
22
- 12 41.66666667
23
- 13 46.15384615
24
- 14 42.85714286
25
- 15 46.66666667
26
- 16 50.00000000
27
- 17 52.94117647
28
- 18 50.00000000
29
- 19 52.63157895
30
- 20 55.00000000
31
- 21 57.14285714
32
- 22 54.54545455
33
- 23 52.17391304
34
- 24 54.16666667
35
- 25 56.00000000
36
- 26 53.84615385
37
- 27 55.55555556
38
- 28 57.14285714
39
- 29 58.62068966
40
- 30 60.00000000
41
- 31 61.29032258
42
- 32 62.50000000
43
- 33 60.60606061
44
- 34 61.76470588
45
- 35 62.85714286
46
- 36 63.88888889
47
- 37 62.16216216
48
- 38 63.15789474
49
- 39 61.53846154
50
- 40 62.50000000
51
- 41 60.97560976
52
- 42 59.52380952
53
- 43 60.46511628
54
- 44 59.09090909
55
- 45 60.00000000
56
- 46 58.69565217
57
- 47 57.44680851
58
- 48 58.33333333
59
- 49 59.18367347
60
- 50 58.00000000
61
- 51 56.86274510
62
- 52 55.76923077
63
- 53 54.71698113
64
- 54 53.70370370
65
- 55 54.54545455
66
- 56 55.35714286
67
- 57 56.14035088
68
- 58 55.17241379
69
- 59 54.23728814
70
- 60 53.33333333
71
- 61 54.09836066
72
- 62 53.22580645
73
- 63 52.38095238
74
- 64 51.56250000
75
- 65 50.76923077
76
- 66 50.00000000
77
- 67 49.25373134
78
- 68 48.52941176
79
- 69 47.82608696
80
- 70 48.57142857
81
- 71 49.29577465
82
- 72 48.61111111
83
- 73 47.94520548
84
- 74 48.64864865
85
- 75 49.33333333
86
- 76 50.00000000
87
- 77 50.64935065
88
- 78 51.28205128
89
- 79 50.63291139
90
- 80 51.25000000
91
- 81 51.85185185
92
- 82 51.21951220
93
- 83 51.80722892
94
- 84 52.38095238
95
- 85 51.76470588
96
- 86 51.16279070
97
- 87 51.72413793
98
- 88 52.27272727
99
- 89 51.68539326
100
- 90 51.11111111
101
- 91 50.54945055
102
- 92 50.00000000
103
- 93 49.46236559
104
- 94 50.00000000
105
- 95 49.47368421
106
- 96 50.00000000
107
- 97 50.51546392
108
- 98 50.00000000
109
- 99 50.50505051
110
- 100 50.00000000
111
- 101 49.50495050
112
- 102 49.01960784
113
- 103 48.54368932
114
- 104 49.03846154
115
- 105 49.52380952
116
- 106 50.00000000
117
- 107 49.53271028
118
- 108 50.00000000
119
- 109 50.45871560
120
- 110 50.90909091
121
- 111 51.35135135
122
- 112 50.89285714
123
- 113 50.44247788
124
- 114 50.87719298
125
- 115 51.30434783
126
- 116 51.72413793
127
- 117 51.28205128
128
- 118 50.84745763
129
- 119 51.26050420
130
- 120 50.83333333
131
- 121 51.23966942
132
- 122 50.81967213
133
- 123 50.40650407
134
- 124 50.00000000
135
- 125 50.40000000
136
- 126 50.00000000
137
- 127 49.60629921
138
- 128 49.21875000
139
- 129 48.83720930
140
- 130 48.46153846
141
- 131 48.85496183
142
- 132 49.24242424
143
- 133 48.87218045
144
- 134 48.50746269
145
- 135 48.14814815
146
- 136 47.79411765
147
- 137 48.17518248
148
- 138 47.82608696
149
- 139 48.20143885
150
- 140 48.57142857
151
- 141 48.93617021
152
- 142 49.29577465
153
- 143 49.65034965
154
- 144 50.00000000
155
- 145 50.34482759
156
- 146 50.68493151
157
- 147 50.34013605
158
- 148 50.67567568
159
- 149 50.33557047
160
- 150 50.00000000
161
- 151 49.66887417
162
- 152 50.00000000
163
- 153 50.32679739
164
- 154 50.00000000
165
- 155 49.67741935
166
- 156 49.35897436
167
- 157 49.68152866
168
- 158 50.00000000
169
- 159 49.68553459
170
- 160 50.00000000
171
- 161 49.68944099
172
- 162 49.38271605
173
- 163 49.07975460
174
- 164 48.78048780
175
- 165 49.09090909
176
- 166 48.79518072
177
- 167 49.10179641
178
- 168 48.80952381
179
- 169 49.11242604
180
- 170 48.82352941
181
- 171 49.12280702
182
- 172 48.83720930
183
- 173 48.55491329
184
- 174 48.85057471
185
- 175 48.57142857
186
- 176 48.29545455
187
- 177 48.02259887
188
- 178 47.75280899
189
- 179 48.04469274
190
- 180 47.77777778
191
- 181 47.51381215
192
- 182 47.25274725
193
- 183 47.54098361
194
- 184 47.28260870
195
- 185 47.56756757
196
- 186 47.84946237
197
- 187 47.59358289
198
- 188 47.87234043
199
- 189 48.14814815
200
- 190 47.89473684
201
- 191 48.16753927
202
- 192 47.91666667
203
- 193 47.66839378
204
- 194 47.93814433
205
- 195 48.20512821
206
- 196 48.46938776
207
- 197 48.22335025
208
- 198 48.48484848
209
- 199 48.24120603
210
- 200 48.00000000
211
- 201 47.76119403
212
- 202 47.52475248
213
- 203 47.29064039
214
- 204 47.05882353
215
- 205 47.31707317
216
- 206 47.08737864
217
- 207 47.34299517
218
- 208 47.11538462
219
- 209 46.88995215
220
- 210 46.66666667
221
- 211 46.91943128
222
- 212 46.69811321
223
- 213 46.47887324
224
- 214 46.26168224
225
- 215 46.51162791
226
- 216 46.75925926
227
- 217 46.54377880
228
- 218 46.78899083
229
- 219 47.03196347
230
- 220 47.27272727
231
- 221 47.05882353
232
- 222 46.84684685
233
- 223 47.08520179
234
- 224 47.32142857
235
- 225 47.55555556
236
- 226 47.34513274
237
- 227 47.13656388
238
- 228 47.36842105
239
- 229 47.16157205
240
- 230 46.95652174
241
- 231 46.75324675
242
- 232 46.98275862
243
- 233 47.21030043
244
- 234 47.43589744
245
- 235 47.23404255
246
- 236 47.03389831
247
- 237 46.83544304
248
- 238 46.63865546
249
- 239 46.44351464
250
- 240 46.66666667
251
- 241 46.47302905
252
- 242 46.69421488
253
- 243 46.91358025
254
- 244 46.72131148
255
- 245 46.93877551
256
- 246 46.74796748
257
- 247 46.96356275
258
- 248 46.77419355
259
- 249 46.58634538
260
- 250 46.40000000
261
- 251 46.21513944
262
- 252 46.03174603
263
- 253 45.84980237
264
- 254 45.66929134
265
- 255 45.49019608
266
- 256 45.70312500
267
- 257 45.91439689
268
- 258 45.73643411
269
- 259 45.55984556
270
- 260 45.76923077
271
- 261 45.59386973
272
- 262 45.41984733
273
- 263 45.24714829
274
- 264 45.07575758
275
- 265 45.28301887
276
- 266 45.48872180
277
- 267 45.31835206
278
- 268 45.14925373
279
- 269 45.35315985
280
- 270 45.55555556
281
- 271 45.38745387
282
- 272 45.58823529
283
- 273 45.42124542
284
- 274 45.25547445
285
- 275 45.09090909
286
- 276 45.28985507
287
- 277 45.48736462
288
- 278 45.32374101
289
- 279 45.16129032
290
- 280 45.00000000
291
- 281 44.83985765
292
- 282 44.68085106
293
- 283 44.52296820
294
- 284 44.36619718
295
- 285 44.21052632
296
- 286 44.05594406
297
- 287 44.25087108
298
- 288 44.09722222
299
- 289 43.94463668
300
- 290 44.13793103
301
- 291 44.32989691
302
- 292 44.52054795
303
- 293 44.70989761
304
- 294 44.55782313
305
- 295 44.40677966
306
- 296 44.25675676
307
- 297 44.10774411
308
- 298 44.29530201
309
- 299 44.48160535
310
- 300 44.66666667
311
- 301 44.51827243
312
- 302 44.70198675
313
- 303 44.55445545
314
- 304 44.40789474
315
- 305 44.26229508
316
- 306 44.11764706
317
- 307 44.29967427
318
- 308 44.15584416
319
- 309 44.01294498
320
- 310 43.87096774
321
- 311 44.05144695
322
- 312 44.23076923
323
- 313 44.08945687
324
- 314 43.94904459
325
- 315 43.80952381
326
- 316 43.67088608
327
- 317 43.84858044
328
- 318 43.71069182
329
- 319 43.88714734
330
- 320 43.75000000
331
- 321 43.61370717
332
- 322 43.78881988
333
- 323 43.65325077
334
- 324 43.51851852
335
- 325 43.69230769
336
- 326 43.55828221
337
- 327 43.73088685
338
- 328 43.59756098
339
- 329 43.46504559
340
- 330 43.33333333
341
- 331 43.20241692
342
- 332 43.37349398
343
- 333 43.54354354
344
- 334 43.41317365
345
- 335 43.58208955
346
- 336 43.75000000
347
- 337 43.62017804
348
- 338 43.49112426
349
- 339 43.36283186
350
- 340 43.52941176
351
- 341 43.40175953
352
- 342 43.56725146
353
- 343 43.44023324
354
- 344 43.31395349
355
- 345 43.18840580
356
- 346 43.35260116
357
- 347 43.51585014
358
- 348 43.39080460
359
- 349 43.26647564
360
- 350 43.42857143
361
- 351 43.30484330
362
- 352 43.18181818
363
- 353 43.34277620
364
- 354 43.22033898
365
- 355 43.09859155
366
- 356 43.25842697
367
- 357 43.13725490
368
- 358 43.01675978
369
- 359 42.89693593
370
- 360 43.05555556
371
- 361 43.21329640
372
- 362 43.37016575
373
- 363 43.25068871
374
- 364 43.13186813
375
- 365 43.01369863
376
- 366 43.16939891
377
- 367 43.05177112
378
- 368 42.93478261
379
- 369 42.81842818
380
- 370 42.97297297
381
- 371 42.85714286
382
- 372 42.74193548
383
- 373 42.62734584
384
- 374 42.51336898
385
- 375 42.66666667
386
- 376 42.55319149
387
- 377 42.70557029
388
- 378 42.85714286
389
- 379 42.74406332
390
- 380 42.63157895
391
- 381 42.78215223
392
- 382 42.93193717
393
- 383 43.08093995
394
- 384 42.96875000
395
- 385 43.11688312
396
- 386 43.00518135
397
- 387 42.89405685
398
- 388 42.78350515
399
- 389 42.93059126
400
- 390 43.07692308
401
- 391 43.22250639
402
- 392 43.36734694
403
- 393 43.25699746
404
- 394 43.14720812
405
- 395 43.29113924
406
- 396 43.43434343
407
- 397 43.32493703
408
- 398 43.21608040
409
- 399 43.35839599
410
- 400 43.25000000
411
- 401 43.39152120
412
- 402 43.53233831
413
- 403 43.42431762
414
- 404 43.56435644
415
- 405 43.70370370
416
- 406 43.59605911
417
- 407 43.48894349
418
- 408 43.38235294
419
- 409 43.52078240
420
- 410 43.41463415
421
- 411 43.30900243
422
- 412 43.20388350
423
- 413 43.09927361
424
- 414 42.99516908
425
- 415 42.89156627
426
- 416 43.02884615
427
- 417 42.92565947
428
- 418 42.82296651
429
- 419 42.95942721
430
- 420 43.09523810
431
- 421 43.23040380
432
- 422 43.12796209
433
- 423 43.02600473
434
- 424 43.16037736
435
- 425 43.29411765
436
- 426 43.19248826
437
- 427 43.09133489
438
- 428 42.99065421
439
- 429 43.12354312
440
- 430 43.02325581
441
- 431 43.15545244
442
- 432 43.28703704
443
- 433 43.18706697
444
- 434 43.08755760
445
- 435 42.98850575
446
- 436 42.88990826
447
- 437 42.79176201
448
- 438 42.92237443
449
- 439 43.05239180
450
- 440 42.95454545
451
- 441 42.85714286
452
- 442 42.76018100
453
- 443 42.66365688
454
- 444 42.56756757
455
- 445 42.47191011
456
- 446 42.60089686
457
- 447 42.50559284
458
- 448 42.41071429
459
- 449 42.31625835
460
- 450 42.44444444
461
- 451 42.57206208
462
- 452 42.47787611
463
- 453 42.60485651
464
- 454 42.51101322
465
- 455 42.41758242
466
- 456 42.54385965
467
- 457 42.66958425
468
- 458 42.57641921
469
- 459 42.48366013
470
- 460 42.60869565
471
- 461 42.51626898
472
- 462 42.42424242
473
- 463 42.33261339
474
- 464 42.45689655
475
- 465 42.36559140
476
- 466 42.27467811
477
- 467 42.39828694
478
- 468 42.52136752
479
- 469 42.64392324
480
- 470 42.55319149
481
- 471 42.46284501
482
- 472 42.37288136
483
- 473 42.49471459
484
- 474 42.40506329
485
- 475 42.31578947
486
- 476 42.43697479
487
- 477 42.34800839
488
- 478 42.46861925
489
- 479 42.37995825
490
- 480 42.50000000
491
- 481 42.41164241
492
- 482 42.32365145
493
- 483 42.23602484
494
- 484 42.14876033
495
- 485 42.06185567
496
- 486 42.18106996
497
- 487 42.29979466
498
- 488 42.21311475
499
- 489 42.12678937
500
- 490 42.24489796
501
- 491 42.15885947
502
- 492 42.07317073
503
- 493 41.98782961
504
- 494 42.10526316
505
- 495 42.22222222
506
- 496 42.33870968
507
- 497 42.25352113
508
- 498 42.16867470
509
- 499 42.08416834
510
- 500 42.20000000
511
- 501 42.11576846
512
- 502 42.03187251
513
- 503 41.94831014
514
- 504 41.86507937
515
- 505 41.78217822
516
- 506 41.89723320
517
- 507 41.81459566
518
- 508 41.92913386
519
- 509 41.84675835
520
- 510 41.96078431
521
- 511 41.87866928
522
- 512 41.79687500
523
- 513 41.71539961
524
- 514 41.63424125
525
- 515 41.74757282
526
- 516 41.86046512
527
- 517 41.97292070
528
- 518 42.08494208
529
- 519 42.19653179
530
- 520 42.30769231
531
- 521 42.22648752
532
- 522 42.33716475
533
- 523 42.25621415
534
- 524 42.36641221
535
- 525 42.28571429
536
- 526 42.20532319
537
- 527 42.12523719
538
- 528 42.04545455
539
- 529 41.96597353
540
- 530 42.07547170
541
- 531 41.99623352
542
- 532 41.91729323
543
- 533 41.83864916
544
- 534 41.94756554
545
- 535 42.05607477
546
- 536 41.97761194
547
- 537 42.08566108
548
- 538 42.19330855
549
- 539 42.11502783
550
- 540 42.03703704
551
- 541 41.95933457
552
- 542 42.06642066
553
- 543 41.98895028
554
- 544 42.09558824
555
- 545 42.01834862
556
- 546 41.94139194
557
- 547 41.86471664
558
- 548 41.97080292
559
- 549 41.89435337
560
- 550 41.81818182
561
- 551 41.92377495
562
- 552 41.84782609
563
- 553 41.77215190
564
- 554 41.69675090
565
- 555 41.62162162
566
- 556 41.72661871
567
- 557 41.83123878
568
- 558 41.93548387
569
- 559 41.86046512
570
- 560 41.78571429
571
- 561 41.71122995
572
- 562 41.81494662
573
- 563 41.91829485
574
- 564 41.84397163
575
- 565 41.94690265
576
- 566 41.87279152
577
- 567 41.79894180
578
- 568 41.72535211
579
- 569 41.65202109
580
- 570 41.57894737
581
- 571 41.50612960
582
- 572 41.60839161
583
- 573 41.53577661
584
- 574 41.46341463
585
- 575 41.39130435
586
- 576 41.49305556
587
- 577 41.59445407
588
- 578 41.69550173
589
- 579 41.62348877
590
- 580 41.72413793
591
- 581 41.82444062
592
- 582 41.75257732
593
- 583 41.85248714
594
- 584 41.95205479
595
- 585 41.88034188
596
- 586 41.97952218
597
- 587 41.90800681
598
- 588 41.83673469
599
- 589 41.76570458
600
- 590 41.69491525
601
- 591 41.62436548
602
- 592 41.55405405
603
- 593 41.65261383
604
- 594 41.75084175
605
- 595 41.68067227
606
- 596 41.77852349
607
- 597 41.70854271
608
- 598 41.63879599
609
- 599 41.73622705
610
- 600 41.66666667
611
- 601 41.59733777
612
- 602 41.52823920
613
- 603 41.62520730
614
- 604 41.55629139
615
- 605 41.48760331
616
- 606 41.41914191
617
- 607 41.51565074
618
- 608 41.44736842
619
- 609 41.37931034
620
- 610 41.31147541
621
- 611 41.40752864
622
- 612 41.33986928
623
- 613 41.43556281
624
- 614 41.53094463
625
- 615 41.46341463
626
- 616 41.55844156
627
- 617 41.65316045
628
- 618 41.58576052
629
- 619 41.51857835
630
- 620 41.45161290
631
- 621 41.38486312
632
- 622 41.31832797
633
- 623 41.25200642
634
- 624 41.18589744
635
- 625 41.12000000
636
- 626 41.21405751
637
- 627 41.14832536
638
- 628 41.08280255
639
- 629 41.17647059
640
- 630 41.26984127
641
- 631 41.36291601
642
- 632 41.45569620
643
- 633 41.39020537
644
- 634 41.32492114
645
- 635 41.25984252
646
- 636 41.19496855
647
- 637 41.13029827
648
- 638 41.06583072
649
- 639 41.15805947
650
- 640 41.25000000
651
- 641 41.18564743
652
- 642 41.12149533
653
- 643 41.05754277
654
- 644 41.14906832
655
- 645 41.08527132
656
- 646 41.02167183
657
- 647 40.95826893
658
- 648 40.89506173
659
- 649 40.83204931
660
- 650 40.76923077
661
- 651 40.70660522
662
- 652 40.64417178
663
- 653 40.58192956
664
- 654 40.67278287
665
- 655 40.61068702
666
- 656 40.70121951
667
- 657 40.63926941
668
- 658 40.72948328
669
- 659 40.66767830
670
- 660 40.60606061
671
- 661 40.54462935
672
- 662 40.48338369
673
- 663 40.42232278
674
- 664 40.51204819
675
- 665 40.60150376
676
- 666 40.54054054
677
- 667 40.62968516
678
- 668 40.71856287
679
- 669 40.80717489
680
- 670 40.89552239
681
- 671 40.83457526
682
- 672 40.77380952
683
- 673 40.71322437
684
- 674 40.65281899
685
- 675 40.59259259
686
- 676 40.53254438
687
- 677 40.47267356
688
- 678 40.41297935
689
- 679 40.35346097
690
- 680 40.44117647
691
- 681 40.52863436
692
- 682 40.46920821
693
- 683 40.40995608
694
- 684 40.49707602
695
- 685 40.43795620
696
- 686 40.37900875
697
- 687 40.46579330
698
- 688 40.40697674
699
- 689 40.34833091
700
- 690 40.28985507
701
- 691 40.37626628
702
- 692 40.31791908
703
- 693 40.25974026
704
- 694 40.34582133
705
- 695 40.28776978
706
- 696 40.37356322
707
- 697 40.31563845
708
- 698 40.25787966
709
- 699 40.34334764
710
- 700 40.28571429
711
- 701 40.37089872
712
- 702 40.45584046
713
- 703 40.54054054
714
- 704 40.62500000
715
- 705 40.56737589
716
- 706 40.50991501
717
- 707 40.45261669
718
- 708 40.39548023
719
- 709 40.47954866
720
- 710 40.42253521
721
- 711 40.50632911
722
- 712 40.58988764
723
- 713 40.67321178
724
- 714 40.75630252
725
- 715 40.69930070
726
- 716 40.64245810
727
- 717 40.58577406
728
- 718 40.52924791
729
- 719 40.47287900
730
- 720 40.41666667
731
- 721 40.36061026
732
- 722 40.30470914
733
- 723 40.38727524
734
- 724 40.33149171
735
- 725 40.41379310
736
- 726 40.35812672
737
- 727 40.30261348
738
- 728 40.38461538
739
- 729 40.32921811
740
- 730 40.27397260
741
- 731 40.21887825
742
- 732 40.16393443
743
- 733 40.24556617
744
- 734 40.32697548
745
- 735 40.40816327
746
- 736 40.35326087
747
- 737 40.43419267
748
- 738 40.51490515
749
- 739 40.59539919
750
- 740 40.54054054
751
- 741 40.48582996
752
- 742 40.56603774
753
- 743 40.64602961
754
- 744 40.72580645
755
- 745 40.67114094
756
- 746 40.61662198
757
- 747 40.56224900
758
- 748 40.50802139
759
- 749 40.45393858
760
- 750 40.53333333
761
-
762
- Final result: 40.5333 +/- 1.7939
763
- Random chance: 25.0000 +/- 1.5822