Update README.md

README.md

@@ -37,4 +37,54 @@ exl3_3.07bpw-h6-custom
  -- Perplexity: 3.935338
 ```
 
-TODO: bf16 baseline as evaluated by the same exllamav3 pipeline
+Additional metrics via `eval/model_diff.py` courtesy of [turboderp](https://huggingface.co/turboderp):
+
+"Plain" exl3_3.0bpw-h6 vs original bf16 weights
+```
+ -- original perplexity:  1.76745635
+ -- original label in top-K:
+      K = 1: 0.8681
+      K = 2: 0.9237
+      K = 3: 0.9411
+      K = 4: 0.9502
+      K = 5: 0.9564
+ -- 3.0bpw-h6 perplexity:  2.14967564
+ -- 3.0bpw-h6 label in top-K:
+      K = 1: 0.8142
+      K = 2: 0.8949
+      K = 3: 0.9231
+      K = 4: 0.9368
+      K = 5: 0.9464
+ -- Top-K agreement, 3.0bpw-h6 vs original:
+      K = 1: 0.8820
+      K = 2: 0.5225
+      K = 3: 0.2585
+      K = 4: 0.1132
+      K = 5: 0.0491
+ -- KL divergence (3.0bpw-h6, original):  0.23334818
+```
+
+exl3_3.07bpw-h6-custom vs original bf16 weights
+```
+ -- original perplexity:  1.76745635
+ -- original label in top-K:
+      K = 1: 0.8681
+      K = 2: 0.9237
+      K = 3: 0.9411
+      K = 4: 0.9502
+      K = 5: 0.9564
+ -- 3.07bpw-h6-custom perplexity:  2.03357968
+ -- 3.07bpw-h6-custom label in top-K:
+      K = 1: 0.8305
+      K = 2: 0.9021
+      K = 3: 0.9286
+      K = 4: 0.9416
+      K = 5: 0.9504
+ -- Top-K agreement, 3.07bpw-h6-custom vs original:
+      K = 1: 0.8981
+      K = 2: 0.5702
+      K = 3: 0.3027
+      K = 4: 0.1461
+      K = 5: 0.0691
+ -- KL divergence (3.07bpw-h6-custom, original):  0.17770892
+```