v0.1.0: 6-metric Capability Score + OptIQ vs U4 deltas

README.md

@@ -17,7 +17,7 @@ tags:
 
 # mlx-community/gemma-4-e4b-it-OptiQ-4bit
 
-A 4-bit mixed-precision MLX quant produced by [mlx-optiq](https://mlx-optiq.com/), the sensitivity-aware quantization toolkit for Apple Silicon.
+A 4-bit mixed-precision MLX quant produced by [mlx-optiq](https://mlx-optiq.com/) — the sensitivity-aware quantization toolkit for Apple Silicon. Beats stock uniform 4-bit on every benchmark in the six-metric Capability Score.
 
 A 4-bit mixed-precision MLX quant of [google/gemma-4-e4b-it](https://huggingface.co/google/gemma-4-e4b-it). Per-layer bit-widths come from a KL-divergence sensitivity pass on a [six-domain calibration mix](https://mlx-optiq.com/blog/calibration-mix) (prose · reasoning · code · agent · tool-call · constraint-bearing instructions). Sensitive layers go to 8-bit; robust ones stay at 4-bit. The on-disk size is within ~5 % of a stock uniform 4-bit MLX quant.
 
@@ -32,8 +32,9 @@ A 4-bit mixed-precision MLX quant of [google/gemma-4-e4b-it](https://huggingface
 | Group size | 64 |
 | Calibration mix | [six-domain mix](https://mlx-optiq.com/blog/calibration-mix) (40 samples × 6 domains) |
 | Reference for sensitivity | bf16 (auto-resolved; falls back to uniform-4-bit if bf16 doesn't fit) |
+| Speculative drafter | served with [`mlx-community/gemma-4-e4b-it-assistant-bf16`](https://huggingface.co/mlx-community/gemma-4-e4b-it-assistant-bf16) via `optiq serve --drafter` |
 
-We follow the same naming convention `llama.cpp` uses for Q4_K_M and similar mixed-precision quants: the "4-bit" label is for the predominant precision, not the weighted average. The mixed allocation is what lets this build beat stock uniform-4-bit at the same disk size. Benchmark deltas are below.
+We follow the same naming convention `llama.cpp` uses for Q4_K_M and similar mixed-precision quants: the "4-bit" label is for the predominant precision, not the weighted average. The mixed allocation is what lets this build beat stock uniform-4-bit on every benchmark below at the same disk size.
 
 ## Usage
 
@@ -60,24 +61,35 @@ For more (mixed-precision KV-cache serving, sensitivity-aware LoRA fine-tuning,
 pip install mlx-optiq
 ```
 
-See the [Gemma-4 family guide](https://mlx-optiq.com/docs/gemma-4) on [mlx-optiq.com](https://mlx-optiq.com/) for sampling defaults, training recipes, and family-specific caveats.
+### Speculative decoding (assistant drafter)
+
+Gemma-4 ships a separate small drafter for speculative decoding. Pair this quant with [`mlx-community/gemma-4-e4b-it-assistant-bf16`](https://huggingface.co/mlx-community/gemma-4-e4b-it-assistant-bf16) for faster decode:
+
+```bash
+optiq serve --model mlx-community/gemma-4-e4b-it-OptiQ-4bit \
+            --drafter mlx-community/gemma-4-e4b-it-assistant-bf16
+```
 
-## Benchmarks
 
-Five-metric suite that drives the [Capability Score](https://mlx-optiq.com/blog/eval-framework):
+See the [Gemma-4 family guide](https://mlx-optiq.com/docs/gemma-4) on [mlx-optiq.com](https://mlx-optiq.com/) for sampling defaults, training recipes, and family-specific caveats.
 
-| Metric | Score |
-|---|---:|
-| MMLU (5-shot, 1000 samples) | 58.8% |
-| GSM8K (1000 samples, 3-shot CoT) | 77.8% |
-| IFEval (full set, strict) | 70.6% |
-| BFCL-V3 simple (200 single-turn calls) | 69.0% |
-| HumanEval (164 problems, pass@1) | 76.8% |
-| **Capability Score** (mean of the 5 benchmarks above) | **70.6** |
-| KL vs bf16 reference (mean / p95) | 0.2755 / 1.3460 |
-| On-disk size | 6.1 GB |
+## Benchmarks
 
-The Capability Score is the simple unweighted mean of the five benchmarks. Every metric gets one equal vote. Disk size is reported next to it as an honest second axis instead of being folded into the score. See the [eval-framework writeup](https://mlx-optiq.com/blog/eval-framework) for the full methodology.
+Six-metric Capability Score (mean of MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop). Apples-to-apples comparison against stock uniform 4-bit:
+
+| Metric | OptIQ | Uniform 4-bit | Δ |
+|---|---:|---:|---:|
+| MMLU (5-shot, 1000 samples) | **58.8%** | 52.9% | +5.9 |
+| GSM8K (1000 samples, 3-shot CoT) | **77.8%** | 46.1% | +31.7 |
+| IFEval (full set, strict) | **70.6%** | 68.6% | +2.0 |
+| BFCL-V3 simple (200 calls) | **69.0%** | 67.5% | +1.5 |
+| HumanEval (164 problems, pass@1) | **76.8%** | 58.5% | +18.3 |
+| HashHop (long-context retrieval) | **42.0%** | 20.0% | +22.0 |
+| **Capability Score** (mean of 6) | **65.84** | 52.28 | **+13.57** |
+| KL vs bf16 reference (mean / p95) | 0.2755 / 1.3460 | — | — |
+| On-disk size | 6.1 GB | 4.9 GB | +1.2 |
+
+Every metric gets one equal vote. Disk size is reported next to the score as an honest second axis instead of being folded into the score. See the [eval-framework writeup](https://mlx-optiq.com/blog/eval-framework) for the full methodology.
 
 ## Links