main README: lineage index
Browse files
README.md
CHANGED
|
@@ -9,81 +9,47 @@ tags:
|
|
| 9 |
- code
|
| 10 |
---
|
| 11 |
|
| 12 |
-
# dspark-glm52-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
-
|
| 25 |
-
|
| 26 |
-
-
|
| 27 |
-
-
|
| 28 |
-
|
| 29 |
-
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
set **overlaps the training view** (no holdout exists in the immutable view), it is
|
| 57 |
-
teacher-forced, and this checkpoint is in **epoch 3** over that same view — so part of the
|
| 58 |
-
late-curve gain is second/third-pass fit rather than generalization. Per this program's
|
| 59 |
-
methodology, keep/kill belongs to the offline 3-task screen (gsm8k + humaneval + mt-bench,
|
| 60 |
-
×32 samples, temp 1.0) and to an sglang serving bench; neither has been run on this
|
| 61 |
-
checkpoint. Trend-eval noise is ±0.03–0.04, so single-eval deltas below ~0.05 are not real.
|
| 62 |
-
|
| 63 |
-
Curve (this run, step → `official_accept_len`; step 0 == the 4.594 restart point):
|
| 64 |
-
|
| 65 |
-
| 50 | 500 | 1500 | 3000 | 4500 | 6000 | 7000 | 7500 | **7650** |
|
| 66 |
-
|---|---|---|---|---|---|---|---|---|
|
| 67 |
-
| 4.566 | 4.082 | 4.207 | 4.353 | 4.479 | 4.595 | 4.653 | 4.676 | **4.704** |
|
| 68 |
-
|
| 69 |
-
The dip to ~4.03 around steps 250–500 is the expected high-LR warm-restart excursion
|
| 70 |
-
(fresh optimizer state entering a 2.3e-3 peak); it recovered and passed the restart point
|
| 71 |
-
by step ~6300. `official_accept_len` is the published formula (per-block cumprod of
|
| 72 |
-
`alpha_k = Σ min(draft_p, target_p)`, then averaged), verified against the speculators
|
| 73 |
-
library's own `dspark/metrics.py` — the same shape as a reference-harness figure, though
|
| 74 |
-
comparison to any published number still requires matched block size, temperature, and target.
|
| 75 |
-
|
| 76 |
-
## Provenance
|
| 77 |
-
|
| 78 |
-
- Checkpoint: `glm52-dspark-moe8-draft-opd/sft_lr7x_ct/checkpoints/iter_0007651` (FSDP
|
| 79 |
-
full-state, best-pinned by `eval/official_accept_len`), converted via
|
| 80 |
-
`TorchSpec/tools/convert_to_hf.py` (native sglang keys, bf16, embed_tokens injected from
|
| 81 |
-
the NVFP4 target).
|
| 82 |
-
- Pipeline identity `7b901cf00f421aff…`; draft config
|
| 83 |
-
`dspark_draft_config_glm_p7_bs8_moe8_plf.json`,
|
| 84 |
-
sha256 `58f164050a6d70039ef333e9e866c2117380207b7eff302272bd39887f39d6f8` (matches the
|
| 85 |
-
run's recorded `pipeline_draft_config_fingerprint`).
|
| 86 |
-
- Tokenizer: use GLM-5.2's (not bundled).
|
| 87 |
-
|
| 88 |
-
Later checkpoints of this run will supersede this snapshot. SFT is followed by a
|
| 89 |
-
verifier-replay Draft-OPD stage.
|
|
|
|
| 9 |
- code
|
| 10 |
---
|
| 11 |
|
| 12 |
+
# dspark-glm52-draft
|
| 13 |
+
|
| 14 |
+
DSpark (DFlash-family single-shot parallel drafting) **draft models for GLM-5.2
|
| 15 |
+
speculative decoding**, optimized for code / coding-chat traffic. This repo consolidates
|
| 16 |
+
the lineage; each historical checkpoint is an immutable **git tag** (pick it from the
|
| 17 |
+
revision dropdown or pass `revision=` to `from_pretrained`/`hf_hub_download`).
|
| 18 |
+
|
| 19 |
+
`main` tracks the current best checkpoint.
|
| 20 |
+
|
| 21 |
+
## Revisions
|
| 22 |
+
|
| 23 |
+
| tag | arch | checkpoint | trend `official_accept_len` | note |
|
| 24 |
+
|---|---|---|---|---|
|
| 25 |
+
| **`bs8-moe8-lr7xct-20260722-iter7651`** (= `main`) | block-8 MoE-8 + DFlare | lr7x-ct iter_7651 | **4.704 / 8** | current best; high-LR warm-restart continuation |
|
| 26 |
+
| `bs8-moe8-dflare-20260715-iter10001` | block-8 MoE-8 + DFlare | dflare iter_10001 | 4.59 / 8 | predecessor run's pre-bump champion |
|
| 27 |
+
| `bs8-moe8-dflare-20260715-iter5251` | block-8 MoE-8 + DFlare | dflare iter_5251 | ~4.49 | earlier dflare snapshot |
|
| 28 |
+
| `bs8-moe8-dflare-20260714` | block-8 MoE-8 + DFlare | dflare (earlier) | — | earliest block-8 export |
|
| 29 |
+
| `bs16-moe-20260705` | **block-16** MoE | 2026-07-05 | — | distinct architecture (γ=16), research arm — NOT a version of the block-8 line |
|
| 30 |
+
|
| 31 |
+
**All trend numbers are teacher-forced metrics on a fixed 96-row set that overlaps the
|
| 32 |
+
training view (no holdout) — NOT serving measurements and NOT SOTA claims.** Keep/kill
|
| 33 |
+
belongs to the offline 3-task screen (gsm8k + humaneval + mt-bench) and an sglang serving
|
| 34 |
+
bench. Trend-eval noise is ±0.03–0.04.
|
| 35 |
+
|
| 36 |
+
## Shared architecture (block-8 line)
|
| 37 |
+
|
| 38 |
+
- DSpark block size 8, 5 draft layers, hidden 6144, 64 heads (head_dim 64).
|
| 39 |
+
- Vanilla MoE-8 (8 routed top-2 + 1 shared per layer, moe_intermediate 3072); Jakiro OFF.
|
| 40 |
+
- DFlare per-layer context fusion, Markov/GRIFFIN correction head (rank 256) + confidence
|
| 41 |
+
head (`confidence_detach_input`, `griffin_fuse`).
|
| 42 |
+
- Reads GLM-5.2 target hidden states from layers [8, 23, 39, 55, 70] of 78.
|
| 43 |
+
- 4.44 B total params (~2.7 B active/token), bf16, 8.3 GB.
|
| 44 |
+
- `draft_vocab_size` 32000 vs target 154880 → serve via sglang's DSpark loader with the
|
| 45 |
+
**d2t full-vocab scatter** (`d2t` int64 [32000], `t2d` bool [154880] ship in the
|
| 46 |
+
safetensors; do NOT bf16-cast them). Tokenizer: use GLM-5.2's (not bundled).
|
| 47 |
+
|
| 48 |
+
## Target & data (block-8 line)
|
| 49 |
+
|
| 50 |
+
- Target (generation AND extraction): **lukealonso/GLM-5.2-NVFP4** (modelopt_fp4, fp8-e4m3
|
| 51 |
+
KV), live target prefill — fully on-policy, no disk target-state cache.
|
| 52 |
+
- SFT view `sha256:c17516837c…` (1,379,922 production DB conversations + 522,176 on-policy
|
| 53 |
+
OPB rows), seq 4096, GLM chat template.
|
| 54 |
+
|
| 55 |
+
See each tag's own README for its exact step, LR schedule, and provenance hashes.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|