cudaoom commited on
Commit
0c7e53f
·
verified ·
1 Parent(s): 24362ac

main README: lineage index

Browse files
Files changed (1) hide show
  1. README.md +44 -78
README.md CHANGED
@@ -9,81 +9,47 @@ tags:
9
  - code
10
  ---
11
 
12
- # dspark-glm52-bs8-moe8-lr7xct-20260722
13
-
14
- **Mid-training SFT checkpoint** (step 7651 / 10365, epoch 3 of 3) of the DSpark draft model
15
- for **GLM-5.2** speculative decoding, optimized for code / coding-chat traffic.
16
-
17
- Supersedes `dspark-glm52-bs8-moe8-dflare-20260714/20260715` (same architecture; this is the
18
- high-LR warm-restart continuation, `official_accept_len` 4.59 → **4.70** on the same trend set).
19
-
20
- ## Architecture
21
-
22
- - **DSpark** (DFlash-family single-shot parallel drafting): block size **8**, 5 draft layers,
23
- hidden 6144, 64 heads (head_dim 64).
24
- - **Vanilla MoE-8**: 8 routed experts (top-2, norm_topk) + 1 shared expert per layer,
25
- moe_intermediate 3072. Jakiro position-MoE **OFF**.
26
- - **DFlare per-layer context fusion ON** (zero-init gates).
27
- - Markov/GRIFFIN logit-correction head (rank 256, vanilla) + confidence head
28
- (`confidence_detach_input`), `griffin_fuse` on.
29
- - Reads target hidden states from GLM-5.2 layers **[8, 23, 39, 55, 70]** (of 78).
30
- - **4.44 B total params** (2.55 B experts, 0.95 B frozen GLM embed_tokens, 0.74 B trunk/heads,
31
- 0.20 B draft lm_head) ≈ **2.7 B active/token**. bf16, 8.3 GB.
32
- - `draft_vocab_size` **32000** vs target vocab 154880: serving through sglang's DSpark loader
33
- requires the **d2t full-vocab scatter conversion** (the loader argmaxes the draft lm_head
34
- directly as target ids); TorchSpec→sglang key renames apply. `d2t` (int64 [32000]) and
35
- `t2d` (bool [154880]) ship in the safetensors and must NOT be cast to bf16.
36
-
37
- ## Training
38
-
39
- - Target (generation AND extraction): **lukealonso/GLM-5.2-NVFP4** (modelopt_fp4, fp8-e4m3 KV),
40
- served live for target prefill — no disk target-state cache, fully on-policy.
41
- - Data: immutable SFT view `sha256:c17516837cd02eecbcd937a20263741343740b05c2775efcd760bd7fbf4e7afc`
42
- (1,379,922 production DB conversations + 522,176 on-policy OPB rows), seq 4096, GLM chat template.
43
- - Objective: decay-weighted CE (0.1) + L1 (0.9) on target states, confidence BCE,
44
- VSD 0.3 (ramp 30@40), MoE aux/z-loss 1e-3, label convention `infill`, 256 anchors.
45
- - Optimizer muon, **peak lr 2.3e-3** cosine → min 6e-6 over a 3-epoch horizon,
46
- warmup_ratio 0.02, seed 43, global batch 512 (micro 8 × accum 8 × dp 8), FlashAttention-4.
47
- - **Weights-only warm restart** from `sft/loadroot_iter9951` (the pre-bump champion of the
48
- predecessor dflare run, `official_accept_len` 4.594), fresh optimizer + fresh schedule.
49
- - W&B: `morphllm/torchspec-p7autoresearch/ts-sft-lr7xct-9951`.
50
-
51
- ## Eval (caveat first)
52
-
53
- `official_accept_len` **4.7040 of 8** at the step-7650 eval on the fixed 96-row trend set.
54
-
55
- **This is a trend metric, not a serving measurement, and it is not a SOTA claim.** The trend
56
- set **overlaps the training view** (no holdout exists in the immutable view), it is
57
- teacher-forced, and this checkpoint is in **epoch 3** over that same view — so part of the
58
- late-curve gain is second/third-pass fit rather than generalization. Per this program's
59
- methodology, keep/kill belongs to the offline 3-task screen (gsm8k + humaneval + mt-bench,
60
- ×32 samples, temp 1.0) and to an sglang serving bench; neither has been run on this
61
- checkpoint. Trend-eval noise is ±0.03–0.04, so single-eval deltas below ~0.05 are not real.
62
-
63
- Curve (this run, step → `official_accept_len`; step 0 == the 4.594 restart point):
64
-
65
- | 50 | 500 | 1500 | 3000 | 4500 | 6000 | 7000 | 7500 | **7650** |
66
- |---|---|---|---|---|---|---|---|---|
67
- | 4.566 | 4.082 | 4.207 | 4.353 | 4.479 | 4.595 | 4.653 | 4.676 | **4.704** |
68
-
69
- The dip to ~4.03 around steps 250–500 is the expected high-LR warm-restart excursion
70
- (fresh optimizer state entering a 2.3e-3 peak); it recovered and passed the restart point
71
- by step ~6300. `official_accept_len` is the published formula (per-block cumprod of
72
- `alpha_k = Σ min(draft_p, target_p)`, then averaged), verified against the speculators
73
- library's own `dspark/metrics.py` — the same shape as a reference-harness figure, though
74
- comparison to any published number still requires matched block size, temperature, and target.
75
-
76
- ## Provenance
77
-
78
- - Checkpoint: `glm52-dspark-moe8-draft-opd/sft_lr7x_ct/checkpoints/iter_0007651` (FSDP
79
- full-state, best-pinned by `eval/official_accept_len`), converted via
80
- `TorchSpec/tools/convert_to_hf.py` (native sglang keys, bf16, embed_tokens injected from
81
- the NVFP4 target).
82
- - Pipeline identity `7b901cf00f421aff…`; draft config
83
- `dspark_draft_config_glm_p7_bs8_moe8_plf.json`,
84
- sha256 `58f164050a6d70039ef333e9e866c2117380207b7eff302272bd39887f39d6f8` (matches the
85
- run's recorded `pipeline_draft_config_fingerprint`).
86
- - Tokenizer: use GLM-5.2's (not bundled).
87
-
88
- Later checkpoints of this run will supersede this snapshot. SFT is followed by a
89
- verifier-replay Draft-OPD stage.
 
9
  - code
10
  ---
11
 
12
+ # dspark-glm52-draft
13
+
14
+ DSpark (DFlash-family single-shot parallel drafting) **draft models for GLM-5.2
15
+ speculative decoding**, optimized for code / coding-chat traffic. This repo consolidates
16
+ the lineage; each historical checkpoint is an immutable **git tag** (pick it from the
17
+ revision dropdown or pass `revision=` to `from_pretrained`/`hf_hub_download`).
18
+
19
+ `main` tracks the current best checkpoint.
20
+
21
+ ## Revisions
22
+
23
+ | tag | arch | checkpoint | trend `official_accept_len` | note |
24
+ |---|---|---|---|---|
25
+ | **`bs8-moe8-lr7xct-20260722-iter7651`** (= `main`) | block-8 MoE-8 + DFlare | lr7x-ct iter_7651 | **4.704 / 8** | current best; high-LR warm-restart continuation |
26
+ | `bs8-moe8-dflare-20260715-iter10001` | block-8 MoE-8 + DFlare | dflare iter_10001 | 4.59 / 8 | predecessor run's pre-bump champion |
27
+ | `bs8-moe8-dflare-20260715-iter5251` | block-8 MoE-8 + DFlare | dflare iter_5251 | ~4.49 | earlier dflare snapshot |
28
+ | `bs8-moe8-dflare-20260714` | block-8 MoE-8 + DFlare | dflare (earlier) | — | earliest block-8 export |
29
+ | `bs16-moe-20260705` | **block-16** MoE | 2026-07-05 | — | distinct architecture (γ=16), research arm — NOT a version of the block-8 line |
30
+
31
+ **All trend numbers are teacher-forced metrics on a fixed 96-row set that overlaps the
32
+ training view (no holdout) — NOT serving measurements and NOT SOTA claims.** Keep/kill
33
+ belongs to the offline 3-task screen (gsm8k + humaneval + mt-bench) and an sglang serving
34
+ bench. Trend-eval noise is ±0.03–0.04.
35
+
36
+ ## Shared architecture (block-8 line)
37
+
38
+ - DSpark block size 8, 5 draft layers, hidden 6144, 64 heads (head_dim 64).
39
+ - Vanilla MoE-8 (8 routed top-2 + 1 shared per layer, moe_intermediate 3072); Jakiro OFF.
40
+ - DFlare per-layer context fusion, Markov/GRIFFIN correction head (rank 256) + confidence
41
+ head (`confidence_detach_input`, `griffin_fuse`).
42
+ - Reads GLM-5.2 target hidden states from layers [8, 23, 39, 55, 70] of 78.
43
+ - 4.44 B total params (~2.7 B active/token), bf16, 8.3 GB.
44
+ - `draft_vocab_size` 32000 vs target 154880 → serve via sglang's DSpark loader with the
45
+ **d2t full-vocab scatter** (`d2t` int64 [32000], `t2d` bool [154880] ship in the
46
+ safetensors; do NOT bf16-cast them). Tokenizer: use GLM-5.2's (not bundled).
47
+
48
+ ## Target & data (block-8 line)
49
+
50
+ - Target (generation AND extraction): **lukealonso/GLM-5.2-NVFP4** (modelopt_fp4, fp8-e4m3
51
+ KV), live target prefill — fully on-policy, no disk target-state cache.
52
+ - SFT view `sha256:c17516837c…` (1,379,922 production DB conversations + 522,176 on-policy
53
+ OPB rows), seq 4096, GLM chat template.
54
+
55
+ See each tag's own README for its exact step, LR schedule, and provenance hashes.