Zenyx-100M-Base
A 151M-parameter dense decoder-only language model trained from scratch on web, code and math text (the "100M" in the name is the original plan; the trained model is 151M).
Status: pre-training in progress (9.36B of a 50B-token budget at step 17,860; newer checkpoints may be in the repository). This is a base model: not instruction-tuned, no safety tuning, and it makes up facts. All numbers below are from mid-training checkpoints (the learning rate has not finished decaying). A post-training stage is planned (below).
At a glance
| Parameters | 151.0M (152.2M with the experimental segment memory, checkpoints from step 17,900) |
| Architecture | 18-layer Llama-style decoder, grouped-query attention, SwiGLU, context 2,048; since step 17,860 an experimental segment memory in two blocks (see Segment memory) |
| Vocabulary | 49,105 tokens (byte-level BPE) |
| Training | 2 × Tesla T4, fp16; Muon + AdamW; 524,288 tokens per step |
| Progress | step 17,860 = 9.36B tokens of 50B; learning rate at 56% of peak, decaying until 15B tokens |
| Latest benchmarks | step 17,860 (core suite, below); step 12,650 (wider suite, MMLU); step 10,900 (code, speed, letter-choice); step 10,800 (wide arithmetic) |
| License | Apache-2.0 |
Results
Base model, no fine-tuning, 0-shot, all test items. Fact1000 is a 1,000-item four-way factual test written by the author (chance 25%); SciQ prompts include the supporting passage. Standard error is about 1.4 points on 1,000 items.
Comparison at the latest checkpoints
| Zenyx-100M (step 17,860) | BetterGPT-150M | Muse2-230M* | Muse2-125M* | Zenyx-V3 (MoE) | |
|---|---|---|---|---|---|
| Parameters | 151.0M | 152.6M | 196.1M | 122.9M | 1.58B (0.38B active) |
| Training tokens | 9.4B | about 15B | not published | not published | about 80B |
| Fact1000 | 73.0% | 76.7% | 56.3% | 51.9% | 79.8% |
| Fact1000, without the arithmetic items | 70.0% | 78.3% | 61.7% | 57.1% | 79.2% |
| SciQ (acc / length-normalised) | 78.9 / 71.9 | 80.7 / 75.6 | 68.3 / 61.7 | 67.3 / 58.7 | 79.4 / 70.6 |
| ARC-Easy | 47.2 / 42.6 | 54.0 / 48.5 | 44.3 / 39.8 | 44.3 / 39.8 | 52.0 / 46.6 |
| ARC-Challenge | 20.6 / 22.4 | 25.5 / 27.2 | 18.9 / 24.1 | 17.3 / 22.6 | 21.6 / 26.1 |
| Pile bits per byte, 22 domains (lower is better) | 1.079 | 1.066 | 1.193 | 1.227 | 0.999 |
| Pile, mean of domains | 1.235 | 1.232 | 1.387 | 1.424 | 1.159 |
| Arithmetic, 14,000 problems, best prompt format | 97.3% (own chat format) | 9.7% (3-shot) | not run | not run | 16.0% (3-shot) |
* Muse2: our own reimplementation of the architecture (no public code), same harness and items, not token-matched. V3 is step 110,000 (arithmetic: step 110,800). BetterGPT is its released checkpoint with its tokenizer's BOS+EOS prefix (the choice moves its scores by about 1 point; the author-default BOS prefix gives Fact1000 75.7%).
- BetterGPT-150M leads on knowledge and language modelling by roughly 2 to 8 points on the rows above, with about twice the training tokens (15B against 9.4B); Zenyx-100M is ahead of both Muse2 models on Fact1000, SciQ, ARC-Easy and the Pile and level on ARC-Challenge (ARC-Challenge is close to the 25% chance level for every model in the table).
- Zenyx-100M leads on arithmetic by a wide margin: BetterGPT and V3 do 2-digit work and little more even with worked examples in the prompt, while Zenyx-100M was trained on arithmetic in its own chat format with a short scratchpad (see Arithmetic).
- Zenyx-V3 (about 10 times the parameters, 9 times the tokens) is ahead of everything else on knowledge and the Pile.
Wider suite (step 12,650)
| Zenyx-100M | BetterGPT-150M | Zenyx-V3 (step 107,200) | |
|---|---|---|---|
| HellaSwag (acc / norm) | 28.7 / 30.5 | 31.5 / 36.6 | 30.7 / 34.2 |
| PIQA | 59.9 / 58.4 | 65.2 / 65.5 | 61.4 / 60.7 |
| WinoGrande | 52.8 | 53.0 | 51.2 |
| OpenBookQA | 16.2 / 24.2 | 20.4 / 31.4 | 19.2 / 28.4 |
| BoolQ | 62.0 / 62.2 | 58.4 / 62.0 | 61.5 / 62.2 |
| COPA (100 items) | 57.0 | 63.0 | 55.0 |
| LAMBADA accuracy | 33.8 | 27.6 | 28.5 |
| WikiText bits per byte | 1.126 | 1.082 | 1.011 |
Most of these are close to chance for models of this size (HellaSwag 25%, PIQA/WinoGrande/COPA 50%); BetterGPT is ahead on all but LAMBADA and BoolQ (WinoGrande is a tie). An independent lm-evaluation-harness scoring agrees with ours within about 1 point for both small models.
Pile bits per byte by domain
Lower is better; bold = best of the four. Zenyx-100M step 17,860; Muse2-230M* and V3 as above. Bits per byte does not depend on the tokenizer.
| Domain | Zenyx-100M | BetterGPT-150M | Muse2-230M* | Zenyx-V3 |
|---|---|---|---|---|
| DM Mathematics | 1.297 | 1.348 | 1.723 | 1.326 |
| GitHub | 0.624 | 0.722 | 0.846 | 0.576 |
| EuroParl | 2.016 | 2.155 | 2.444 | 1.964 |
| Enron Emails | 1.283 | 1.249 | 1.416 | 1.222 |
| YouTube Subtitles | 1.880 | 2.011 | 2.290 | 1.816 |
| Wikipedia (en) | 1.052 | 1.080 | 1.123 | 0.987 |
| PhilPapers | 1.401 | 1.415 | 1.667 | 1.332 |
| HackerNews | 1.352 | 1.305 | 1.448 | 1.278 |
| StackExchange | 0.993 | 1.025 | 1.207 | 0.917 |
| ArXiv | 0.933 | 0.911 | 1.131 | 0.855 |
| PubMed Central | 0.820 | 0.797 | 0.922 | 0.741 |
| Pile-CC | 1.179 | 1.124 | 1.199 | 1.098 |
| BookCorpus2 | 1.227 | 1.155 | 1.205 | 1.142 |
| USPTO Backgrounds | 0.866 | 0.828 | 0.938 | 0.779 |
| OpenWebText2 | 1.215 | 1.188 | 1.281 | 1.126 |
| Ubuntu IRC | 1.807 | 1.852 | 2.121 | 1.714 |
| OpenSubtitles | 1.427 | 1.358 | 1.399 | 1.333 |
| Gutenberg (PG-19) | 1.364 | 1.298 | 1.430 | 1.268 |
| NIH ExPorter | 0.957 | 0.906 | 1.018 | 0.861 |
| Books3 | 1.427 | 1.398 | 1.495 | 1.330 |
| PubMed Abstracts | 0.976 | 0.932 | 1.072 | 0.873 |
| FreeLaw | 1.074 | 1.040 | 1.129 | 0.958 |
| 22 domains, byte-weighted | 1.079 | 1.066 | 1.193 | 0.999 |
| Mean of domains | 1.235 | 1.232 | 1.387 | 1.159 |
Zenyx-100M beats BetterGPT in 8 of 22 domains (GitHub, YouTube Subtitles, EuroParl, DM Mathematics, StackExchange, Wikipedia (en), Ubuntu IRC, PhilPapers) and is 2 to 6% behind in the other 14 (overall 1.2% behind). It is ahead of Muse2-230M* in 20 of 22 domains.
Over training
| Step | Tokens (B) | Fact1000 (no arithmetic) | SciQ acc / norm | ARC-Easy acc / norm | Pile bits/byte | Arithmetic (14,000) |
|---|---|---|---|---|---|---|
| 1,810 | 0.95 | 50.6% (53.6%) | not run | not run | not run | not run |
| 6,000 | 3.15 | 58.5% (61.7%) | 72.4 / 67.3 | 44.4 / 40.1 | 1.145 | 93.7% |
| 7,400 | 3.88 | 60.9% (63.8%) | 71.0 / 63.4 | 44.0 / 40.2 | not run | 72.1% |
| 8,910 | 4.67 | 60.1% (62.5%) | 72.3 / 65.8 | 42.3 / 39.6 | 1.120 | 95.6% |
| 9,500 | 4.98 | 61.9% (65.3%) | 74.2 / 69.2 | 44.5 / 41.2 | 1.120 | 95.2% |
| 9,700 | 5.09 | 61.7% (64.2%) | 72.7 / 67.9 | 45.6 / 41.6 | 1.119 | not run |
| 10,800 | 5.66 | 68.1% (66.0%) | 75.2 / 68.5 | 45.4 / 40.7 | 1.111 | 89.4% |
| 12,650 | 6.63 | 70.5% (68.0%) | 76.1 / 70.8 | 46.0 / 42.3 | 1.103 | 91.1% |
| 14,500 | 7.60 | 68.9% (66.9%) | 75.7 / 71.6 | 47.3 / 43.9 | 1.095 | 85.5% |
| 14,700 | 7.71 | 70.1% (67.2%) | 76.6 / 72.5 | 46.9 / 42.2 | 1.090 | 96.5% |
| 16,100 | 8.44 | 64.6% (67.3%) | 73.6 / 66.9 | 44.1 / 40.4 | 1.084 | 98.0% |
| 17,700 | 9.28 | 67.1% (69.2%) | 72.5 / 66.3 | 46.1 / 41.7 | 1.078 | 97.0% |
| 17,860 | 9.36 | 73.0% (70.0%) | 78.9 / 71.9 | 47.2 / 42.6 | 1.079 | 97.3% |
Compared with step 6,000, Fact1000 without the arithmetic items rose from 61.7% to 70.0%, SciQ from 72.4 to 78.9 and ARC-Easy from 44.4 to 47.2 (accuracy), the Pile from 1.145 to 1.079; ARC-Challenge stays at about chance. Since step 14,700 (one standard error is about 1.4 points on 1,000 items, 1.0 on ARC-Easy): Fact1000 without the arithmetic items +2.8, SciQ +2.3, ARC-Easy +0.3, ARC-Challenge -0.9, the Pile -0.0115 bits per byte. The curve is not monotone: SciQ and ARC-Easy dipped at step 16,100 and recovered by 17,860, and Fact1000 without the arithmetic items was 67.2, 67.3, 69.2 and 70.0 at 14,700, 16,100, 17,700 and 17,860, so differences of a point or two between checkpoints are within noise. The Pile figure was 1.0902, 1.0842, 1.0777 and 1.0786 bits per byte at the same four steps. The all-items Fact1000 score changed by +2.9 points since step 14,700 and by +4.9 since step 10,800 (the standard error of such a difference is about 2). Arithmetic oscillates between checkpoints (72% to 98% over the run). 2-digit multiplication: 92% at step 9,500, 58% at 10,800, 95% at 12,650, 23% at 14,500 (the first scratchpad line was right 95% of the time, but 76% of the replies then added a spurious second step), 87% at 14,700 and 93% at 17,860; its cause is not established.
Arithmetic
The training data includes arithmetic reasoning examples. The model answers arithmetic in a fixed chat-style format with a one-line scratchpad (no step-by-step working; the answer is computed in one forward pass):
<|im_start|>user
What is 683 + 479?<|im_end|>
<|im_start|>assistant
<think>
683 + 479 = 1162
</think>
1162<|im_end|>
14,000 problems (7 types × 2,000, fixed seeds, exact match), step 17,860: 2-digit addition 100.0%, 2-digit subtraction 97.3%, 3-digit addition 100.0%, 3-digit subtraction 92.3%, 4-digit addition 99.9%, 5-digit addition 98.9%, 2-digit × 2-digit 93.0%; all 97.3% (13,624 of 14,000). Without the chat format the same model scores under 1%: this is a trained, narrow skill.
Wide suite (7,636 further problems in 28 types, exact match, step 10,800): 7.3% overall (13.0% at step 9,500; not rerun at the latest step). Best types: negative numbers, mul / div 31%; powers 30%; decimals, add / sub 30%; negative numbers, add / sub 23%. Near zero: 6-8 digit numbers, fractions, GCD / LCM, series sums, averages, unit conversion, linear equations and word problems. Between steps 9,500 and 10,800, 3-digit × 1-digit multiplication fell from 94.1% to 22.7% and 3-digit × 2-digit from 61.2% to 0.0%: replies now often follow a correct first scratchpad line with a spurious second step (for example 691 * 8 = 5528 | 691 + 5528 = 6219). This is the same chaining that caused the 2-digit multiplication dip above, and the failure seen at step 7,400 coming back.
Wide suite by type
| Type | Items | Exact match | Reply finished |
|---|---|---|---|
| 3-digit × 1-digit | 286 | 22.7% | 100% |
| 3-digit × 2-digit | 286 | 0.0% | 100% |
| 3-digit × 3-digit | 286 | 10.5% | 100% |
| 6-8 digit addition | 286 | 0.7% | 100% |
| 6-8 digit subtraction | 286 | 0.0% | 100% |
| exact division | 286 | 16.4% | 100% |
| division rounded down | 286 | 2.1% | 100% |
| remainder / mod | 286 | 0.0% | 72% |
| negative numbers, add / sub | 286 | 23.1% | 100% |
| negative numbers, mul / div | 286 | 30.8% | 99% |
| decimals, add / sub | 286 | 29.7% | 100% |
| decimals, multiply | 286 | 2.1% | 100% |
| order of operations | 286 | 18.9% | 100% |
| powers | 186 | 30.1% | 90% |
| square / cube roots | 199 | 0.0% | 51% |
| percentages | 286 | 15.4% | 100% |
| fractions | 286 | 0.0% | 86% |
| greatest common divisor | 286 | 0.0% | 42% |
| least common multiple | 286 | 0.0% | 77% |
| factorial / choose / permutations | 101 | 1.0% | 89% |
| sums of series | 286 | 0.0% | 63% |
| rounding | 286 | 0.0% | 76% |
| linear equations in x | 286 | 1.0% | 100% |
| word problems, one step | 286 | 1.0% | 88% |
| word problems, two steps | 286 | 0.0% | 78% |
| average | 286 | 0.0% | 80% |
| median | 286 | 0.0% | 83% |
| unit conversion | 286 | 0.0% | 67% |
Model details
| Component | Setting |
|---|---|
| Width / depth | d_model 768, 18 blocks |
| Attention | 12 query heads × 64, 4 key/value heads (grouped-query), causal, per-head QK-norm |
| Positions | RoPE, θ = 500,000 |
| MLP | SwiGLU, hidden size 2,048 |
| Normalisation | RMSNorm, pre-norm; fp32 residual stream with fp16 sub-layers (the T4 has no bf16) |
| Embeddings | tied input/output, 49,152 rows |
| Bias / dropout | none |
Sixteen of the 18 layers are plain grouped-query attention (blocks 8 and 12 additionally carry the segment memory below): sliding-window, gated linear and DeltaNet-style alternatives were benchmarked on a T4 at 2k context and none beat flash-attention GQA on speed. KV cache: 18,432 bytes per token in fp16.
Segment memory (experimental)
From step 17,860 two blocks (8 and 12, counting from 0) can also read the keys and values of the earlier 2,048-token windows of the same text, so a window may use up to 32,768 tokens of context although every forward pass stays 2,048 tokens. For each query head the layer looks up the 32 best-matching stored keys (exact top-32; the stored keys carry no position, so RoPE is untouched), and adds the weighted values to its normal attention output through a per-head tanh gate that starts at exactly zero: at the moment it was switched on the model computed the same function as the step-17,860 checkpoint. The memory has its own query projection (initialised as a copy of the layer's) and its own optimiser for the gates; no gradient flows into the stored keys and values. Design after Memorizing Transformers (Wu et al., 2022) and the lessons of Hugging Face's failed Infini-attention re-implementation.
| Memory blocks | 8 and 12; generation ignores the memory |
| New parameters | 1,179,800 (152.2M in total) |
| Layout | per GPU and step, 8 runs of 16 consecutive windows; memory reset at every optimiser step; 524,288 tokens per step as before |
| Cost | 2.2× slower training (49 s against 22 s per step on 2 × T4), 11.7 GB against 11.1 GB |
Status: unproven. The model is not yet shown to use it for anything beyond repeated text. Tiny-model tests show the mechanism trains (slowly); the first held-out probe on this model is pending, and the memory may be switched off again if it does not help. Every benchmark number on this card is from step 17,860 or earlier, i.e. of the memory-free model. Checkpoints from step 17,900 contain 6 extra tensors (mem_gate, mem_q_proj.weight, mem_q_norm.weight in blocks 8 and 12): load them with strict=False or through zenyx100m.memattn.
Tokenizer
Byte-level BPE with a SuperBPE-style second stage (merges may cross spaces; digits stay single tokens; 248 special tokens). On 63 MB of text from the training sources it needs about 9% fewer tokens than the DeepSeek-V4-Pro tokenizer (4.56 against 4.14 bytes per token) with a vocabulary 2.6 times smaller; it ties on Wikipedia and is worse on markdown and GSM8K. Only English, code and math were measured. File: tokenizer_candidates_v3/F-49152-f70/tokenizer.json.
Training
| Optimiser | Muon on hidden matrices (lr 0.012), AdamW on embeddings and norms (lr 2e-3) |
| Schedule | warmup, constant until 3.9B tokens, then linear decay to 10% of peak between 3.9B and 15B tokens, held to the 50B-token budget |
| Batch | 524,288 tokens per step; about 21-25k tokens/s on 2 × T4 |
| Data | English web, code, math and chat text |
Other measurements
- MMLU (step 12,650), 5-shot, all 14,042 questions: 25.0% ± 0.7 (chance 25%; Zenyx-V3 26.9%).
- Python (step 10,900), twelve small functions run against unit tests (primes, factorial, palindrome, gcd, Fibonacci, FizzBuzz and similar): signature + docstring as plain completion 8 of 12 with greedy decoding and 43% of sampled attempts pass (8 samples per task, T = 0.6); chat format 5 of 12 and 34%; forcing
<think>first lowers it to 2 of 12 and 7%. (Step 6,000: 4 of 12 and 25%.) It still fails harder tasks. - Letter-choice multiple choice (step 10,900) in chat format is at chance (SciQ 19%, ARC-Easy 23%, ARC-Challenge 25%, 200 questions each) although the standard scoring of the same questions gives 75%, 45% and 24%: the model has the knowledge but not the "read four lettered options and pick one" skill.
- Decode speed (step 10,900), one T4, fp16, CUDA graph +
torch.compile: 349 tokens/s at batch 1, 2,572 tokens/s total at batch 8, 9,475 tokens/s at batch 64. - Long generation: greedy decoding repeats itself within about 30 words; suggested: sampling (temperature 0.8, top-k 50) with a repetition penalty of 1.1-1.2.
Planned post-training
A supervised fine-tuning stage is being prepared; no instruct model has been released and no result exists yet. ChatML with the tokenizer's real special tokens (<|im_start|>, <|im_end|>, <think>, </think>, <tool_call>, <tool_response>), loss on assistant tokens only; system prompts that must win over conflicting user instructions; a Reasoning effort: none | low | medium | high line with reasoning budgets of 256 / 512 / 1,024 tokens; English, code and math data only, audited before inclusion. The numbers in this card are the pre-SFT baseline.
Intended use and limitations
- Use: research, and as a starting point for fine-tuning. Not an assistant: it ignores system prompts and may continue a prompt with an unrelated document.
- Facts: expect errors and invented details; long generations repeat themselves.
- Arithmetic works in its own chat format only (see above) and not for big numbers, negatives, decimals, fractions, powers, algebra or word problems.
- Reasoning, MMLU, letter-choice multiple choice: at chance. Code (step 10,900): passes 8 of 12 small functions, not harder ones. The
<think>habit exists only for arithmetic. - Context 2,048 tokens; English only; not evaluated for bias or harmful content.
- Evaluation: one training run, mid-training checkpoints; Fact1000 and the question sets are written by the author; Muse2 numbers use an unofficial reimplementation.
Repository contents
| Path | Contents |
|---|---|
checkpoints/<run>/step_N/ |
model weights (model.pt), optimiser state, training state; only the two newest checkpoints are kept |
tokenizer_candidates_v3/ |
tokenizers, including the one used |
wheels/, runs/, data/ |
flash-attention build, training manifests |
The model code is not published yet.