LowOnMind-300k

A decoder-only language model with 296,960 parameters, pretrained from scratch on 200M tokens of HuggingFaceFW/fineweb-edu (sample-10BT).

It is an extreme variant of DedeProGames/DynamicMind-Mini (8.9M params), scaled down roughly 30x. The name is the point: this model is not built to be good. It is built to find out what a third of a million parameters actually buys you.

A 3.3x scale-up trained under identical conditions — same tokenizer, same data, same token budget, same aspect ratio — is published as DedeProGames/LowOnMind-1M. The two models form a controlled pair; see Scaling comparison below.

Architecture

	value
hidden_size	64
intermediate_size	136 (2.12x hidden)
num_hidden_layers	6
heads (q / kv)	4 / 2 (GQA, head_dim 16)
vocab_size	1024 (own byte-level BPE)
context	512
embeddings	tied
total params	296,960

Parameter split: 65,536 in the tied embedding, 231,360 across the six blocks, 64 in the final norm.

Changes from DynamicMind-Mini

• QK-Norm — per-head RMSNorm on queries and keys before RoPE. Costs 32 params per block and is what makes an LR of 2e-3 survivable at this scale.
• Precomputed RoPE — cos/sin are built once and reused instead of being recomputed every forward, and the cache re-expands automatically for sequences longer than max_position_embeddings.
• Scaled residual init — o_proj and down_proj initialize at std / sqrt(2 * num_layers) instead of a flat 0.02.
• 1024-token vocabulary — required to fit the parameter budget. At 8192 tokens the embedding alone would have been 8192 x hidden, larger than the entire model. The cost is compression: roughly 2.35 characters per token.

Training

data	HuggingFaceFW/fineweb-edu, sample-10BT
tokens	200M (6,103 steps x 32,768)
tokens/param	673
sequence length	512
batch size	64
optimizer	AdamW, betas (0.9, 0.95), wd 0.1
lr	2e-3 peak, cosine to 2e-4, 250 warmup
grad clip	1.0
hardware	1x Tesla T4
wall clock	43 min

The token budget is deliberately small in absolute terms, but at 673 tokens per parameter this run sits about 30x past the Chinchilla-optimal ratio. The model is parameter-starved, not data-starved: train and validation loss stayed within 0.05 of each other for the entire run, so there is no overfitting to speak of.

Language modelling results

metric	value
validation loss	3.2982 nats/token
validation perplexity	27.06
bits per character	2.03

Perplexity is not comparable across tokenizers — with a 1024-token vocabulary, 27.06 means something very different than it would at 50k. Bits per character (loss / ln 2 / 2.35 chars-per-token) is the portable number.

BananaMind Base Bench 1.1

Evaluated on BananaMind/BananaMind-Base-Bench-1.1, a 350-item English continuation-likelihood benchmark for base models. Scoring follows the official specification: context and each of the four continuations tokenized separately with add_special_tokens=False, no BOS, selection by highest mean conditional token log-probability.

Run validity: dataset SHA-256 matched, full schema validation passed, and no context required truncation against the 512-token window.

Category	Acc	Weighted acc	Elo
language_completion	46.0%	52.5%	921
commonsense	34.0%	35.6%	848
quantitative	32.0%	35.0%	938
logical_reasoning	24.0%	25.4%	905
world_knowledge	22.0%	21.6%	728
code_completion	14.0%	15.0%	789
context_tracking	14.0%	15.9%	707

	Overall
Overall Elo	833
Chance-level Elo (this item grid)	805
Raw accuracy	26.6%
95% CI	[22.0%, 31.2%]
z vs. chance	+0.69
Weighted accuracy	27.9%
Chance	25.0%

Sample output

Prompt: "The ", temperature 0.8, top_p 0.95.

The vehicle has been shown that the verb is a chief.
No:
- Even a good certain steps of the day, it is good and then a statement.
- It is a number of positive checking gain in the south of this power.
The risk of the main pottery, the straight guidelines of the brain's deployment.
- She is a little fruitful vegetables.
- It's an increasingly a brain, and this is, as a perfect men on the storage.
- Exodus of Donah, a conference, it is an aware of the message.
- The main straw of the problem.
- You're potential that the moments can be written in the majority of the paradigm.
- New York

This sample is more informative than the perplexity number. With a 1024-token byte-level vocabulary, words like "increasingly", "guidelines" and "deployment" do not exist as tokens — the model assembled each from four or five fragments in the correct order without misspelling one. That is orthography learned as a process, not as a lookup table. It also picked up document structure from FineWeb-Edu: bullet markers, a No: header, capitalization after periods, apostrophes in contractions. Even the invented word, "Donah", is invented with English phonotactics.

What it does not have is meaning beyond a span of about four tokens. "an aware of the message", "a little fruitful vegetables", "an increasingly a brain" — each three-to-four-word fragment is grammatical and the seams between them are not. That is the signature of ~2 bits per character: locally perfect grammar, zero tracking.

Scaling comparison vs LowOnMind-1M

DedeProGames/LowOnMind-1M is the same family at 985,152 parameters (3.32x), trained with the identical tokenizer, dataset, token budget (200M), schedule shape, and hidden/layers aspect ratio (10.7). Parameter count is the only thing that changed, so the comparison below isolates its effect.

metric	300k	1M	delta
validation loss	3.2982	2.9908	-0.3074
bits per character	2.03	1.836	-0.194
real-word rate (generated text)	not measured	98.0% (ceiling 98.4%)	—
BananaMind overall accuracy	26.6%	28.9%	+2.3pp (not significant, z=+0.67)
BananaMind language_completion	46.0% (+3.43σ vs chance)	52.0% (+4.41σ vs chance)	+6.0pp (not significant on its own, z≈0.6)

The clean result is on the intrinsic LM metrics. Validation loss and bits per character moved by a large, unambiguous margin — those are averaged over millions of validation tokens, so the estimate is tight. 3.3x the parameters bought a real, measurable drop in both.

The benchmark result is not clean, and that's informative on its own. 350 four-way items is not enough statistical power to confirm a 2.3-point accuracy gain; the unpaired z-test (z = 0.67) falls well short of significance, and LowOnMind-1M's own confidence interval still overlaps chance. The one category that clears significance against chance in both models is language_completion — consistent with the real-word-rate result and with the qualitative difference in the two models' sample outputs, where LowOnMind-1M shows register-appropriate structure (bibliography-style formatting after a date-led prompt, policy-style prose after a directive one) that this model does not. That agreement across three independent measurements is worth more than any single number here, including the Elo score.

Read the two models together, not the 300k score alone, if the question is "does scale help at this size." It does, clearly, on the metrics with enough samples to say so, and the benchmark is simply underpowered to confirm or deny it on its own.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DedeProGames/LowOnMind-300k")
model = AutoModelForCausalLM.from_pretrained(
    "DedeProGames/LowOnMind-300k", trust_remote_code=True
)

ids = tok("The ", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=64, use_cache=False)[0]))

trust_remote_code=True is required — the architecture ships as custom modeling code in the repository. use_cache=False is required: this implementation has no KV cache, and generation recomputes the full window each step.

Limitations

At ~300k parameters this model learns spelling, spacing, morphology and short n-grams, but does not produce coherent text. It has no factual knowledge, no reasoning ability, and no capacity to track state across a passage. It should not be used for anything except research into the lower end of the scaling curve.