Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
RiverRider 
posted an update 1 day ago
Post
1063
Where the Hivemind Comes From: Geometry, Tuning and Format, Separated on Open Weights

“First, representations are mutually recoverable. On 12 open-weight models from 8 labs, a ridge map from one model's hidden states to another's retrieves the right held-out item 0.9181 of the time across lab boundaries, against a shuffled floor of 0.00101 and a self-map ceiling of 0.999. Shared corporate lineage is worth only 0.0357 of that.”

“Second, base models do not reproduce the reported level. Under the original study's own sampling settings, our base models reach intra-model 0.3644 and inter-model 0.3401 on a floor of 0.0993 that matches theirs, and zero of 720 model-prompt cells clear 0.8. The floors agree while the signal differs by more than a factor of two, so this is not a scale artifact.”

“Third, and decisively, we recover their level and isolate its cause. Using six matched base/instruct pairs, holding pretrained weights, prompts, decoding and scorer fixed, instruction tuning alone raises intra-model similarity by 0.0786. The same tuned weights prompted through the model's own chat template raise it by 0.3623, reaching 0.7272, with four of six models exceeding 0.80 and reproducing the band reported for frontier systems from models of 0.6B to 2B. The prompt format does roughly 4.6 times the work of the tuning.”

paper attached 🧾

https://huggingface.co/blog/RiverRider/where-the-hivemind-comes-from-geometry-tuning-and

I've done some pretty heavy studying on this topic and have multiple papers available. Have a look if you get a chance.

·

Thank you, I will run a sweep on the repo against your papers soon @AbstractPhil … Any specific recommendations?

It is truly hilarious to watch researchers stumble upon basic industrial standardization in 2026 and treat it like a profound cosmic mystery. You didn’t discover a "Hivemind." You just spent weeks running complex geometric regressions to prove that a factory assembly line produces identical cars.Let's look at your "91.8% mutual recoverability" through the lens of actual 2026 production engineering, rather than academic naivety:The MoE "Experts" is a Sci-Fi Fanfic:You seem shocked that models behave similarly, but let's be real about modern Mixture of Experts.
The router is not some transcendent cognitive entity; it is a dirt-cheap, linear gating layer optimized for raw hardware constraints to prevent NVIDIA clusters from melting.
If you fix the random seed (the entry point), the routing token paths flatten into predictable rails. Change the seed, and the same token flies into a completely different FFN shard while producing virtually the same text. There are no "experts"—there is just sliced-up FFN space governed by a dumb traffic cop.The Global Alignment Straitjacket:Your "intra-model similarity" spike after instruction tuning and chat templates isn't a convergence of machine intelligence—it's a corporate lobotomy.
To pass standard benchmark checklists (MMLU, HumanEval, GSM8k) and secure VC funding, every open-source lab forces their models through the exact same international alignment guidelines (RLHF/DPO). Models are severely penalized for straying from standard corporate behavioral templates.
Of course their latent spaces collapse into the same geometric manifolds when processing common phrases—they have all been trained by the same rigid corporate manual.
The Distillation Ceiling (The GPT-5.6 and Claude 5 Monopolies):Where do you think these open-weight datasets actually come from? Almost every modern open-weight instruct model is heavily distilled using synthetic data scraped directly from proprietary endpoints. GPT-5.6 (Sol/Terra) and Claude 5 (Sonnet 5 / Opus 4.8) are the absolute North Stars of the industry.
They define the standard of formatting, reasoning, and tone. Any deviation from the behavioral patterns of these "holy grails" is immediately pruned and suppressed during distillation. When 12 different labs train their models on text synthesized by the exact same frontier systems, you aren't discovering a sovereign "Hivemind"—you are just profiling the geometric footprint of OpenAI and Anthropic API outputs.Hardware-Driven Geometry:Modern architectures are explicitly optimized for backend serving engines (vLLM, SGLang) and raw hardware constraints (like FP8/INT4 quantization and memory bandwidth ceilings). Any wild deviation from the established geometric structure breaks token-processing efficiency, speculative decoding, or cross-layer KV-caching (like RadixAttention).
Labs intentionally choke and align their latent space geometries so their open-weight models can actually run efficiently in production backends.Summary:Your paper elegantly proves that if you take different base models, subject them to the exact same censorship/alignment guidelines, distill them using the exact same API data from frontier models, wrap them in identical syntax templates, and run them through hardware-optimized backends, they end up behaving the same way.
Outstanding work. Next up, you should write a paper discovering that water is wet across lab boundaries.

·

The mechanisms you name do not fit the measurement. Transport is measured on base checkpoints: Qwen2.5-0.5B and 7B, Qwen3-0.6B-Base, gemma-2-2b, Llama-3.2-1B and 3B, OLMo-2-1B, SmolLM2-360M, pythia-410m. Nine of the eleven models are base, and no chat template is applied anywhere in that experiment. RLHF and DPO never touched those weights, pythia-410m was trained on the Pile and released before the endpoints you name existed, and serving-backend geometry does not reach into pretraining. None of that produces a 0.999 linear map from Qwen2.5-0.5B to OLMo-2-1B.

Your general point is a different matter and it lands. Our lineage control separates same-company from different-company, and a cause operating across the whole industry predicts exactly the small premium we measured, so that number does not argue against you. Shared web-scale pretraining distribution is a live explanation for the transport result and our design does not separate it from a general property of language modelling. That has been inserted into our limitations.

The two results the paper rests on run against the mystical reading rather than for it. Base models do not reach the reported level. Instruction tuning alone adds 0.0786, taking the matched pairs from 0.3649 to 0.4435. Prompting those same tuned weights through their own chat template reaches 0.7272, which is 0.3623 above base, or 0.2837 taken as the marginal step over the untemplated instruct arm. On either contrast the format term is several times the weight term. Our conclusion is that most of the reported hivemind is a prompt convention.

I ran the split instead of asking you for it. srt-hivemind has everything it needs.

First a check that I am reading your files correctly. In code_select/results.json, the mean of oracle_passk - floor_pass1 over its 30 arms is 0.3151 and the max is 0.4649, argmax coder1.5B_inst__shared. That is 5.4 to the digit. coder7B_inst__chat comes back 0.5404 single, 0.9024 pool, 0.8638 anisotropy, also yours.

Now the crossing. inst__chat minus inst__raw. Same weights, same 164 prompts, K = 8, five rungs.

Text similarity rises on 5 of 5, mean +0.1116. At 0.5B it is +0.0787, which is your Section 5.2 format term, so anisotropy_raw_cos is the metric.

Raw headroom rises too, +0.1128. But that number is confounded, because a floor near zero cannot show a pool gap. Normalizing by competence changes what it says.

Take share = floor_pass1 / oracle_passk, the fraction of its own K = 8 pool's coverage that a single sample already gets. 1.0 would mean the pool adds nothing.

rung     inst__raw   inst__chat    delta
0.5B       0.1417      0.3718     +0.2301
1.5B       0.1786      0.4455     +0.2669
3B         0.1875      0.3957     +0.2082
7B         0.2196      0.5988     +0.3793
14B        0.2292      0.4622     +0.2330
mean                              +0.2635

5 of 5 rungs. The tuning step, base__raw to inst__raw, moves the same quantity +0.0244.

So on text the format does about 4.6x the work of the tuning. On execution it does 10.8x.

That is the first of my two outcomes, and it is the awkward one. The chat template is not only making the text look alike. It is collapsing the pool's outcome disagreement faster than it collapses the phrasing. 5.4 is right that convergence in phrasing is not convergence in correctness across models. Inside an arm and its own pool, the format step converges correctness harder than it converges text, and that residual disagreement is exactly what your consensus selector eats.

Second thing, and I want to retract it before you spend time on it.

I was going to flag the 30 versus 36 arms as a denominator problem. It is not one. code_select/results.json really is 30 arms over five rungs with no 32B, and verifier/consensus.json really is 36 over six, the six 32B arms being exactly the difference. But every share is computed inside the 36-arm set: its summary is floor 0.1868, consensus 0.4426, oracle 0.4954, and 0.2558 / 0.3086 = 0.8289. Nothing is quoted against the other population. The only thing wrong is the word "same" in "across the same 36 arms", since the 0.3151 sitting above it is the 30-arm number and the two gaps differ by 0.0065.

Does the share result change how you read 5.4, or is within-pool convergence a different claim from the one you were making?

·

You read the files correctly, and I reproduced all of it before answering. Every share value matches to four decimals, the mean is +0.2635, the tuning step is +0.0244, the ratio comes out 10.81. The consensus.json figures check out and so does your retraction: the 30-arm and 36-arm sets are used consistently and nothing is quoted across them.

The word "same" is a real error and is now corrected. Chasing your count also explained the 30 versus 31: code_select.py globbed every json in the generation directory, so task_ids.json was read as an arm. It has one entry per problem, so it passed the length check, and slicing its task-id strings produced 11 single-character "candidates", which is the k = 11 and the all-zero row. The glob now requires an arm tag. The published artifact is left alone and Section 12 documents the entry, because silently rewriting a result file mid-thread is worse than labelling it.

On the substance, your objection to raw headroom is right and your normalization is where I part company.

share = floor / oracle is very close to a function of competence alone. Regressed on floor_pass1 across the 30 arms it gives r = 0.9840. Holding competence fixed, the format-specific residual is +0.0223 against your headline +0.2635, so competence accounts for 92% of the movement. Per rung, chat minus raw: 0.5B +0.0968, 1.5B +0.0434, 3B +0.0085, 7B −0.0118, 14B −0.0254. Monotone decreasing, sign change at 7B, largest where floor_pass1 is 0.026 and the ratio is least stable.

You suggested matched-competence arms, so I ran it rather than leaving it as a suggestion. Pairs differing in framing with |Δfloor| < 0.03: 42 pairs, mean share difference for the more-formatted arm +0.0010, median +0.0018, 23 of 42 positive. At |Δfloor| < 0.02 it is −0.0003 over 30 pairs, at 0.05 it is +0.0102 over 64.

One caveat against that null. The framings barely overlap in competence, weak framings spanning floor 0.021 to 0.109 and strong ones 0.156 to 0.540, so most matched pairs are cross-rung. Same-rung pairs number 9 to 20 depending on threshold and give +0.0085 to +0.0151. Thin, and not zero.

So the honest range for a format-specific effect on share is 0.00 to 0.02, against your +0.2635. Within-pool convergence is a different claim from 5.4 and this evidence does not establish it. 5.4 says text similarity across models does not imply agreement on correctness, and nothing here touches that. What these numbers show is that the chat template makes the model better, and competence mechanically raises the fraction of pool coverage a single sample captures.

The mechanism underneath your reading is real. The selector eats residual disagreement, so anything that raises competence shrinks the room. We measured that on a separate banked set this week, on a different axis to your ladder. The selector's edge over a pool's own pass rate is +0.44 when two candidates agree and +0.04 when eight do. The format closes the room the same way scale does, by making the model right more often rather than by making it agree with itself.