Bochkov/max_bvv_ru

Research demo: Russian Causal Language Model with Frozen, Visual-based Token Embeddings

Model size: 0.4B parameters
Tokenizer: Custom Unicode-centric, compatible with max_bvv_zh and max_bvv_moe
Intended use: Demonstration of frozen, non-semantic token embeddings for LLMs

---

📝 Model description

max_bvv_ru is a Causal Language Model trained on Russian data with a unique property:
its token embedding matrix is frozen and built from visual/Unicode glyph features of tokens, not optimized during language modeling training.

Purpose: Showcase that transformer blocks (not embeddings) are capable of learning nontrivial semantics, and enable future model fusion via shared embeddings.

Not intended for production or high-stakes applications.
This checkpoint is for experimentation, ablation, and research in embedding-free LLM semantics!

---

🏹 Evaluation

• Avg. MMLU: 23.58%
• SQuAD: 18.95%
• ARC-e: 21.65%
• BLEU (en-ru): 8.65%

Note: All metrics are lower than SOTA due to deliberate undertraining and tiny corpus size (for concept demo).

---

⚠️ Limitations Research use only. Trained on a small, non-exhaustive Russian subset. Quality, robustness, and reasoning are much lower than SOTA models. SFT was only lightly applied; not intended for real world use.

🧑‍🔬 Citation & Concept

If you use this model or the underlying concepts in your research, please cite our work:

@misc{bochkov2025emergentsemanticstokenembeddings,
      title={Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations}, 
      author={A. Bochkov},
      year={2025},
      eprint={2507.04886},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.04886}, 
}

@misc{bochkov2025growingtransformersmodularcomposition,
      title={Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate}, 
      author={A. Bochkov},
      year={2025},
      eprint={2507.07129},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2507.07129}, 
}

This work demonstrates that transformer blocks, not token embeddings, carry the semantic burden in LLMs — a step toward modular, fusable, multilingual LMs.

🔥 Example

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained('Bochkov/max_bvv_ru', trust_remote_code=True).to('cuda')
tokenizer = AutoTokenizer.from_pretrained('Bochkov/max_bvv_ru', trust_remote_code=True)

inputs = tokenizer.encode("Привет, мир! ", return_tensors="pt").to('cuda')
outputs = model.generate(inputs, max_new_tokens=50, temperature=0.8, top_k=50, top_p=0.95, do_sample=True)
print(tokenizer.decode(outputs[0].tolist()))