pro_bvv_ru: 200M param frozen-embedding EN-RU LM

Description

Experimental 200M parameter language model jointly trained on an English-Russian (EN-RU) corpus with frozen, visually-motivated token embeddings. Designed for demonstration of cross-lingual learning without updating embeddings.

Training details

• EN-RU corpus (mix), ~9B tokens total.
• 10% SFT (EN and RU QA datasets).
• Frozen, non-semantic token embeddings (Unicode+bigram/trigram mixed).
• Rest of the model is trainable.

Evaluation

Task	pro_bvv_ru
MMLU	22.63% ± 0.19%
ARC-e	23.63% ± 0.95%
ARC-c	22.91% ± 1.52%
C-SENSE	20.26% ± 0.44%
BLEU [en-ru]	6.14% ± 0.21%
BLEU [ru-en]	8.07% ± 0.43%

---

⚠️ Limitations Research use only. Trained on a small subset. Quality, robustness, and reasoning are much lower than SOTA models. SFT was only lightly applied; not intended for real world use.

🧑‍🔬 Citation & Concept

If you use this model or the underlying concepts in your research, please cite our work:

@misc{bochkov2025emergentsemanticstokenembeddings,
      title={Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations}, 
      author={A. Bochkov},
      year={2025},
      eprint={2507.04886},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.04886}, 
}

@misc{bochkov2025growingtransformersmodularcomposition,
      title={Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate}, 
      author={A. Bochkov},
      year={2025},
      eprint={2507.07129},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2507.07129}, 
}

This work demonstrates that transformer blocks, not token embeddings, carry the semantic burden in LLMs — a step toward modular, fusable, multilingual LMs.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tokenizer = AutoTokenizer.from_pretrained('Bochkov/pro_bvv_ru')
model = AutoModelForCausalLM.from_pretrained('Bochkov/pro_bvv_ru', trust_remote_code=True).to('cuda')
inputs = torch.tensor([tokenizer.encode("Example input: ")], device='cuda')
outputs = model.generate(inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0]))