nemo_bvv_ru

This is nemo_bvv_ru, a proof-of-concept Russian language causal language model trained with completely frozen, precomputed token embeddings based on SOTA Mistral/Nemo tokenizer (visual string composition, not standard semantic embeddings).

Model Details

Parameters: ~400M
Vocabulary: 131072 tokens (Mistral Nemo, precomputed embeddings)
Architecture: 12-layer, 12-head, hidden size 1024 GPT-style transformer
Frozen Embeddings: Yes (not learned during training)
Training: Limited Russian corpus, SFT up to 10%
Intended Purpose: Research, demonstration of shared-embedding/fusion paradigm

Performance

This model is a concept exploration, not a commercial or high-accuracy system.

MMLU (average): 8.80%
ARC-e: 19.53%
ARC-c: 21.34%
Commonsense-QA: 19.51%
SQUAD: 6.80%
BLEU [en-ru]: 4.68% / [ru-en]: 5.71%

Limitations

Trained on a tiny research corpus for demonstration only.
Accuracy is far below large-scale production models.
Not suitable for commercial or mission-critical tasks.
No safety or fairness tuning.

🧑‍🔬 Citation & Concept

If you use this model or the underlying concepts in your research, please cite our work:

@misc{bochkov2025emergentsemanticstokenembeddings,
      title={Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations}, 
      author={A. Bochkov},
      year={2025},
      eprint={2507.04886},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.04886}, 
}

@misc{bochkov2025growingtransformersmodularcomposition,
      title={Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate}, 
      author={A. Bochkov},
      year={2025},
      eprint={2507.07129},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2507.07129}, 
}

This work demonstrates that transformer blocks, not token embeddings, carry the semantic burden in LLMs — a step toward modular, fusable, multilingual LMs.

Usage Example

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained('Bochkov/nemo_bvv_ru', trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained('Bochkov/nemo_bvv_ru', trust_remote_code=True).to('cuda')
inputs = tokenizer("Привет, мир! ", return_tensors="pt").to('cuda')
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Bochkov
/

nemo_bvv_ru