lyfeyvutha/nllb_350M_en_km_v1

NLLB-350M-EN-KM-v1

Model Description

This model is a compact English-to-Khmer neural machine translation model created through knowledge distillation from NLLB-200. This is the proof-of-concept version (1 epoch) demonstrating the feasibility of the distillation approach.

• Developed by: Chealyfey Vutha
• Model type: Sequence-to-sequence transformer for machine translation
• Language(s): English to Khmer (en → km)
• License: CC-BY-NC 4.0
• Base model: facebook/nllb-200-distilled-600M
• Teacher model: facebook/nllb-200-1.3B
• Parameters: 350M (42% reduction from 600M baseline)

Model Details

Architecture

• Encoder layers: 3 (reduced from 12)
• Decoder layers: 3 (reduced from 12)
• Hidden size: 1024
• Attention heads: 16
• Total parameters: ~350M

Training Procedure

• Distillation method: Temperature-scaled knowledge distillation
• Teacher model: NLLB-200-1.3B
• Temperature: 5.0
• Lambda (loss weighting): 0.5
• Training epochs: 1 (proof of concept)
• Training data: 316,110 English-Khmer pairs (generated via DeepSeek API)
• Hardware: NVIDIA A100-SXM4-80GB

Intended Uses

Direct Use

This model is intended for:

• English-to-Khmer translation tasks
• Research on knowledge distillation for low-resource languages
• Proof-of-concept demonstrations
• Computational efficiency research

Downstream Use

• Integration into translation applications
• Fine-tuning for domain-specific translation
• Baseline for further model compression research

How to Get Started with the Model

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, GenerationConfig

# Configuration
CONFIG = {
"model_name": "lyfeyvutha/nllb_350M_en_km_v10",
"tokenizer_name": "facebook/nllb-200-distilled-600M",
"source_lang": "eng_Latn",
"target_lang": "khm_Khmr",
"max_length": 128
}

# Load model and tokenizer
model = AutoModelForSeq2SeqLM.from_pretrained(CONFIG["model_name"])
tokenizer = AutoTokenizer.from_pretrained(
CONFIG["tokenizer_name"],
src_lang=CONFIG["source_lang"],
tgt_lang=CONFIG["target_lang"]
)

# Set up generation configuration
khm_token_id = tokenizer.convert_tokens_to_ids(CONFIG["target_lang"])
generation_config = GenerationConfig(
max_length=CONFIG["max_length"],
forced_bos_token_id=khm_token_id
)

# Translate
text = "Hello, how are you?"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, generation_config=generation_config)
translation = tokenizer.decode(outputs, skip_special_tokens=True)
print(translation)

Training Details

Training Data

• Dataset size: 316,110 English-Khmer sentence pairs
• Data source: Synthetic data generated using DeepSeek translation API
• Preprocessing: Tokenized using NLLB-200 tokenizer with max length 128

Training Hyperparameters

• Batch size: 48
• Learning rate: 3e-5
• Optimizer: AdamW
• LR scheduler: Cosine
• Training epochs: 1
• Hardware: NVIDIA A100-SXM4-80GB with CUDA 12.8

Evaluation

Testing Data

The model was evaluated on the Asian Language Treebank (ALT) corpus, containing manually translated English-Khmer pairs.

Metrics

Metric	Value
chrF Score	21.3502
BERTScore F1	0.8983

Results

This proof-of-concept model demonstrates that knowledge distillation can achieve reasonable translation quality with significantly reduced parameters (350M vs 600M baseline).

Limitations and Bias

Limitations

• Limited training: Only 1 epoch of training; performance may improve with extended training
• Synthetic data: Training data generated via API may not capture all linguistic nuances
• Domain specificity: Performance may vary across different text domains
• Resource constraints: Optimized for efficiency over maximum quality

Bias Considerations

• Training data generated via translation API may inherit biases from the source model
• Limited evaluation on diverse Khmer dialects and registers
• Potential cultural and contextual biases in translation choices

Citation

@misc{nllb350m_en_km_v1_2025, title={NLLB-350M-EN-KM-v1: Proof of Concept English-Khmer Neural Machine Translation via Knowledge Distillation}, author={Chealyfey Vutha}, year={2025}, url={https://huggingface.co/lyfeyvutha/nllb_350M_en_km_v1} }

Model Card Contact

For questions or feedback about this model card: [email protected]