metythorn/khmer-ngram-models

Khmer N-gram Language Models

This repository contains KenLM n-gram language models for Khmer language.

Files

• khmer_bigrams.binary: Bigram language model
• khmer_trigrams.binary: Trigram language model

Usage

import kenlm

# Load models
bigram_model = kenlm.LanguageModel("khmer_bigrams.binary")
trigram_model = kenlm.LanguageModel("khmer_trigrams.binary")

# Score a sentence
sentence = "ខ្ញុំ ស្រលាញ់ ភាសា ខ្មែរ ។"
bigram_score = bigram_model.score(sentence)
trigram_score = trigram_model.score(sentence)

print(f"Bigram score: {bigram_score}")
print(f"Trigram score: {trigram_score}")

Installation

pip install kenlm

Download Models

from huggingface_hub import hf_hub_download

# Download bigram model
bigram_path = hf_hub_download(
    repo_id="YOUR_USERNAME/khmer-ngram-models",
    filename="khmer_bigrams.binary"
)

# Download trigram model
trigram_path = hf_hub_download(
    repo_id="YOUR_USERNAME/khmer-ngram-models", 
    filename="khmer_trigrams.binary"
)