hugohow/5gram-french-kenLM

Install

pip install kenlm huggingface_hub

Simply run the following code:

import kenlm
from huggingface_hub import hf_hub_download

# Téléchargement du fichier du modèle depuis le Hub Hugging Face
model_file = hf_hub_download(
    repo_id="hugohow/5gram-french-kenLM", filename="lm_5gram_big.bin"
)

# Chargement du modèle KenLM
model = kenlm.Model(model_file)

# Get perplexity
print(model.perplexity("je vais à la maison"))
# 20.6 (low perplexity, since sentence style is formal and with no grammar mistakes)

print(model.perplexity("jevais à la maisond"))
# 17041.7 (high perplexity, since the sentence is colloquial and contains grammar mistakes)