Tajik FastText Word Embedding Model

This repository contains a pretrained FastText model for the Tajik language, trained on a large corpus of Tajik texts. The model supports subword information, allowing it to generate embeddings even for rare or unseen (OOV) words.

The model is suitable for use in various NLP tasks such as:

Semantic analysis
Text classification
Machine translation
Synonym detection and thesaurus building
Enhancing other models through embedding initialization

Licensed under the MIT License, which allows free usage in both research and commercial applications.

📊 Model Overview

Parameter	Value
Model Type	FastText (with subwords)
Vector Size	300
Vocabulary Size	145,232
OOV Support	Yes
Context Window	5
Min Word Count	≥ 5

📚 Training Corpus

Books (Total: 99)

Programming: 6
History: 4
Religion: 12
Scientific: 3
Children's literature: 6
Prose: 19
Poetry: 21
Textbooks: 28

Articles (Total: 134,497)

Asia-Plus: 20,471
Khovar: 21,557
Ovozi Tojik: 7,495
Farazh: 4,679
Wikipedia: 80,295

Total Corpus Statistics

Documents: 134,596
Tokens: 33,535,383
Unique Lemmas: 649,308

🧪 Model Comparison with Meta FastText

We evaluated our model against Meta’s pretrained FastText using semantic similarity and Spearman correlation:

Model	Spearman Correlation	OOV Support
FastText (Meta)	0.703	Yes
FastText (ours)	0.622	Yes

While Meta FastText achieves better overall performance, our model demonstrates strong results on Tajik-specific morphology and semantics.

🔍 Example Similar Words

Word	Nearest Neighbors (FastText)
кӯдак	кӯдаку(0.82), хурдкӯдак(0.81), кӯдакам(0.81), кӯдакат(0.81), кӯдаке(0.81)
муаллим	муаллиме(0.90), муаллимат(0.89), муаллимин(0.89), муаллиму(0.88), муаллима(0.88)
об	оби(0.79), обро(0.74), обмӯрии(0.70), обшустаи(0.68), обшуста(0.66)
мард	марда(0.87), мардхӯ(0.85), мардвор(0.85), мардро(0.83), зан(0.82)
деҳа	деҳайи(0.83), деҳаю(0.80), деҳавз(0.78), деҳакӣ(0.76), деҳодеҳ(0.74)
китоб	китобӣ(0.84), китобгуна(0.83), китобча(0.81), китобсӯзӣ(0.81), китобро(0.81)
меҳмон	меҳмонӣ(0.86), меҳмоншо(0.85), меҳмонат(0.83), меҳмонҳона(0.82), меҳмони(0.82)
шаҳр	шаҳрӯ(0.82), шаҳрча(0.80), бушаҳр(0.79), шаҳрат(0.79), навшаҳр(0.79)
падар	падаршӯ(0.89), падарӣ(0.84), падаршӯву(0.84), падаре(0.84), падаршон(0.83)
модар	модаршӯ(0.86), модаршӯяш(0.83), модару(0.81), модаре(0.81), модарвор(0.80)

🧩 Handling OOV (Out-of-Vocabulary) Words

FastText supports generating vectors for unknown words via subword units (n-grams). Here are some examples:

Unknown Word	Closest Matches (FastText)
кӯдакона	кӯдаконаи(0.82), кӯдаконат(0.81), кӯдаконае(0.81)
меҳмонамон	меҳмон(0.77), меҳмонҳо(0.77), меҳмонам(0.76)
муаллимон	муаллимони(0.89), муаллимоне(0.88), муаллимону(0.83)
деҳоти	дарҷамоати(0.79), чамоати(0.74), ҷамоати(0.81)
саводнок	саводнокӣ(0.88), саводнокиву(0.85), саводнокии(0.84)

📌 Features for Tajik Language

Our model performs well on:

Semantic similarity: e.g., "мард" ↔ "зан", "китоб" ↔ "китобгуна"
Morphological variants: e.g., "кӯдак" → "кӯдаку", "кӯдаки"
Rare/compound words: thanks to subword representations like "саводнок", "деҳоти"

💡 Usage Example

from gensim.models import FastText

model = FastText.load("tajik_fasttext.model")
vector = model.wv["падар"]  # Get vector for a word
similar_words = model.wv.most_similar("модар")  # Find similar words

🗂️ Files Included

File	Description
`tajik_fasttext.model`	Gensim FastText model file
`*.npy` files	Supporting NumPy arrays for vectors

📚 Citation

If you use this model, please cite:

@misc{ArabovMK_Tajik_FastText,
  author = {ArabovMK},
  title = {Tajik FastText Word Embeddings},
  year = 2025,
  publisher = {Hugging Face},
  url = {https://huggingface.co/ArabovMK/tajik-fasttext-model}
}

Last updated: 2025-05-10