ZamAI-Llama3-8B-Pashto

ZamAI-Llama3-8B-Pashto is a fine-tuned version of Meta's Llama-3 8B large language model for the Pashto language, developed by tasal9.

---

Model Details

• Base Model: meta-llama/Meta-Llama-3-8B
• Parameter Count: 8B (8,000,000,000)
• Fine-Tuned By: tasal9
• Language: Pashto (پښتو)
• Adapter: LoRA (Low-Rank Adaptation)
• Framework: PyTorch, Hugging Face Transformers, PEFT

---

Training Data

• Source: Pashto raw text corpus
• Size: ~104,000 documents
• Preprocessing: Split, cleaned, and tokenized to 512 tokens max per document

---

Training Procedure

• Hardware: NVIDIA A100 GPU
• Batch Size: 4 per device, gradient accumulation steps: 2
• Precision: bfloat16 (bf16) / float16 (fp16)
• LoRA Configuration: r=16, alpha=32, dropout=0.05
• Epochs: 1
• Checkpoint Interval: every 500 steps
• Optimizer: AdamW
• Scheduler: Linear warmup

---

Usage

Load Adapter Weights

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel, PeftConfig

base_model_id = "meta-llama/Meta-Llama-3-8B"
adapter_id = "tasal9/ZamAI-Llama3-8B-Pashto"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(base_model_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base_model, adapter_id)

inputs = tokenizer("ستړی مشي!", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Inference Example

prompt = "د پښتو ژبې لپاره مصنوعي هوښیار مرستندوی"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Checkpoints

• Latest: checkpoint-11700
• Each checkpoint includes model adapter weights (adapter_model.safetensors), tokenizer files, optimizer state, and training metadata.

---

Intended Use

• Research: Pashto NLP, LLM adaptation, LoRA fine-tuning experiments
• Applications: Chatbots, text generation, educational tools for Pashto

---

Limitations

• Trained for one epoch; performance may be limited for production
• Only covers data seen during fine-tuning; may not generalize to all Pashto dialects or domains

---

Citation

If you use this model, please cite:

@misc{tasal9_zamai_llama3_8b_pashto_2025,
  title={ZamAI-Llama3-8B-Pashto},
  author={tasal9},
  howpublished={\url{https://huggingface.co/tasal9/ZamAI-Llama3-8B-Pashto}},
  year={2025}
}

---

License

• Base model: Meta Llama 3 license
• Adapter/fine-tuning: Apache 2.0

---

Contact

• tasal9 on Hugging Face