Upload README.md

README.md

@@ -0,0 +1,185 @@
+<div align="center">
+  <img src="https://huggingface.co/bharatgenai/Param-1-2.9B-Instruct/resolve/main/BharatGen%20Logo%20(1).png" width="60%" alt="BharatGen" />
+</div>
+<hr>
+<div align="center">
+  <a href="#" style="margin: 4px; pointer-events: none; cursor: default;">
+    <img alt="Paper" src="https://img.shields.io/badge/Paper-Coming%20Soon-lightgrey?style=flat" />
+  </a>
+  <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank" style="margin: 4px;">
+    <img alt="License" src="https://img.shields.io/badge/License-CC--BY--4.0-blue.svg" />
+  </a>
+  <a href="#" target="_blank" style="margin: 4px;">
+    <img alt="Blog" src="https://img.shields.io/badge/Blog-Read%20More-brightgreen?style=flat" />
+  </a>
+</div>
+
+# AyurParam
+BharatGen introduces AyurParam, a domain-specialized large language model fine-tuned from Param-1-2.9B-Instruct on a high-quality Ayurveda dataset. It is designed to handle Ayurvedic queries, classical text interpretation, clinical guidance, and wellness knowledge. Ayurveda offers vast traditional medical wisdom, yet most language models lack domain-specific understanding. AyurParam bridges this gap by combining Param-1’s bilingual strengths with a curated Ayurvedic knowledge base, enabling contextually rich and culturally grounded responses.
+
+## 🏗 Model Architecture
+AyurParam inherits the architecture of Param-1-2.9B-Instruct:
+* Hidden size: 204
+* Intermediate size: 7168
+* Attention heads: 16
+* Hidden layers: 32
+* Key-value heads: 8
+* Max position embeddings: 2048
+* Activation: SiLU
+* Positional Embeddings: Rotary (RoPE, theta=10000)
+* Attention Mechanism: Grouped-query attention
+* Precision: bf16-mixed
+* Base model: Param-1-2.9B-Instruct
+
+## 📚 Data Preparation
+AyurParam’s training corpus was carefully crafted to ensure deep Ayurvedic knowledge, Sanskrit/English bilingual accessibility, and clinical relevance.
+Steps involved:
+1. Source Gathering
+  * 15k+ passages from classical Ayurvedic texts (digitized and curated).
+  * 10k+ passages from AYUSH ministry guidelines, research papers, and clinical case discussions.
+2. Question Generation
+  * 5 curated Q&A pairs generated per passage using an open-source LLM + domain expert review.
+3. Domain Taxonomy & Personas
+  * Built an Ayurveda-specific taxonomy (Dosha, Dhatu, Mala, Srotas, Nidana, Chikitsa, etc.).
+  * Defined multiple personas: student, vaidya (physician), researcher, policymaker, wellness coach.
+4. Dataset Construction
+  * 1.5M Q&A pairs grounded in taxonomy and personas.
+  * 4M multi-turn conversation samples created.
+  * Sanskrit terminology preserved with transliteration and explanations.
+
+
+## 🏋️ Training Setup
+* Base model: Param-1-2.9B-Instruct
+* Training framework: Hugging Face + TRL (SFT) + torchrun multi-node setup
+* Prompt template: Custom-designed for Ayurvedic inference
+* Scheduler: Linear with warmup
+* Epochs: 3
+* Total training samples: ~8M
+* Test samples: ~800k
+* Base learning rate: 5e-6
+* Minimum learning rate: 0
+* Additional tokens: <user>, <assistant>, <context>, <system_prompt>
+* Vocab size: 256k + 4
+* Global batch size: 1024
+* Micro batch size: 4
+* Gradient accumulation steps: 32
+
+
+## 🚀 Inference Example
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+model_name = "bharatgenai/AyurParam"
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    trust_remote_code=True,
+    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.bfloat32,
+    device_map="auto"
+)
+
+# Example Ayurvedic query
+user_input = "What is the Samprapti (pathogenesis) of Amavata according to Ayurveda?"
+
+# Prompt styles
+# 1. Generic QA: <user> ... <assistant>
+# 2. Context-based QA: <context> ... <user> ... <assistant>
+# 3. Multi-turn conversation (supports up to 5 turns): <user> ... <assistant> ... <user> ... <assistant>
+
+prompt = f"<user> {user_input} <assistant>"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+with torch.no_grad():
+    output = model.generate(
+        **inputs,
+        max_new_tokens=300,
+        do_sample=True,
+        top_k=50,
+        top_p=0.95,
+        temperature=0.6,
+        eos_token_id=tokenizer.eos_token_id,
+        use_cache=False
+    )
+
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+
+## 📊 Benchmark Results: Ayur Param vs Baselines
+- [BhashaBench-Ayur benchmark]([https://github.com/BharatGen-IITB-TIH/BhashaBench](https://huggingface.co/datasets/bharatgenai/BhashaBench-Ayur))
+---
+
+## 1. Overall Performance
+
+### Similar Range Models
+| Model                | bba   | bba_English | bba_Hindi |
+|-----------------------|-------|-------------|-----------|
+| Llama-3.2-1B-Instruct | 26.41 | 26.77       | 25.82     |
+| Qwen2.5-3B-Instruct   | 32.68 | 35.22       | 28.46     |
+| granite-3.1-2b        | 31.10 | 33.39       | 27.30     |
+| Llama-3.2-3B-Instruct | 33.20 | 35.31       | 29.67     |
+| gemma-2-2b-it         | 28.40 | 29.38       | 26.79     |
+| **AyurParam**        | **39.97** | **41.12** | **38.04** |
+
+### Larger Models
+| Model                                   | bba   | bba_English | bba_Hindi |
+|-----------------------------------------|-------|-------------|-----------|
+| Indic-gemma-7B-Navarasa-2.0             | 35.13 | 37.12       | 31.83     |
+| Pangea-7B                               | 37.41 | 40.69       | 31.93     |
+| aya-23-8B                               | 31.97 | 33.84       | 28.87     |
+| gpt-oss-20b                             | 36.34 | 38.30       | 33.09     |
+| Llama-3.1-8B-Instruct                   | 34.76 | 36.86       | 31.26     |
+| gemma-2-27b-it                          | 37.99 | 40.45       | 33.89     |
+| Nemotron-4-Mini-Hindi-4B-Instruct       | 33.54 | 33.38       | 33.82     |
+| **AyurParam**        | **39.97** | **41.12** | **38.04** |
+
+---
+
+## 2. Question Difficulty
+
+### Similar Range Models
+| Difficulty | Llama-3.2-1B | Qwen2.5-3B | granite-3.1-2b | Llama-3.2-3B | gemma-2-2b-it | **AyurParam** |
+|------------|--------------|------------|----------------|--------------|---------------|----------------|
+| **Easy**   | 27.44        | 35.55      | 33.90          | 36.42        | 29.96         | **43.93**      |
+| **Medium** | 25.23        | 29.57      | 28.06          | 29.66        | 26.83         | **35.95**      |
+| **Hard**   | 25.39        | 28.23      | 26.81          | 28.51        | 24.96         | **31.21**      |
+
+### Larger Models
+| Difficulty | Indic-gemma-7B | Pangea-7B | aya-23-8B | gpt-oss-20b | Llama-3.1-8B | gemma-2-27b-it | Nemotron-4-Mini-Hindi-4B | **AyurParam** |
+|------------|----------------|-----------|-----------|-------------|--------------|----------------|--------------------------|----------------|
+| **Easy**   | 38.54          | 41.45     | 35.51     | 42.03       | 39.43        | 43.47          | 36.08                    |**43.93**      |
+| **Medium** | 31.72          | 32.94     | 28.29     | 30.27       | 29.36        | 31.90          | 30.80                    |**35.95**      |
+| **Hard**   | 27.23          | 31.77     | 25.11     | 26.67       | 30.50        | 30.78          | 29.50                    |**31.21**      |
+
+---
+
+## 3. Question Type
+
+### Similar Range Models
+| Type                 | Llama-3.2-1B | Qwen2.5-3B | granite-3.1-2b | Llama-3.2-3B | gemma-2-2b-it | **AyurParam** |
+|----------------------|--------------|------------|----------------|--------------|---------------|----------------|
+| Assertion/Reasoning  | 59.26        | 51.85      | 33.33          | 40.74        | 33.33         | **44.44**      |
+| Fill in the blanks   | 26.97        | 29.21      | 21.35          | 34.83        | 32.02         | **29.78**      |
+| MCQ                  | 26.34        | 32.70      | 31.22          | 33.17        | 28.33         | **40.12**      |
+| Match the column     | 26.83        | 29.27      | 29.27          | 29.27        | 36.59         | **24.39**      |
+
+### Larger Models
+| Type                 | Indic-gemma-7B | Pangea-7B | aya-23-8B | gpt-oss-20b | Llama-3.1-8B | gemma-2-27b-it | Nemotron-4-Mini-Hindi-4B | **AyurParam** |
+|----------------------|----------------|-----------|-----------|-------------|--------------|----------------|--------------------------|----------------|
+| Assertion/Reasoning  | 59.26          | 62.96     | 18.52     | 25.93       | 29.63        | 55.56          | 37.04                    | **44.44**      |
+| Fill in the blanks   | 35.39          | 24.16     | 30.90     | 32.02       | 26.97        | 35.96          | 30.34                    | **29.78**      |
+| MCQ                  | 35.10          | 37.53     | 32.05     | 36.39       | 34.83        | 37.98          | 33.60                    | **40.12**      |
+| Match the column     | 31.71          | 34.15     | 17.07     | 46.34       | 46.34        | 39.02          | 24.39                    | **24.39**      |
+
+---
+From the above results, **Ayur Param not only outperforms all similar-sized models** but also achieves **competitive or better performance than larger models** across multiple metrics.
+
+
+## License
+This dataset is released under the [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).
+
+## Contact
+For any questions or feedback, please contact:
+- Sravan Kumar ([email protected])
+- Kundeshwar Pundalik ([email protected])