Update README.md

README.md

@@ -1,3 +1,66 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+tags:
+- moe
+- llm
+- efficient-inference
+pipeline_tag: text-generation
+---
+
+# TC-MoE: Augmenting Mixture of Experts with Ternary Expert Choice
+
+## Model Description
+
+TC-MoE is a novel Mixture-of-Experts (MoE) architecture that enhances traditional MoE models through expert space expansion. By applying the ternary set {-1, 0, 1} to each original expert, TC-MoE achieves:
+
+- ​**9% reduction** in activated experts compared to Top-K routing
+- ​**1.1% average performance gain** on language understanding benchmarks
+- Flexible efficiency-effectiveness trade-off via reward mechanism
+
+Key innovations:
+- 🎯 ​**Ternary Expert Expansion**: Creates parameter-sharing expert variants (-1, 0, +1) without significant computational overhead
+- ⚖️ ​**Adaptive Load Balancing**: Novel load balance loss for expert workload distribution
+- 🎮 ​**Reward-Driven Routing**: Dynamic control of expert activation ratios
+
+## Model Overview
+
+- ​**Architecture**: Decoder-only transformer based on LLaMA
+- ​**Pretraining Data**: 
+  - RedPajama (100B tokens)
+- ​**Model Size**:
+  - Base (681M/2.3B params) 
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("stiger1000/TC-MoE")
+tokenizer = AutoTokenizer.from_pretrained("stiger1000/TC-MoE")
+
+inputs = tokenizer("The capital of France is", return_tensors="pt")
+outputs = model.generate(**inputs, max_length=50)
+print(tokenizer.decode(outputs[0]))
+```
+
+## Training Details
+
+- **Optimizer**: AdamW (β₁=0.9, β₂=0.95)
+- **Learning Rate**: 1e-4 with cosine decay
+- **Batch Size**: 4M tokens
+- **Loss Components**:
+  - Language Modeling Loss
+  - Load Balance Loss (α₁=0.01)
+  - Reward Loss (α₂=0.0)
+
+## Citation
+```bibtex
+@inproceedings{yan2025tcmoe,
+  title={TC-MoE: Augmenting Mixture of Experts with Ternary Expert Choice},
+  author={Yan, Shen and Bin, Xingyan and Zhang, Sijun and Wang, Yisen and Lin, Zhouchen},
+  booktitle={The Thirteenth International Conference on Learning Representations},
+  year={2025}
+}
+```
+
+📚 **Repository**: [GitHub](https://github.com/stiger1000/TC-MoE)