---
license: apache-2.0
datasets:
- liuhaotian/LLaVA-Instruct-150K
language:
- en
base_model:
- microsoft/Phi-3.5-mini-instruct
pipeline_tag: text-generation
---

🎉 CompeteSMoE-5.1B

CompeteSMoE-5.1B is a lightweight and integrated variant of the Mixture-of-Experts (MoE) architecture, built upon the Phi-3.5 Mini and SigLIP baselines. This version incorporates the latest CompeteSMoE algorithm enhancements. CompeteSMoE-5.1B demonstrates strong performance across a range of MoE routing strategies, including both standard and star-to-art routing methods. It achieves competitive results compared to recent MoE architectures, such as SharedE-V2 and SharedE-V3, which are inspired by DeepSeek. Despite the architectural innovations of these models especially their use of shared experts CompeteSMoE-5.1B consistently delivers superior or comparable results.

📝 Note: This version of CompeteSMoE-5.1B was trained on a small-scale dataset. 🚧 We're actively working on a stronger, more robust release — coming soon! 🚀 Stay tuned for updates. 💡

### Hardware Resources

| Stage             | MoE Method           | Hardware  |
|-------------------|----------------------|-----------|
| Pre-Training      |                      | 4xH100    |
| Pre-FineTuning    |                      | 4xH100    |
| VIT               | CompeteSMoE               | 4xH100    |

--- 

### Citation Information
More details can be found in our paper.

If you use CompeteSMoE, please cite it using this BibTeX:

```
@misc{nguyen2025competesmoe,
    title={CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via Competition},
    author={Nam V. Nguyen and Huy Nguyen and Quang Pham and Van Nguyen and Savitha Ramasamy and Nhat Ho},
    year={2025},
    eprint={2505.13380},
    archivePrefix={arXiv},
    primaryClass={cs.AI}
}
```