|
|
--- |
|
|
license: apache-2.0 |
|
|
--- |
|
|
|
|
|
|
|
|
<p align="center"> |
|
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/623d8ca4c29adf5ef6175615/q3Anm7o-MoNYjB8JztGVT.png" width="60%" /> |
|
|
</p> |
|
|
|
|
|
<font size=3><div align='center' > |
|
|
[[📖 arXiv Paper](https://arxiv.org/abs/2502.10391)] |
|
|
[[📊 R1-Reward Code](https://github.com/yfzhang114/r1_reward)] |
|
|
[[📝 R1-Reward Data](https://huggingface.co/datasets/yifanzhang114/R1-Reward-RL)] |
|
|
</div></font> |
|
|
|
|
|
# Training Multimodal Reward Model Through Stable Reinforcement Learning |
|
|
|
|
|
🔥 We are proud to open-source **R1-Reward**, a comprehensive project for improve reward modeling through reinforcement learning. This release includes: |
|
|
|
|
|
* **R1-Reward Model:** A state-of-the-art (SOTA) multimodal reward model demonstrating substantial gains (Voting@15): |
|
|
* **13.5%** improvement on VL Reward-Bench. |
|
|
* **3.5%** improvement on MM-RLHF Reward-Bench. |
|
|
* **14.6%** improvement on Multimodal Reward Bench. |
|
|
* **StableReinforce Algorithm:** A novel reinforcement learning method that enhances the Reinforce++ approach by improving training loss stability, advantage estimation, and reward function design. |
|
|
* **Open-Source Resources:** We provide the R1-Reward model, the R1-Reward RL training dataset, and inference code for IXC-Reward，MM-RLHF Reward and R1-Reward on the three benchmarks in Figure 1. |
|
|
|
|
|
|
|
|
 |
|
|
|
|
|
|
|
|
## Citation |
|
|
|
|
|
If you find it useful for your research and applications, please cite related papers/blogs using this BibTeX: |
|
|
```bibtex |
|
|
@article{zhang2025r1, |
|
|
title={R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning}, |
|
|
author={Zhang, Yi-Fan and Lu, Xingyu and Hu, Xiao and Fu, Chaoyou and Wen, Bin and Zhang, Tianke and Liu, Changyi and Jiang, Kaiyu and Chen, Kaibing and Tang, Kaiyu and others}, |
|
|
journal={arXiv preprint arXiv:2505.02835}, |
|
|
year={2025} |
|
|
} |
|
|
``` |
|
|
|
|
|
## Related Projects |
|
|
- [MM-RLHF: The Next Step Forward in Multimodal LLM Alignment](https://mm-rlhf.github.io/) |
|
|
- [MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?](https://github.com/yfzhang114/MME-RealWorld) |
|
|
- [MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs](https://arxiv.org/abs/2411.15296) |
|
|
- [Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models](https://github.com/yfzhang114/SliME) |
|
|
- [VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction](https://github.com/VITA-MLLM/VITA) |
|
|
|
|
|
|
