Dataset and reward models for "On the Robustness of Reward Models for Language Model Alignment (ICML 2025)"