Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions

Promoting Our Recent Work!

This article essentially focuses on how to overcome the inherent limitations of reinforcement learning (RL) and develop new training paradigms. We present an initial exploration in this work and will continue to advance this line of research. All code and models have been fully open-sourced!

In recent years, large language models (LLMs) have made significant progress in reasoning abilities, largely thanks to RLHF (Reinforcement Learning from Human Feedback). However, existing RL methods are essentially “in-distribution optimizers”—they mainly improve model performance on problems within the scope of existing knowledge, making it difficult to surpass the capability ceiling of the base model. As a result, RL struggles to facilitate the acquisition of new knowledge and the development of higher-order reasoning skills.

Supervised Fine-Tuning (SFT) is widely used in LLMs to introduce new knowledge and reasoning patterns through high-quality demonstration data. SFT is particularly effective at improving model performance on problems beyond its original capabilities, especially for smaller models. However, SFT heavily relies on high-quality demonstration data and generally underperforms RL in terms of out-of-distribution (OOD) generalization. The respective strengths and weaknesses of RL and SFT inspire an important research direction: how to effectively combine the two approaches to enhance both reasoning and generalization abilities, while reducing dependence on expensive demonstration data, thereby breaking through existing cognitive bottlenecks.

In this work, we systematically analyze the dynamic behaviors of RL and SFT during training. Our experiments show that RL excels at consolidating and improving performance within the model’s existing capability range, while SFT is more effective at driving progress on more challenging problems. Notably, SFT may lead to performance degradation on simple problems and tends to generate more verbose answers, whereas RL offers limited improvement on difficult problems. Based on these findings, we propose a new training method—ReLIFT (Reinforcement Learning Interleaved with Online Fine-Tuning). During RL training, ReLIFT dynamically collects hard problems that the model struggles with, obtains high-quality chain-of-thought (CoT) demonstrations for these cases, and alternates between RL and SFT training to fully leverage their complementary strengths.

Experiments on five mathematical benchmarks and one out-of-distribution benchmark show that ReLIFT achieves a new SOTA accuracy of 51.1% on the Qwen2.5-Math-7B model, a 5.2 percentage point improvement over the strongest zero-RL baseline. Moreover, ReLIFT requires only 13% of the detailed demonstration data to outperform pure RL and pure SFT methods, and it significantly reduces the length of generated answers (about one-tenth that of SFT), greatly improving reasoning efficiency and practicality. Additionally, ReLIFT demonstrates superior generalization and stability even on smaller and weaker base models.

In summary, ReLIFT effectively overcomes the fundamental limitations of RL, offering efficiency, scalability, and strong generalization. It provides new insights and evidence for the continued advancement of reasoning abilities in large language models.

Our ultimate goal is to design a data engine where data annotation and model optimization proceed simultaneously, continuously pushing the boundaries of model capabilities!