Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
Abstract
Waypoint-OPD improves policy-shift distillation by gating endpoint shifts using directional agreement between early and late teacher policy changes, boosting reasoning performance with minimal overhead.
Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories. However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training. The same endpoint shift can result from early and late shifts that reinforce or partially cancel each other. We investigate whether agreement between early and late teacher policy shifts can guide selective transfer. We introduce Waypoint-guided On-Policy Distillation (Waypoint-OPD), which uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes. We gate the endpoint shift according to the directional agreement between the early and late shifts, suppressing it when they point in opposing directions. Compared with endpoint-only transfer, our method requires only one additional frozen-teacher scoring pass. Using RL-trained Qwen3-4B teachers, we evaluate our method with Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B as student models in mathematical and code reasoning. In all six domain--student settings, Waypoint-OPD achieves the highest domain-average score among the compared distillation methods. Ablations with the Qwen3-8B student show that removing or shuffling gates reduces mathematical reasoning performance. These results support the use of directional agreement between early and late teacher policy shifts to weight the endpoint shift during student training.
Community
We introduce Waypoint-guided On-Policy Distillation (Waypoint-OPD), which uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes.
This is an automated message from the ResearchStudio team.
We created an interactive ResearchStudio Reel for this paper. It includes a visual poster, a video, and a blog, all available for download in editable formats.
Open the ResearchStudio Reel →
Download all files from Hugging Face
Please give this comment a thumbs up if you find the Reel helpful!
Want to explore or create Reels for more papers? Visit the ResearchStudio demo.
hf papers read 2610.21601
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Reward-Aligned Reweighting for On-Policy Distillation (2026)
- Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shifts (2026)
- Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards (2026)
- The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation (2026)
- Beyond Prompt Count: How Data Shapes Transfer in On-Policy Distillation (2026)
- Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation (2026)
- What Shared Prefixes Hide: Trajectory Dropout for On-Policy Distillation (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Nice result that the waypoint position barely matters (steps 500/600/700 give 56.18/56.14/56.13 on Qwen3-8B math), and that shuffled gates (51.20) end up below no gate at all (53.50). That makes the per-prefix agreement look like a real signal, not just a reweighting effect. One thing I couldn't find: since AIME 2024 is also the checkpoint-selection set, how do the math averages compare with AIME24 left out? And did you try a much earlier waypoint, or several waypoints, or is a 1200-step run too smooth for that to change anything?
Thanks for the great questions!
- Excluding AIME 2024: All detailed results are reported in the paper. The gains remain clearly positive even without AIME 2024; we just didn't include the exact average for that subset.
- Earlier waypoints: We actually tested steps 100 and 200, but performance was noticeably worse. At such an early stage, the model likely hasn't developed enough capacity to provide reliable guidance.
- Multiple waypoints: Great point! We haven't tried this yet, but we definitely plan to explore it in future work.
Get this paper in your agent:
hf papers read 2610.21601 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper