Improve model card: Add detailed framework and results sections (#2)

- Improve model card: Add detailed framework and results sections (184401420a7eb6095c5d50a1f609cb9ed46b9171)


Co-authored-by: Niels Rogge <[email protected]>

README.md

@@ -1,5 +1,6 @@
 ---
 base_model: Qwen/Qwen2.5-3B-Instruct
+library_name: transformers
 license: apache-2.0
 pipeline_tag: text-generation
 tags:
@@ -11,7 +12,6 @@ tags:
 - reasoning
 - verl
 paper: https://huggingface.co/papers/2507.14295
-library_name: transformers
 ---
 
 # Qwen2.5-3B-UFO
@@ -19,9 +19,30 @@ library_name: transformers
 This model is based on **Qwen2.5-3B-Instruct** and trained with **PPO (Proximal Policy Optimization)** on the **MetaMathQA** dataset for mathematical reasoning, as presented in the paper [A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning](https://huggingface.co/papers/2507.14295).
 
 Github: https://github.com/lichengliu03/unary-feedback
-
 Website: https://unary-feedback.github.io/
 
+## Overview
+
+**"Let's Try Again"** addresses a critical gap in language model training: while single-turn reinforcement learning (RL) improves reasoning, these models fail in **multi-turn interactive scenarios**, often repeating the same wrong answers despite feedback.
+
+### Key Problem
+Single-turn RL models lose the ability to revise reasoning across multiple turns. In 70% of failure cases, they produce identical answers across 5 interaction rounds, unable to incorporate simple feedback like "try again."
+
+### Solution: UFO Framework
+**Unary Feedback as Observation (UFO)** transforms static datasets into multi-turn training by:
+- Using only minimal feedback signals ("Try Again")
+- Treating failure feedback as part of the observation
+- Enabling models to learn from historical mistakes
+
+### Results
+- **14% improvement** in multi-turn success rates
+- **10% reduction** in average interaction turns
+- Better performance even in single-turn scenarios
+- **90% non-repetitive answers** (vs 80% baseline)
+
+### Impact
+UFO enables effective multi-turn RL training on existing static datasets without expensive annotations, making it practical to train models that can learn from sparse feedback and improve iteratively through trial-and-error, just like humans do.
+
 ## Model Info
 
 - **Base model**: Qwen/Qwen2.5-3B-Instruct
@@ -42,6 +63,66 @@ Website: https://unary-feedback.github.io/
 - **Clip Ratio**: 0.2-0.28
 - **Temperature**: 1.0 (train), 0.5 (eval)
 
+## UFO Framework Details
+
+The UFO framework transforms static single-turn datasets into multi-turn interactive training through a simple yet effective approach.
+
+<p align="center"><img src="public/fig1.png" width="800px" alt="UFO Framework Flow" /></p>
+<p align="center" style="font-size: 16px; max-width: 800px; margin: 0 auto;">
+The UFO framework flow: Static datasets are transformed into multi-turn episodes where models receive minimal feedback ("Try Again") and learn to revise their reasoning across multiple attempts.
+</p>
+
+### Problem Formulation
+
+We model multi-turn problem solving as a finite-horizon Markov Decision Process (MDP) where:
+- **State**: Encodes the original question and history of past attempts with feedback
+- **Action**: All possible answers the model can generate
+- **Reward**: Binary signal (1 for correct, 0 for incorrect)
+- **Transition**: Agent generates answer, receives feedback, episode continues until success or max turns
+
+### Unary Feedback as Observation (UFO)
+
+The core innovation is treating minimal feedback as part of the observation:
+
+```
+Question: What is the value of x + y?
+Attempt 1: [wrong answer]
+Feedback: Try Again.
+Attempt 2: [correct answer]
+```
+
+**Key Features:**
+- Only **negative feedback** (e.g., "Try Again") is included in context
+- No positive confirmation signals are ever shown
+- Model must learn to revise based solely on failure history
+- Episodes terminate immediately upon correct answer
+
+### Training with PPO
+
+We use Proximal Policy Optimization (PPO) to train the policy:
+- Agent observes input with full interaction history
+- Generates answer and receives binary reward
+- Policy updates using clipped surrogate objective
+- Value function provides advantage estimates for stable training
+
+### Reward Design
+
+Two complementary strategies encourage efficient reasoning:
+
+**1. Exponential Reward Decay:**
+```
+DecayReward(t) = γ^t if correct, 0 otherwise
+```
+Favors solving problems in fewer turns.
+
+**2. Repetition Penalty:**
+```
+Penalty(τ) = λ · (1 - E(τ)/T)
+```
+Penalizes duplicate answers, encouraging diverse reasoning strategies.
+
+This framework enables effective multi-turn RL training on static datasets without requiring expensive annotations or complex environments.
+
 ## Usage
 
 ```python
@@ -96,6 +177,52 @@ This model is optimized for mathematical reasoning with PPO, and compared to the
 - May not perform as well on general tasks
 - Recommended for math, logic, and reasoning tasks
 
+## Key Results
+
+### Multi-Turn Reasoning Performance
+
+We compare our multi-turn UFO model against a strong single-turn PPO baseline. For a fair comparison, the baseline is evaluated on 5 independent samples (Pass@5), while our model uses 5 sequential attempts with feedback (Succ@5). Success is recorded if any of the 5 responses is correct. We also analyze the impact of varying the maximum number of interaction turns at training.
+
+<p align="center">
+<img src="public/compare_baseline.png" width="46.2%" alt="UFO Performance Comparison" />
+<img src="public/multi-turn_training.png" width="45%" alt="Multi-turn Training Process" />
+</p>
+<p align="center" style="font-size: 14px; color: #666;">
+Left: Multi-turn (5-turn) RL significantly outperforms single-turn baseline. Right: Performance comparison with different training turns (1, 5, and 10).
+</p>
+
+**Key Findings:**
+- **+14% success rate** over single-turn PPO baseline
+- Benefits generalize to both multi-turn and single-turn inference
+- Best results with 5-turn training; more turns yield diminishing returns
+
+### Effectiveness of Unary Feedback
+
+To further investigate the role of unary feedback, we compare model performance under different feedback availability conditions. In scenario (a), unary feedback is provided during both training and validation phases, while in scenario (b), unary feedback is available only during training but not at validation. The results show that access to unary feedback during both phases substantially improves validation success rate. In contrast, providing unary feedback solely during training does not yield improvements, indicating that the benefit of unary feedback is contingent on its availability at inference time.
+
+<p align="center"><img src="public/feedback_comparisons_side_by_side.png" width="80%" alt="Effectiveness of Unary Feedback" /></p>
+<p align="center" style="font-size: 14px; color: #666;">
+Success rate comparison under different unary feedback settings: (a) feedback in both training and validation; (b) feedback only in training.
+</p>
+
+**Key Insights:**
+- Feedback in both training and validation is crucial for improvement
+- Feedback only in training phase does **not** help at inference
+
+### Reward Design Impact
+
+**Exponential Reward Decay:**
+- Decreases the average number of actions required to solve problems by ~10%
+- Encourages faster and more efficient problem solving
+
+**Answer Diversity:**
+- Non-repetitive answer ratio increases from 79.7% to 92.8%
+- Multi-turn RL with UFO encourages answer diversity and strengthens robustness
+
 ## License
 
-This model is licensed under Apache 2.0.
+This model is licensed under Apache 2.0.
+
+## Acknowledgements
+
+We thank the [DeepSeek](https://github.com/deepseek-ai/DeepSeek-R1) team for providing the DeepSeek-R1 model and early conceptual inspirations. We are grateful to the [veRL](https://github.com/volcengine/verl) team for their infrastructure support and the [RAGEN](https://github.com/RAGEN-AI/RAGEN) team for their multi-turn RL framework.