README.md

---
license: apache-2.0
base_model: Qwen/Qwen2.5-3B-Instruct
tags:
- qwen2.5
- ppo
- rlhf
- metamath
- math
- reasoning
- verl
pipeline_tag: text-generation
---

# Qwen2.5-3B-UFO-1turn

This model is based on **Qwen2.5-3B-Instruct** and trained with **PPO (Proximal Policy Optimization)** on the **MetaMathQA** dataset for mathematical reasoning.

Github: https://github.com/lichengliu03/unary-feedback

Website: https://unary-feedback.github.io/

## Model Info

- **Base model**: Qwen/Qwen2.5-3B-Instruct
- **Training method**: PPO (full-parameter fine-tuning, not LoRA)
- **Training data**: MATH_MetaMathQA
- **Training steps**: 200 steps
- **Framework**: VERL
- **Tensor parallel**: 2x GPU distributed training
- **Model size**: ~6GB

## Training Config

- **Micro Batch Size**: 1 per GPU
- **PPO Mini Batch Size**: 8
- **Actor Learning Rate**: auto
- **Critic Learning Rate**: auto
- **KL Penalty**: 0.001
- **Clip Ratio**: 0.2-0.28
- **Temperature**: 1.0 (train), 0.5 (eval)

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("LichengLiu03/qwen2.5-3b-ppo-metamath-full")
model = AutoModelForCausalLM.from_pretrained(
    "LichengLiu03/qwen2.5-3b-ppo-metamath-full",
    torch_dtype=torch.float16,
    device_map="auto"
)

# Example math problem
prompt = "Solve this math problem: If a circle has a radius of 5cm, what is its area?"
inputs = tokenizer(prompt, return_tensors="pt")

# Generate answer
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.7,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```

## Features

This model is optimized for mathematical reasoning with PPO, and compared to the base model, it improves:

- ✅ Math problem understanding
- ✅ Logical reasoning accuracy
- ✅ Clarity of solution steps
- ✅ Calculation accuracy

## Technical Details

- **Tensor parallel training**: 2 GPUs, distributed
- **Memory optimization**: gradient checkpointing and mixed precision
- **Reward modeling**: based on MetaMathQA correctness and reasoning quality
- **Policy optimization**: PPO for stable training

## Limitations

- Mainly optimized for mathematical reasoning
- May not perform as well on general tasks
- Recommended for math, logic, and reasoning tasks

## License

This model is licensed under Apache 2.0.