Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts
Paper
•
2504.21117
•
Published
•
25
Jen
bird-of-paradise
AI & ML interests
My research interests focus on transformer architecture and RL for reasoning tasks. I'm passionate about ML and committed to contributing to advancements in the field through both technical work and community engagement.
Recent Activity
updated
a dataset
3 months ago
bird-of-paradise/transformer-from-scratch-tutorial
updated
a model
3 months ago
bird-of-paradise/deepseek-mla
Organizations
None yet
bird-of-paradise's activity
upvoted
a
paper
18 days ago
updated
a
dataset
3 months ago
updated
2
models
3 months ago
published
2
models
3 months ago
Potential bug in the tutorial
3
#1 opened 4 months ago
by
wkokng
reacted to
lewtun's
post with 🔥
5 months ago
Post
2342
This paper (
HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs (2412.18925)) has a really interesting recipe for inducing o1-like behaviour in Llama models:
* Iteratively sample CoTs from the model, using a mix of different search strategies. This gives you something like Stream of Search via prompting.
* Verify correctness of each CoT using GPT-4o (needed because exact match doesn't work well in medicine where there are lots of aliases)
* Use GPT-4o to reformat the concatenated CoTs into a single stream that includes smooth transitions like "hmm, wait" etc that one sees in o1
* Use the resulting data for SFT & RL
* Use sparse rewards from GPT-4o to guide RL training. They find RL gives an average ~3 point boost across medical benchmarks and SFT on this data already gives a strong improvement.
Applying this strategy to other domains could be quite promising, provided the training data can be formulated with verifiable problems!
* Iteratively sample CoTs from the model, using a mix of different search strategies. This gives you something like Stream of Search via prompting.
* Verify correctness of each CoT using GPT-4o (needed because exact match doesn't work well in medicine where there are lots of aliases)
* Use GPT-4o to reformat the concatenated CoTs into a single stream that includes smooth transitions like "hmm, wait" etc that one sees in o1
* Use the resulting data for SFT & RL
* Use sparse rewards from GPT-4o to guide RL training. They find RL gives an average ~3 point boost across medical benchmarks and SFT on this data already gives a strong improvement.
Applying this strategy to other domains could be quite promising, provided the training data can be formulated with verifiable problems!
- 1 reply
liked
a
Space
5 months ago