Kenneth Hamilton PRO

Benchmarks nowadays focus on accuracy. It would be great if we could factor in token cost, i.e. delivering the right answer with the fewest tokens. This would motivate the training to be inference efficient.

I used to complain that models don't bother to think if a problem is worthy of reasoning, and push the burden to users. We should do better on this.

Whoa. Good point.

I guess the short answer is that they handle subjective questions better and they improve model output traceability (i.e., better understanding of what informed the model's response).

Agree with your general thoughts on reasoning models though, they aren't the best solution for every use case.