It's long been my view that LMArena isn't a fully reliable measure of real-world LLM performance. I suspect many users might click somewhat randomly, perhaps favoring answers based on superficial qualities like length, formatting, or speed, rather than deeper assessment.

Since all the Arena dialogues are publicly available on Hugging Face, a crowdsourced evaluation system utilizing that data seems like it could be quite valuable. It would also be interesting to see more development in automated evaluation systems, perhaps along the lines of "Arena-Hard-Auto" (though keeping such systems updated and robust is a challenge). However, building an effective automated evaluator would likely require training a specialized model on a large corpus, because I'm fairly certain that using a current powerful model like GPT-4-Turbo (or any other) for evaluation would introduce bias, favoring responses that align with its own style.

Where do u have that info about qwen 3 from? I'm really curious now...