Hi! As the model is multilingual, this is a PR to add other languages than English to the language tag to improve the referencing. Note that 29 languages are announced in the README, but only 13 are explicitly listed. I was therefore only able to add these 13 languages.

Hi! Thanks for the comment. I've been looking where in Qwen 2.5 paper where the languages are listed but I cant see to find it.

They are not visible in their paper but in their blog post: https://qwenlm.github.io/blog/qwen2.5/

Hi, noted on the PR (Saw the remarks). Do keep in mind this model is more fitted for English, so other languages will suffer.

I have a feeling that it isn't right to blanket open PRs to all qwen2.5 models being able to do a specific language without testing each of them.
Tldr, while I'm fine with the PR, I'm hesitant to say that it can support these languages... I'll merge it in along with the other 3

On a tangent, In my company, we have been working on a similar problem as well. When companies claim they support X languages, a number refuse to elaborate what they tested it on. There's in a sense, no multi-lingual benchmark that can really test all languages.

I'm purposefully excluding MTL'd benchmarked questions since those may not be representative of the ability of language comprehension. An Example would be:

I could mimic how to speak French for example and score really well in a French test, but when it comes to navigating the real world and interacting with strangers, my French sucks.

KaraKaraWitch changed pull request status to merged

Sign up or log in to comment