mesolitica/Malaysian-Qwen2.5-7B-Instruct

Malaysian Qwen 2.5 7B Instruct

Continue finetuning https://huggingface.co/Qwen/Qwen2.5-7B-Instruct on highly curated 1.5B tokens Malaysian instruction dataset.

Improvement

1. Support respond in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
2. Able to code in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
3. Multi-turn Malaysian context such as related to Malaysian Legislation, politics, religions and languages.

Training session

Finetune on mesolitica/Malaysian-SFT to make the model understand Malaysian context.

How we train

1. LoRA on ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", "lm_head"].
2. 128 Rank with alpha 256, or alpha of 2.0
3. Multipacking 8192 context length with proper SDPA causal masking to prevent document contamination and also make sure proper position ids.
4. Chunk CCE loss for LoRA.
5. WanDB at https://wandb.ai/huseinzol05/lora-embedding-128-qwen2.5-7b-malaysian-8k

Source code at https://github.com/mesolitica/malaya/tree/master/session/qwen2.5

Benchmark

MalayMMLU

Probability next tokens

Based on 0-shot official MalayMMLU First token accuracy,

                          Model   Accuracy   shot by_letter        category
0  Malaysian-Qwen2.5-7B-Instruct  72.042571  0shot      True            STEM
1  Malaysian-Qwen2.5-7B-Instruct  70.690204  0shot      True        Language
2  Malaysian-Qwen2.5-7B-Instruct  66.536571  0shot      True  Social science
3  Malaysian-Qwen2.5-7B-Instruct  67.306308  0shot      True          Others
4  Malaysian-Qwen2.5-7B-Instruct  71.808874  0shot      True      Humanities
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
Model : Malaysian-Qwen2.5-7B-Instruct
Metric : first
Shot : 0shot
average accuracy 69.26031470697559
accuracy for STEM 72.04257060990585
accuracy for Language 70.69020356234097
accuracy for Social science 66.53657126337092
accuracy for Others 67.30630846725833
accuracy for Humanities 71.80887372013652

While the original model,

                 Model   Accuracy   shot by_letter        category
0  Qwen2.5-7B-Instruct  70.609906  0shot      True            STEM
1  Qwen2.5-7B-Instruct  68.034351  0shot      True        Language
2  Qwen2.5-7B-Instruct  63.486557  0shot      True  Social science
3  Qwen2.5-7B-Instruct  64.164068  0shot      True          Others
4  Qwen2.5-7B-Instruct  69.101251  0shot      True      Humanities
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
Model : Qwen2.5-7B-Instruct
Metric : first
Shot : 0shot
average accuracy 66.52211621856027
accuracy for STEM 70.60990585345887
accuracy for Language 68.03435114503816
accuracy for Social science 63.486556808326114
accuracy for Others 64.16406812185176
accuracy for Humanities 69.10125142207053

First token match using vLLM

Based on 0-shot exact first token match using vLLM Guided Decoding,

                           Model   Accuracy  shot        category
0  Malaysian-Qwen2.5-7B-Instruct  70.159640     0            STEM
1  Malaysian-Qwen2.5-7B-Instruct  66.682570     0        Language
2  Malaysian-Qwen2.5-7B-Instruct  62.893900     0  Social science
3  Malaysian-Qwen2.5-7B-Instruct  64.379947     0          Others
4  Malaysian-Qwen2.5-7B-Instruct  66.780432     0      Humanities
Model : Malaysian-Qwen2.5-7B-Instruct
Metric : full
Shot : 0
average accuracy 65.57221327386115
accuracy for STEM 70.15963978714696
accuracy for Language 66.68256997455471
accuracy for Social science 62.89389997108991
accuracy for Others 64.37994722955145
accuracy for Humanities 66.78043230944255

While the original model,

                 Model   Accuracy  shot        category
0  Qwen2.5-7B-Instruct  70.978305     0            STEM
1  Qwen2.5-7B-Instruct  68.177481     0        Language
2  Qwen2.5-7B-Instruct  64.238219     0  Social science
3  Qwen2.5-7B-Instruct  64.643799     0          Others
4  Qwen2.5-7B-Instruct  70.443686     0      Humanities
Model : Qwen2.5-7B-Instruct
Metric : full
Shot : 0
average accuracy 67.13748812621319
accuracy for STEM 70.97830536225952
accuracy for Language 68.17748091603053
accuracy for Social science 64.23821913847932
accuracy for Others 64.64379947229551
accuracy for Humanities 70.44368600682593

Acknowledgement

Special thanks to https://www.sns.com.my for 8x H100 node!