mesolitica/Malaysian-Qwen2.5-14B-Instruct

Malaysian Qwen 2.5 14B Instruct

Continue finetuning https://huggingface.co/Qwen/Qwen2.5-14B-Instruct on highly curated 1.5B tokens Malaysian instruction dataset.

Improvement

1. Support respond in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
2. Able to code in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
3. Multi-turn Malaysian context such as related to Malaysian Legislation, politics, religions and languages.

Training session

Finetune on mesolitica/Malaysian-SFT to make the model understand Malaysian context.

How we train

1. LoRA on ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", "lm_head"].
2. 128 Rank with alpha 256, or alpha of 2.0
3. Multipacking 8192 context length with proper SDPA causal masking to prevent document contamination and also make sure proper position ids.
4. Chunk CCE loss for LoRA.
5. WanDB at https://wandb.ai/huseinzol05/lora-embedding-128-qwen2.5-14b-malaysian-8k?nw=nwuserhuseinzol05

Source code at https://github.com/mesolitica/malaya/tree/master/session/qwen2.5

Benchmark

MalayMMLU

Probability next tokens

Based on 0-shot official MalayMMLU First token accuracy,

                            Model   Accuracy   shot by_letter        category
0  Malaysian-Qwen2.5-14B-Instruct  74.785100  0shot      True            STEM
1  Malaysian-Qwen2.5-14B-Instruct  74.777354  0shot      True        Language
2  Malaysian-Qwen2.5-14B-Instruct  69.326395  0shot      True  Social science
3  Malaysian-Qwen2.5-14B-Instruct  67.618134  0shot      True          Others
4  Malaysian-Qwen2.5-14B-Instruct  73.265074  0shot      True      Humanities
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
Model : Malaysian-Qwen2.5-14B-Instruct
Metric : first
Shot : 0shot
average accuracy 71.71354231198117
accuracy for STEM 74.78510028653295
accuracy for Language 74.77735368956743
accuracy for Social science 69.32639491182422
accuracy for Others 67.61813384504677
accuracy for Humanities 73.2650739476678

While the original model,

                  Model   Accuracy   shot by_letter        category
0  Qwen2.5-14B-Instruct  73.311502  0shot      True            STEM
1  Qwen2.5-14B-Instruct  72.773537  0shot      True        Language
2  Qwen2.5-14B-Instruct  67.505059  0shot      True  Social science
3  Qwen2.5-14B-Instruct  65.819141  0shot      True          Others
4  Qwen2.5-14B-Instruct  70.557452  0shot      True      Humanities
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
Model : Qwen2.5-14B-Instruct
Metric : first
Shot : 0shot
average accuracy 69.72287614091604
accuracy for STEM 73.31150225133032
accuracy for Language 72.7735368956743
accuracy for Social science 67.50505926568373
accuracy for Others 65.8191412808827
accuracy for Humanities 70.55745164960182

First token match using vLLM

Based on 0-shot exact first token match using vLLM Guided Decoding,

                            Model   Accuracy  shot        category
0  Malaysian-Qwen2.5-14B-Instruct  72.656570     0            STEM
1  Malaysian-Qwen2.5-14B-Instruct  71.278626     0        Language
2  Malaysian-Qwen2.5-14B-Instruct  66.551026     0  Social science
3  Malaysian-Qwen2.5-14B-Instruct  64.403934     0          Others
4  Malaysian-Qwen2.5-14B-Instruct  70.853242     0      Humanities
Model : Malaysian-Qwen2.5-14B-Instruct
Metric : full
Shot : 0
average accuracy 68.80601329864123
accuracy for STEM 72.65656979124027
accuracy for Language 71.27862595419847
accuracy for Social science 66.55102630818156
accuracy for Others 64.40393379707365
accuracy for Humanities 70.8532423208191

While the original model,

                  Model   Accuracy  shot        category
0  Qwen2.5-14B-Instruct  74.580434     0            STEM
1  Qwen2.5-14B-Instruct  72.694020     0        Language
2  Qwen2.5-14B-Instruct  68.141081     0  Social science
3  Qwen2.5-14B-Instruct  66.562725     0          Others
4  Qwen2.5-14B-Instruct  70.739477     0      Humanities
Model : Qwen2.5-14B-Instruct
Metric : full
Shot : 0
average accuracy 70.17304753644736
accuracy for STEM 74.58043389275481
accuracy for Language 72.6940203562341
accuracy for Social science 68.14108123735183
accuracy for Others 66.56272487407053
accuracy for Humanities 70.73947667804323

Acknowledgement

Special thanks to https://www.sns.com.my for 8x H100 node!