Malaysian Qwen 2.5 14B Instruct

Continue finetuning https://huggingface.co/Qwen/Qwen2.5-14B-Instruct on highly curated 1.5B tokens Malaysian instruction dataset.

We provide 2 different revisions,

Rank 128, Alpha 256, 1b271d6112b14efc349a4d8c7f4589cbe76384a7
Rank 256, Alpha 512, 889ae31abda87cbf080e722677d67e43fd6b295a

Improvement

Support respond in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
Able to code in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
Multi-turn Malaysian context such as related to Malaysian Legislation, politics, religions and languages.

Training session

Finetune on mesolitica/Malaysian-SFT to make the model understand Malaysian context.

How we train

LoRA on ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", "lm_head"].
Multipacking 8192 context length with proper SDPA causal masking to prevent document contamination and also make sure proper position ids.
Chunk CCE loss for LoRA.

Revision 1b271d6112b14efc349a4d8c7f4589cbe76384a7

Rank 128, Alpha 256.
WanDB at https://wandb.ai/huseinzol05/lora-embedding-128-qwen2.5-14b-malaysian-8k

Source code at https://github.com/mesolitica/malaya/tree/master/session/qwen2.5

Revision 889ae31abda87cbf080e722677d67e43fd6b295a

Rank 256, Alpha 512.
WanDB at https://wandb.ai/huseinzol05/lora-embedding-256-qwen2.5-14b-malaysian-8k

Source code at https://github.com/mesolitica/malaya/tree/master/session/qwen2.5

Benchmark

MalayMMLU

Probability next tokens

Based on 0-shot official MalayMMLU First token accuracy,

Revision 1b271d6112b14efc349a4d8c7f4589cbe76384a7,

                            Model   Accuracy   shot by_letter        category
0  Malaysian-Qwen2.5-14B-Instruct  74.785100  0shot      True            STEM
1  Malaysian-Qwen2.5-14B-Instruct  74.777354  0shot      True        Language
2  Malaysian-Qwen2.5-14B-Instruct  69.326395  0shot      True  Social science
3  Malaysian-Qwen2.5-14B-Instruct  67.618134  0shot      True          Others
4  Malaysian-Qwen2.5-14B-Instruct  73.265074  0shot      True      Humanities
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
Model : Malaysian-Qwen2.5-14B-Instruct
Metric : first
Shot : 0shot
average accuracy 71.71354231198117
accuracy for STEM 74.78510028653295
accuracy for Language 74.77735368956743
accuracy for Social science 69.32639491182422
accuracy for Others 67.61813384504677
accuracy for Humanities 73.2650739476678

Revision 889ae31abda87cbf080e722677d67e43fd6b295a,

While the original model,

                  Model   Accuracy   shot by_letter        category
0  Qwen2.5-14B-Instruct  73.311502  0shot      True            STEM
1  Qwen2.5-14B-Instruct  72.773537  0shot      True        Language
2  Qwen2.5-14B-Instruct  67.505059  0shot      True  Social science
3  Qwen2.5-14B-Instruct  65.819141  0shot      True          Others
4  Qwen2.5-14B-Instruct  70.557452  0shot      True      Humanities
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
Model : Qwen2.5-14B-Instruct
Metric : first
Shot : 0shot
average accuracy 69.72287614091604
accuracy for STEM 73.31150225133032
accuracy for Language 72.7735368956743
accuracy for Social science 67.50505926568373
accuracy for Others 65.8191412808827
accuracy for Humanities 70.55745164960182

First token match using vLLM

Based on 0-shot exact first token match using vLLM Guided Decoding,