mesolitica/Malaysian-Llama-3.2-3B-Instruct

Malaysian Llama-3.2-3B-Instruct

Continue finetuning https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct on highly curated 1.5B tokens Malaysian instruction dataset.

Improvement

1. Support respond in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
2. Able to code in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
3. Multi-turn Malaysian context such as related to Malaysian Legislation, politics, religions and languages.

Training session

Finetune on mesolitica/Malaysian-SFT to make the model understand Malaysian context.

How we train

1. LoRA on ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", "lm_head"].
2. 128 Rank with alpha 256, or alpha of 2.0
3. Multipacking 8192 context length with proper SDPA causal masking to prevent document contamination and also make sure proper position ids.
4. Chunk CCE loss for LoRA.
5. WanDB at https://wandb.ai/huseinzol05/lora-embedding-128-llama3.2-3b-malaysian-8k?nw=nwuserhuseinzol05

Source code at https://github.com/mesolitica/malaya/tree/master/session/llama3

Benchmark

MalayMMLU

Probability next tokens

Based on 0-shot official MalayMMLU First token accuracy,

                             Model   Accuracy   shot by_letter        category
0  Malaysian-Llama-3.2-3B-Instruct  57.634056  0shot      True            STEM
1  Malaysian-Llama-3.2-3B-Instruct  59.351145  0shot      True        Language
2  Malaysian-Llama-3.2-3B-Instruct  57.559988  0shot      True  Social science
3  Malaysian-Llama-3.2-3B-Instruct  57.303910  0shot      True          Others
4  Malaysian-Llama-3.2-3B-Instruct  60.022753  0shot      True      Humanities
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
Model : Malaysian-Llama-3.2-3B-Instruct
Metric : first
Shot : 0shot
average accuracy 58.43555115020857
accuracy for STEM 57.63405648792468
accuracy for Language 59.35114503816794
accuracy for Social science 57.55998843596415
accuracy for Others 57.30390981050611
accuracy for Humanities 60.02275312855517

While the original model,

                   Model   Accuracy   shot by_letter        category
0  Llama-3.2-3B-Instruct  56.733524  0shot      True            STEM
1  Llama-3.2-3B-Instruct  58.460560  0shot      True        Language
2  Llama-3.2-3B-Instruct  54.206418  0shot      True  Social science
3  Llama-3.2-3B-Instruct  52.554569  0shot      True          Others
4  Llama-3.2-3B-Instruct  60.659841  0shot      True      Humanities
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
Model : Llama-3.2-3B-Instruct
Metric : first
Shot : 0shot
average accuracy 56.453145004749516
accuracy for STEM 56.73352435530086
accuracy for Language 58.460559796437664
accuracy for Social science 54.20641803989592
accuracy for Others 52.554569441112974
accuracy for Humanities 60.659840728100114

First token match using vLLM

Based on 0-shot exact first token match using vLLM,

                             Model   Accuracy  shot        category
0  Malaysian-Llama-3.2-3B-Instruct  51.944331     0            STEM
1  Malaysian-Llama-3.2-3B-Instruct  50.795165     0        Language
2  Malaysian-Llama-3.2-3B-Instruct  52.732003     0  Social science
3  Malaysian-Llama-3.2-3B-Instruct  52.026865     0          Others
4  Malaysian-Llama-3.2-3B-Instruct  54.539249     0      Humanities
Model : Malaysian-Llama-3.2-3B-Instruct
Metric : full
Shot : 0
average accuracy 52.35617230413414
accuracy for STEM 51.94433074089234
accuracy for Language 50.795165394402034
accuracy for Social science 52.73200346921075
accuracy for Others 52.02686495562485
accuracy for Humanities 54.53924914675768

While the original model,

                   Model   Accuracy  shot        category
0  Llama-3.2-3B-Instruct  50.511666     0            STEM
1  Llama-3.2-3B-Instruct  49.825064     0        Language
2  Llama-3.2-3B-Instruct  48.352125     0  Social science
3  Llama-3.2-3B-Instruct  48.213001     0          Others
4  Llama-3.2-3B-Instruct  51.990899     0      Humanities
Model : Llama-3.2-3B-Instruct
Metric : full
Shot : 0
average accuracy 49.58906372609755
accuracy for STEM 50.51166598444535
accuracy for Language 49.82506361323155
accuracy for Social science 48.35212489158716
accuracy for Others 48.21300071959703
accuracy for Humanities 51.990898748577926

Acknowledgement

Special thanks to https://www.sns.com.my for 8x H100 node!