mesolitica/Malaysian-Qwen2.5-1.5B-Reasoning-SFT

Malaysian Qwen 2.5 1.5B Instruct Reasoning SFT

Continue finetuning https://huggingface.co/mesolitica/Malaysian-Qwen2.5-1.5B-Instruct on highly curated Malaysian Reasoning dataset.

Improvement

1. Reasoning on Math, Science, Translation, Dialects, Multiple choices, coding and Maktabah Al Bakri.
2. Warmup reasoning.

Training session

Finetune on mesolitica/Malaysian-Reasoning to make the model better reasoning on Malaysian context.

How we train

1. Full parameters on 12k context length.
2. WanDB at https://wandb.ai/huseinzol05/fpf-qwen2.5-1.5b-malaysian-12k-reasoning

Source code at https://github.com/mesolitica/malaya/tree/master/session/qwen2.5

Benchmark

Dialect Translation

All the benchmarks generate using vLLM, evaluation based on sacrebleu CHRF max@5.

Source code for evaluation at https://github.com/mesolitica/malaya/tree/master/session/qwen2.5/evaluate-dialect

Dialect to standard Malay,

Standard Malay to dialect,

MalayMMLU

Special thanks

Special thanks to https://www.sns.com.my and Nvidia for 8x H100 node!