winglian/qwen3-14b-math

See axolotl config

axolotl version: 0.10.0.dev0

base_model: Qwen/Qwen3-14B-Base

plugins:
  - axolotl.integrations.liger.LigerPlugin
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin

liger_rms_norm: true
liger_glu_activation: true

# torch_compile: true

dataloader_prefetch_factor: 4
dataloader_num_workers: 2
dataloader_pin_memory: true


chat_template: qwen3
datasets:
  - path: winglian/OpenThoughts-114k-math-correct
    type: chat_template
    split: train
    split_thinking: true
    eot_tokens:
      - "<|im_end|>"

dataset_prepared_path: last_run_prepared
val_set_size: 0.01
output_dir: ./outputs/model-out-math-14b

sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true

wandb_project: kd-14b-tests
wandb_entity: axolotl-ai
wandb_name: sft-14b

gradient_accumulation_steps: 1
micro_batch_size: 4
num_epochs: 2
optimizer: adamw_torch_fused
adam_beta2: 0.99
lr_scheduler: rex
learning_rate: 1e-5
max_grad_norm: 0.1
save_safetensors: true

bf16: true
tf32: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
logging_steps: 1
flash_attention: true

warmup_steps: 100
evals_per_epoch: 4
saves_per_epoch: 1
weight_decay: 0.1
special_tokens:
  eos_token: <|im_end|>
deepspeed: deepspeed_configs/zero2_torch_compile.json

outputs/model-out-math-14b

This model is a fine-tuned version of Qwen/Qwen3-14B-Base on the winglian/OpenThoughts-114k-math-correct dataset. It achieves the following results on the evaluation set:

• Loss: 0.3439

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• total_train_batch_size: 32
• total_eval_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.99) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss
0.5092	0.0016	1	0.5219
0.3572	0.2504	159	0.3653
0.3438	0.5008	318	0.3549
0.3345	0.7512	477	0.3491
0.3209	1.0016	636	0.3451
0.3013	1.2520	795	0.3465
0.2926	1.5024	954	0.3459
0.283	1.7528	1113	0.3439

Framework versions

• Transformers 4.51.3
• Pytorch 2.7.0+cu128
• Datasets 3.5.1
• Tokenizers 0.21.1