Hugging Face's logo

winglian
/
qwen3-4b-math

Text Generation
Transformers
Safetensors
qwen3
Generated from Trainer
conversational
text-generation-inference
Model card Files
xet
Community
qwen3-4b-math / README.md
winglian's picture
winglian
Upload folder using huggingface_hub
3b68b0d verified
preview code
|
raw
history blame contribute delete
3.55 kB
metadata
library_name: transformers
license: apache-2.0
base_model: Qwen/Qwen3-4B-Base
tags:
  - generated_from_trainer
datasets:
  - winglian/OpenThoughts-114k-math-correct
model-index:
  - name: outputs/model-out-math-4b
    results: []

Built with Axolotl

See axolotl config

axolotl version: 0.10.0.dev0

base_model: Qwen/Qwen3-4B-Base

plugins:
  - axolotl.integrations.liger.LigerPlugin
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin

liger_rms_norm: true
liger_glu_activation: true

# torch_compile: true

dataloader_prefetch_factor: 4
dataloader_num_workers: 2
dataloader_pin_memory: true

chat_template: qwen3
datasets:
  - path: winglian/OpenThoughts-114k-math-correct
    type: chat_template
    split: train
    split_thinking: true
    eot_tokens:
      - "<|im_end|>"

dataset_prepared_path: last_run_prepared
val_set_size: 0.01
output_dir: ./outputs/model-out-math-4b

sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true

wandb_project: kd-4b-math
wandb_entity: axolotl-ai
wandb_name: sft-4b

gradient_accumulation_steps: 1
micro_batch_size: 4
num_epochs: 2
optimizer: adamw_torch_fused
adam_beta2: 0.95
lr_scheduler: rex
learning_rate: 3e-5
max_grad_norm: 0.1
save_safetensors: true

bf16: true
tf32: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
logging_steps: 1
flash_attention: true

warmup_steps: 100
evals_per_epoch: 4
saves_per_epoch: 1
weight_decay: 0.1
special_tokens:
  eos_token: <|im_end|>
deepspeed: deepspeed_configs/zero2_torch_compile.json

outputs/model-out-math-4b

This model is a fine-tuned version of Qwen/Qwen3-4B-Base on the winglian/OpenThoughts-114k-math-correct dataset. It achieves the following results on the evaluation set:

  • Loss: 0.3929

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 3e-05
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 8
  • total_train_batch_size: 32
  • total_eval_batch_size: 32
  • optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.95) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 100
  • num_epochs: 2.0

Training results

Training Loss Epoch Step Validation Loss
0.5644 0.0016 1 0.5801
0.4038 0.2504 159 0.4154
0.3914 0.5008 318 0.4035
0.3812 0.7512 477 0.3960
0.3626 1.0016 636 0.3915
0.316 1.2520 795 0.3958
0.3171 1.5024 954 0.3963
0.2944 1.7528 1113 0.3929

Framework versions

  • Transformers 4.51.3
  • Pytorch 2.7.0+cu128
  • Datasets 3.5.1
  • Tokenizers 0.21.1