See axolotl config
axolotl version: 0.9.2
base_model: Qwen/Qwen3-0.6B-Base
# Automatically upload checkpoint and final model to HF
# hub_model_id: username/custom_model_name
plugins:
- axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
strict: false
chat_template: qwen3
datasets:
- path: timarni/reasoning_SFT
type: chat_template
split: train
field_messages: conversations
# message_property_mappings:
# role: from
# content: value
val_set_size: 0.1
output_dir: ./outputs/qwen3_reasoning_sft
dataset_prepared_path: last_run_prepared
# To be sure that no LORA is done
adapter: null
lora: false
merge_lora: false
sequence_len: 4096 #2048
sample_packing: true
eval_sample_packing: true
pad_to_sequence_len: true
wandb_project: mnlp_project
wandb_entity: tim-arni
wandb_watch:
wandb_name: qwen3_reasoning_sft
wandb_log_model:
gradient_accumulation_steps: 2 # 16 following https://unsloth.ai/blog/qwen3
micro_batch_size: 1 # 2
num_epochs: 6
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 0.00005 # 0.0002
cosine_min_lr_ratio: 0.1
bf16: auto
tf32: true
gradient_checkpointing: offload
logging_steps: 1
gradient_clipping: 1.0
flash_attention: true
warmup_ratio: 0.03
evals_per_epoch: 4
saves_per_epoch: 2
save_total_limit: 25
weight_decay: 1e-4
special_tokens:
outputs/qwen3_reasoning_sft
This model is a fine-tuned version of Qwen/Qwen3-0.6B-Base on the timarni/reasoning_SFT dataset. It achieves the following results on the evaluation set:
- Loss: 0.8020
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 2
- total_train_batch_size: 8
- total_eval_batch_size: 4
- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 47
- num_epochs: 6.0
Training results
Training Loss | Epoch | Step | Validation Loss |
---|---|---|---|
0.965 | 0.0037 | 1 | 0.8999 |
0.8101 | 0.2505 | 67 | 0.7453 |
0.6077 | 0.5009 | 134 | 0.7342 |
0.5874 | 0.7514 | 201 | 0.7270 |
0.4362 | 1.0 | 268 | 0.7260 |
0.6779 | 1.2505 | 335 | 0.7269 |
0.505 | 1.5009 | 402 | 0.7310 |
0.4969 | 1.7514 | 469 | 0.7274 |
0.309 | 2.0 | 536 | 0.7332 |
0.5954 | 2.2505 | 603 | 0.7428 |
0.4302 | 2.5009 | 670 | 0.7514 |
0.4301 | 2.7514 | 737 | 0.7491 |
0.23 | 3.0 | 804 | 0.7559 |
0.5296 | 3.2505 | 871 | 0.7683 |
0.3761 | 3.5009 | 938 | 0.7857 |
0.3916 | 3.7514 | 1005 | 0.7818 |
0.1842 | 4.0 | 1072 | 0.7863 |
0.4926 | 4.2505 | 1139 | 0.7980 |
0.3469 | 4.5009 | 1206 | 0.8004 |
0.3697 | 4.7514 | 1273 | 0.7908 |
0.1665 | 5.0 | 1340 | 0.7925 |
0.4773 | 5.2505 | 1407 | 0.8187 |
0.3364 | 5.5009 | 1474 | 0.8071 |
0.3622 | 5.7514 | 1541 | 0.8020 |
Framework versions
- Transformers 4.51.3
- Pytorch 2.5.1+cu121
- Datasets 3.5.1
- Tokenizers 0.21.1
- Downloads last month
- 4
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
🙋
Ask for provider support
Model tree for timarni/qwen3_reasoning_sft
Base model
Qwen/Qwen3-0.6B-Base