See axolotl config
axolotl version: 0.9.2
base_model: timarni/qwen3_s1k
# Automatically upload checkpoint and final model to HF
# hub_model_id: username/custom_model_name
plugins:
- axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
strict: false
chat_template: qwen3
datasets:
- path: timarni/MNLP_STEM_IT_HARD
type: alpaca
split: train
val_set_size: 0.15
output_dir: ./outputs/qwen3_s1k_it_hard
dataset_prepared_path: last_run_prepared
sequence_len: 2048 # 4096
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true
# To be sure that no LORA is done
adapter: null
lora: false
merge_lora: false
wandb_project: mnlp_project
wandb_entity: tim-arni
wandb_watch:
wandb_name: qwen3_s1k_it_hard
wandb_log_model:
gradient_accumulation_steps: 4 # 2
micro_batch_size: 2 # 1
num_epochs: 5
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 0.00001 # 0.00005
cosine_min_lr_ratio: 0.1
bf16: auto
tf32: true
gradient_checkpointing: offload
gradient_checkpointing_kwargs:
use_reentrant: false
resume_from_checkpoint:
logging_steps: 1
flash_attention: true
warmup_ratio: 0.03
evals_per_epoch: 4
saves_per_epoch: 2
save_total_limit: 10
weight_decay: 0.001
special_tokens:
outputs/qwen3_s1k_it_hard
This model is a fine-tuned version of timarni/qwen3_s1k on the timarni/MNLP_STEM_IT_HARD dataset. It achieves the following results on the evaluation set:
- Loss: 0.1654
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 13
- num_epochs: 5.0
Training results
Training Loss | Epoch | Step | Validation Loss |
---|---|---|---|
2.0345 | 0.0109 | 1 | 2.0409 |
0.1118 | 0.2514 | 23 | 0.1711 |
0.0848 | 0.5027 | 46 | 0.1647 |
0.0884 | 0.7541 | 69 | 0.1625 |
0.1029 | 1.0 | 92 | 0.1623 |
0.0555 | 1.2514 | 115 | 0.1616 |
0.0767 | 1.5027 | 138 | 0.1618 |
0.0743 | 1.7541 | 161 | 0.1612 |
0.0747 | 2.0 | 184 | 0.1619 |
0.0571 | 2.2514 | 207 | 0.1647 |
0.0543 | 2.5027 | 230 | 0.1628 |
0.0573 | 2.7541 | 253 | 0.1643 |
0.0601 | 3.0 | 276 | 0.1630 |
0.057 | 3.2514 | 299 | 0.1641 |
0.0438 | 3.5027 | 322 | 0.1647 |
0.0564 | 3.7541 | 345 | 0.1648 |
0.0677 | 4.0 | 368 | 0.1648 |
0.0519 | 4.2514 | 391 | 0.1656 |
0.0487 | 4.5027 | 414 | 0.1653 |
0.0714 | 4.7541 | 437 | 0.1654 |
Framework versions
- Transformers 4.51.3
- Pytorch 2.5.1+cu121
- Datasets 3.5.1
- Tokenizers 0.21.1
- Downloads last month
- 5
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
🙋
Ask for provider support