timarni/qwen3_stem_pretrained

See axolotl config

axolotl version: 0.9.2

######################################
#  CONTINUED PRE-TRAINING EXAMPLE    #
######################################

base_model: Qwen/Qwen3-0.6B-Base   # the checkpoint you start from
strict: false

# 1⃣  Replace `datasets:` with `pretraining_dataset:`
pretraining_dataset:
  - path: timarni/pretrain-textbooks   # or HF dataset id
    type: completion   # accepted values: text | completion | HF dataset

  - path: timarni/pretrain-wikipedia   # or HF dataset id
    type: completion   # accepted values: text | completion | HF dataset

# 2⃣  Remove chat / instruction-tuning options
chat_template:
# adapter / lora stay null/false (full-parameter training)

# 3⃣  Training hyper-params (see Section 3)
sequence_len: 4096
sample_packing: true
eval_sample_packing: true
pad_to_sequence_len: true

micro_batch_size: 1
gradient_accumulation_steps: 2
max_steps: 12000 # num_epochs does not work for streaming mode in pretraining
learning_rate: 1e-5
lr_scheduler: cosine
warmup_steps: 100
weight_decay: 0.01

optimizer: adamw_torch
bf16: auto
tf32: true
flash_attention: true
gradient_checkpointing: offload

val_set_size: 0.0        # usually no dev set for plain pre-training
output_dir: ./outputs/qwen3_pretraining_full
dataset_prepared_path: last_run_prepared

wandb_project: mnlp_project
wandb_entity: tim-arni
wandb_name: qwen3-0.6B-pretraining_full

outputs/qwen3_pretraining_full

This model is a fine-tuned version of Qwen/Qwen3-0.6B-Base on an unknown dataset.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• total_eval_batch_size: 4
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• training_steps: 12000

Training results

Framework versions

• Transformers 4.51.3
• Pytorch 2.5.1+cu121
• Datasets 3.5.1
• Tokenizers 0.21.1