See axolotl config

axolotl version: 0.7.0

# モデルの設定
base_model: /notebooks/plamo-2-1b-gorilla-chat2              # HuggingFace上のモデル名
model_type: AutoModelForCausalLM         # モデルのロードに使用するクラス
tokenizer_type: AutoTokenizer           # トークナイザのロードに使用するクラス
trust_remote_code: true                 # リモートのカスタムコードを信頼してモデルをロード

hub_model_id: zamagi/fft-1
hub_strategy: "end"
push_dataset_to_hub:
hf_use_auth_token: true

plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_cross_entropy: false
liger_rope: true
liger_rms_norm: true
liger_swiglu: true
liger_fused_linear_cross_entropy: true

# 8bit/4bit設定（8bitモードでメモリ削減）
load_in_8bit: false   #f                      # 8bit量子化されたモデルをロード
load_in_4bit: false                     # 4bit量子化は使用しない
strict: false                           # 重みの厳密な一致を要求しない（追加トークン等がある場合に許容）

chat_template: tokenizer_default

# データセットの設定
datasets:
  - path: Aratako/Magpie-Tanuki-Qwen2.5-72B-Answered
    type: chat_template
    field_messages: messages
    message_field_role: role
    message_field_content: content
    roles_to_train: ["assistant"]       # 学習対象とする役割（アシスタントの発話のみ学習）
    train_on_eos: last
#  - path: Aratako/magpie-qwen2.5-32b-reasoning-100k-formatted
#    type: chat_template
#    field_messages: conversations
#    message_field_role: role
#    message_field_content: content
#  - path: Aratako/magpie-reasoning-llama-nemotron-70b-100k-filtered
#    type: chat_template
#    field_messages: conversations
#    message_field_role: role
#    message_field_content: content
#  - path: Aratako/Open-Platypus-Japanese-masked-formatted
#    type: chat_template
#    field_messages: conversations
#    message_field_role: role
#    message_field_content: content
#  - path: kanhatakeyama/wizardlm8x22b-logical-math-coding-sft_additional-ja
#    type: chat_template
#    field_messages: messages
#    message_field_role: role
#    message_field_content: content
#  - path: kanhatakeyama/ramdom-to-fixed-multiturn-Calm3
#    split: 20240806filtered
#    type: chat_template
#    field_messages: messages
#    message_field_role: role
#    message_field_content: content
#  - path: Aratako/magpie-ultra-v0.1-formatted
#    type: chat_template
#    field_messages: conversations
#    message_field_role: role
#    message_field_content: content
#  - path: Aratako/orca-agentinstruct-1M-v1-selected
#    type: chat_template
#    field_messages: messages
#    message_field_role: role
#    message_field_content: content
#  - path: Aratako/Synthetic-JP-EN-Coding-Dataset-801k-50k
#    type: chat_template
#    field_messages: messages
#    message_field_role: role
#    message_field_content: content

shuffle_merged_datasets: true
dataset_prepared_path: /notebooks/data/fft-data
val_set_size: 0.003
output_dir: /notebooks/data/27b-fft-out-1
dataset_keep_in_memory: false

gpu_memory_limit: 48GiB

sequence_len: 2048
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true

adapter:
lora_model_dir:
lora_r:
lora_alpha:
lora_dropout:
lora_target_linear:
lora_fan_in_fan_out:


# トレーニングの設定
gradient_accumulation_steps: 4
micro_batch_size: 8
num_epochs: 2
optimizer: paged_adamw_8bit
lr_scheduler:
cosine_min_lr_ratio: 0.1
learning_rate: 0.00001
max_steps: 5000

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

#wandb: false
#wandb_project: 27b-fft
#wandb_entity: aratako-lm
#wandb_watch:
#wandb_name: attempt-01
#wandb_log_model:

gradient_checkpointing: true
early_stopping_patience:
auto_resume_from_checkpoints: true
local_rank:
logging_steps: 1
xformers_attention: 
flash_attention: 

save_strategy: steps
save_steps: 100
save_total_limit: 2

warmup_steps: 50
eval_steps: 100
eval_batch_size: 1
eval_table_size:
eval_max_new_tokens:

debug:
deepspeed: /notebooks/axolotl/deepspeed_configs/zero3_bf16.json
weight_decay: 0.01
fsdp:
fsdp_config:


# 出力の保存設定
output_dir: /notebooks/output/plamo-2-1b-gorilla-chat3    # チェックポイントや最終モデルの出力先ディレクトリ
hub_model_id: zamagi/plamo-2-1b-gorilla-chat3   # (オプション) Hugging Face Hubにアップロードする場合のリポジトリ名

plamo-2-1b-gorilla-chat3

This model was trained from scratch on the Aratako/Magpie-Tanuki-Qwen2.5-72B-Answered dataset. It achieves the following results on the evaluation set:

• Loss: 1.1070

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 8
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 50
• training_steps: 750

Training results

Training Loss	Epoch	Step	Validation Loss
1.2715	0.0027	1	1.2753
1.2207	0.2660	100	1.1846
1.1895	0.5319	200	1.1564
1.2285	0.7979	300	1.1334
0.9658	1.0638	400	1.1341
1.0254	1.3298	500	1.1209
0.9521	1.5957	600	1.1129
1.0186	1.8617	700	1.1070

Framework versions

• Transformers 4.49.0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.1