g3-12b-it-unalign

This model is a fine-tuned version of unsloth/gemma-3-12b-it on the ToastyPigeon/unalign-v2 dataset. It achieves the following results on the evaluation set:

• Loss: 1.4684

So, it seems alright. I noticed however that the responses got pretty short at the end of the 2nd epoch. Not like, unusably short, but generally shorter than I personally like.

The epoch 1 test gguf is based on this commit.

I personally prefer epoch 1 to epoch 2, and will likely update this or make a second proper commit for epoch 1.

Update: I did indeed make a second commit for the epoch 1 checkpoint.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 69
• optimizer: Use apollo_adamw_layerwise with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=proj=random,rank=1,scale=128.0,scale_type=tensor,update_proj_gap=200
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 5
• num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss
7.8965	0.0118	1	6.4897
4.4934	0.2	17	4.0497
3.9523	0.4	34	3.7484
3.5624	0.6	51	3.3152
2.7168	0.8	68	2.4773
2.1303	1.0	85	1.9483
1.8215	1.2	102	1.7577
1.7199	1.4	119	1.6561
1.5771	1.6	136	1.5611
1.5599	1.8	153	1.5124
1.4831	2.0	170	1.4684

Framework versions

• Transformers 4.50.0.dev0
• Pytorch 2.5.1+cu124
• Datasets 3.4.1
• Tokenizers 0.21.1