dgx1_distil_w2v2_base_mozilla_12_to_6_batch_16_epoch_50_try4

This model is a fine-tuned version of rohitp1/dgx1_distil_w2v2_base_mozilla_12_to_6_batch_16_epoch_50_try3 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 43.2001
• Wer: 0.3569

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 16
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 256
• total_train_batch_size: 4096
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.2
• num_epochs: 50
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
126.3084	4.41	150	44.4592	0.3532
118.4174	8.82	300	45.5311	0.3651
122.1544	13.23	450	41.0647	0.3677
141.5393	17.64	600	43.9298	0.3685
135.1174	22.06	750	41.6142	0.3668
126.6081	26.47	900	43.2001	0.3569

Framework versions

• Transformers 4.25.1
• Pytorch 1.12.1
• Datasets 2.8.0
• Tokenizers 0.13.2