| # Orpheus Music Transformer Training Logs | |
| *** | |
| ## Training hyperparameters for both models (base and bridge) | |
| - GPU: Single Nvidia GH200 96GB | |
| - learning_rate: 1e-04 | |
| - train_batch_size: 9 | |
| - gradients_accumulation_value: 8 | |
| - gradients_clip_value: 1.0 | |
| - eval_batch_size: 9 | |
| - optimizer: PyTorch Adam with default parameters | |
| - lr_scheduler_type: constant | |
| - precision: bfloat16 | |
| - embeddings: RoPE | |
| - attention: flash | |
| - num_epochs: 3 | |
| *** | |
| ## [Base model train/eval metrics](https://huggingface.co/asigalov61/Orpheus-Music-Transformer/tree/main/logs/base) | |
| ### Cross-entropy losses/accs (averaged last 100 values) | |
| - train_loss: 0.6559235458076 | |
| - train_acc: 0.8074536085128784 | |
| - eval_loss: 0.6947438363730908 | |
| - eval_acc: 0.7946965831518173 | |
| - epochs: 3 | |
| - num_steps: 96332 | |
| - training_time: 194 hours / 8 days | |
| *** | |
| ### Project Los Angeles | |
| ### Tegridy Code 2025 |