README.md

---
tags:
- transformers
- llama
- trl
- orpeheutts
- tts
- Texttospeech
license: apache-2.0
language:
- es
datasets:
- sirekist98/spanish_tts_noauddataset_24khz
base_model:
- canopylabs/3b-es_it-pretrain-research_release
pipeline_tag: text-to-speech
---
# Spanish TTS Model with Emotions and Multiple Voices

This repository contains a fine-tuned Spanish Text-to-Speech (TTS) model based on [`canopylabs/3b-es_it-pretrain-research_release`](https://huggingface.co/canopylabs/3b-es_it-pretrain-research_release). The model supports multiple voices and nuanced emotions, trained using [Unsloth](https://github.com/unslothai/unsloth) and [SNAC](https://huggingface.co/hubertsiuzdak/snac_24khz) for audio tokenization.

➡️ **Try it online**: [https://huggingface.co/spaces/sirekist98/orpheustts\_spanish\_tuned](https://huggingface.co/spaces/sirekist98/orpheustts_spanish_tuned)

---

## 👨‍💻 Model Summary

* **Base model**: `canopylabs/3b-es_it-pretrain-research_release`
* **Fine-tuned with**: LoRA adapters (64 rank, alpha 64)
* **Audio tokenization**: SNAC (24kHz)
* **Input format**: `source (emotion): text`
* **Dataset**: \~109k samples, 11 emotions × 11 speakers
* **Training framework**: Unsloth + Hugging Face Transformers

---

## 🚀 Training Overview

The model was trained on a curated subset of the dataset [`sirekist98/spanish_tts_noauddataset_24khz`](https://huggingface.co/datasets/sirekist98/spanish_tts_noauddataset_24khz). We selected combinations of speaker (`source`) and `emotion` with at least 1000 samples, resulting in a balanced dataset of over 109,000 examples.

Each sample was tokenized using SNAC and embedded in a prompt structured as:

```text
source (emotion): text
```

This prompt was then used to generate audio tokens, enabling the model to learn nuanced emotional prosody and voice control.

We trained the model for 1 epoch using gradient accumulation (batch size 8 × 4 steps) with 4-bit quantization on an NVIDIA L4 GPU.

---

## 🔊 Inference

You can run inference using the demo space: [Orpheus TTS Spanish Fine-Tuned](https://huggingface.co/spaces/sirekist98/orpheustts_spanish_tuned).

To run inference locally with full control:

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
from snac import SNAC

# --- Minimal config ---
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
BASE  = "canopylabs/3b-es_it-pretrain-research_release"
LORA  = "sirekist98/spanish_tts_emotions"
SNAC_ID = "hubertsiuzdak/snac_24khz"

VOICE = "alloy"
EMOTION_ID = "intense_fear_dread_apprehension_horror_terror_panic"
TEXT = "Estoy atrapado, por favor ayúdame."
prompt = f"{VOICE} ({EMOTION_ID}): {TEXT}"

# --- Load models ---
tokenizer  = AutoTokenizer.from_pretrained(BASE)
base_model = AutoModelForCausalLM.from_pretrained(
    BASE,
    torch_dtype=torch.float16 if device.type == "cuda" else torch.float32
)
model      = PeftModel.from_pretrained(base_model, LORA).to(device).eval()
snac_model = SNAC.from_pretrained(SNAC_ID).to(device)

# --- Prepare input (same as your Space) ---
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
start_tok = torch.tensor([[128259]], dtype=torch.long).to(device)
end_toks  = torch.tensor([[128009, 128260]], dtype=torch.long).to(device)

input_ids = torch.cat([start_tok, input_ids, end_toks], dim=1)
MAX_LEN   = 4260
pad_len   = MAX_LEN - input_ids.shape[1]
pad       = torch.full((1, pad_len), 128263, dtype=torch.long).to(device)
input_ids = torch.cat([pad, input_ids], dim=1)
attention_mask = torch.cat(
    [torch.zeros((1, pad_len), dtype=torch.long),
     torch.ones((1, input_ids.shape[1] - pad_len), dtype=torch.long)],
    dim=1
).to(device)

# --- Generate ---
generated = model.generate(
    input_ids=input_ids,
    attention_mask=attention_mask,
    max_new_tokens=1200,
    do_sample=True,
    temperature=0.6,
    top_p=0.95,
    repetition_penalty=1.1,
    num_return_sequences=1,
    eos_token_id=128258,
    use_cache=True
)

# --- Post-process (find 128257, remove 128258, multiple of 7, subtract 128266) ---
AUDIO_TOKEN_OFFSET = 128266
token_to_find      = 128257
token_to_remove    = 128258

idxs = (generated == token_to_find).nonzero(as_tuple=True)
cropped = generated[:, idxs[1][-1].item() + 1:] if len(idxs[1]) > 0 else generated
cleaned = cropped[cropped != token_to_remove]
codes   = cleaned[: (len(cleaned) // 7) * 7].tolist()
codes   = [int(t) - AUDIO_TOKEN_OFFSET for t in codes]

# --- SNAC decode (same layout as your Space) ---
layer_1, layer_2, layer_3 = [], [], []
for i in range((len(codes) + 1) // 7):
    b = 7 * i
    if b + 6 >= len(codes):
        break
    layer_1.append(codes[b + 0])
    layer_2.append(codes[b + 1] - 4096)
    layer_3.append(codes[b + 2] - 2 * 4096)
    layer_3.append(codes[b + 3] - 3 * 4096)
    layer_2.append(codes[b + 4] - 4 * 4096)
    layer_3.append(codes[b + 5] - 5 * 4096)
    layer_3.append(codes[b + 6] - 6 * 4096)

dev_snac = snac_model.quantizer.quantizers[0].codebook.weight.device
layers = [
    torch.tensor(layer_1).unsqueeze(0).to(dev_snac),
    torch.tensor(layer_2).unsqueeze(0).to(dev_snac),
    torch.tensor(layer_3).unsqueeze(0).to(dev_snac),
]

with torch.no_grad():
    audio = snac_model.decode(layers).squeeze().cpu().numpy()

# 'audio' is the 24kHz waveform.
# Optional:
# from scipy.io.wavfile import write as write_wav
# write_wav("output.wav", 24000, audio)
```

---

## 🗣️ Available Voices

You can generate speech using the following voices (`source`):

```
alloy, ash, ballad, coral, echo, fable, nova, onyx, sage, shimmer, verse
```

## 🌧️ Available Emotions for each voice


---

## alloy

* intense\_interest\_fascination\_curiosity\_and\_intrigue
* intense\_fear\_dread\_apprehension\_and\_horror
* intense\_ecstasy\_pleasure\_bliss\_rapture\_and\_beatitude
* intense\_numbness\_detachment\_insensitivity\_and\_apathy
* intense\_contempt\_disdain\_loathing\_and\_detestation
* intense\_astonishment\_surprise\_amazement\_and\_shock
* intense\_confusion\_bewilderment\_disorientation\_and\_perplexity
* intense\_pride\_dignity\_self\_confidence\_and\_honor
* intense\_sourness\_tartness\_and\_acidity
* intense\_sympathy\_compassion\_warmth\_trust\_and\_tenderness

## ash

* intense\_interest\_fascination\_curiosity\_and\_intrigue
* intense\_fear\_dread\_apprehension\_and\_horror
* intense\_ecstasy\_pleasure\_bliss\_rapture\_and\_beatitude
* intense\_numbness\_detachment\_insensitivity\_and\_apathy
* intense\_astonishment\_surprise\_amazement\_and\_shock
* intense\_sympathy\_compassion\_warmth\_trust\_and\_tenderness

## ballad

* intense\_interest\_fascination\_curiosity\_and\_intrigue
* intense\_fear\_dread\_apprehension\_and\_horror
* intense\_ecstasy\_pleasure\_bliss\_rapture\_and\_beatitude
* intense\_numbness\_detachment\_insensitivity\_and\_apathy
* intense\_contempt\_disdain\_loathing\_and\_detestation
* intense\_astonishment\_surprise\_amazement\_and\_shock
* intense\_confusion\_bewilderment\_disorientation\_and\_perplexity
* intense\_helplessness\_powerlessness\_desperation\_and\_submission
* intense\_pride\_dignity\_self\_confidence\_and\_honor
* intense\_sourness\_tartness\_and\_acidity

## coral

* intense\_fear\_dread\_apprehension\_and\_horror
* intense\_ecstasy\_pleasure\_bliss\_rapture\_and\_beatitude
* intense\_numbness\_detachment\_insensitivity\_and\_apathy
* intense\_contempt\_disdain\_loathing\_and\_detestation
* intense\_confusion\_bewilderment\_disorientation\_and\_perplexity
* intense\_helplessness\_powerlessness\_desperation\_and\_submission
* intense\_pride\_dignity\_self\_confidence\_and\_honor
* intense\_sourness\_tartness\_and\_acidity
* intense\_sympathy\_compassion\_warmth\_trust\_and\_tenderness

## echo

* intense\_interest\_fascination\_curiosity\_and\_intrigue
* intense\_ecstasy\_pleasure\_bliss\_rapture\_and\_beatitude
* intense\_numbness\_detachment\_insensitivity\_and\_apathy
* intense\_contempt\_disdain\_loathing\_and\_detestation
* intense\_astonishment\_surprise\_amazement\_and\_shock
* intense\_helplessness\_powerlessness\_desperation\_and\_submission
* intense\_pride\_dignity\_self\_confidence\_and\_honor
* intense\_sympathy\_compassion\_warmth\_trust\_and\_tenderness

## fable

* intense\_interest\_fascination\_curiosity\_and\_intrigue
* intense\_fear\_dread\_apprehension\_and\_horror
* intense\_ecstasy\_pleasure\_bliss\_rapture\_and\_beatitude
* intense\_numbness\_detachment\_insensitivity\_and\_apathy
* intense\_contempt\_disdain\_loathing\_and\_detestation
* intense\_helplessness\_powerlessness\_desperation\_and\_submission
* intense\_sourness\_tartness\_and\_acidity

## nova

* intense\_ecstasy\_pleasure\_bliss\_rapture\_and\_beatitude
* intense\_contempt\_disdain\_loathing\_and\_detestation
* intense\_astonishment\_surprise\_amazement\_and\_shock
* intense\_confusion\_bewilderment\_disorientation\_and\_perplexity
* intense\_helplessness\_powerlessness\_desperation\_and\_submission
* intense\_pride\_dignity\_self\_confidence\_and\_honor
* intense\_sourness\_tartness\_and\_acidity
* intense\_sympathy\_compassion\_warmth\_trust\_and\_tenderness

## onyx

* intense\_interest\_fascination\_curiosity\_and\_intrigue
* intense\_fear\_dread\_apprehension\_and\_horror
* intense\_numbness\_detachment\_insensitivity\_and\_apathy
* intense\_confusion\_bewilderment\_disorientation\_and\_perplexity
* intense\_helplessness\_powerlessness\_desperation\_and\_submission
* intense\_pride\_dignity\_self\_confidence\_and\_honor
* intense\_sympathy\_compassion\_warmth\_trust\_and\_tenderness

## sage

* intense\_interest\_fascination\_curiosity\_and\_intrigue
* intense\_fear\_dread\_apprehension\_and\_horror
* intense\_ecstasy\_pleasure\_bliss\_rapture\_and\_beatitude
* intense\_numbness\_detachment\_insensitivity\_and\_apathy
* intense\_astonishment\_surprise\_amazement\_and\_shock
* intense\_confusion\_bewilderment\_disorientation\_and\_perplexity
* intense\_pride\_dignity\_self\_confidence\_and\_honor
* intense\_sourness\_tartness\_and\_acidity
* intense\_sympathy\_compassion\_warmth\_trust\_and\_tenderness

## shimmer

* intense\_interest\_fascination\_curiosity\_and\_intrigue
* intense\_fear\_dread\_apprehension\_and\_horror
* intense\_ecstasy\_pleasure\_bliss\_rapture\_and\_beatitude
* intense\_numbness\_detachment\_insensitivity\_and\_apathy
* intense\_contempt\_disdain\_loathing\_and\_detestation
* intense\_astonishment\_surprise\_amazement\_and\_shock
* intense\_confusion\_bewilderment\_disorientation\_and\_perplexity
* intense\_helplessness\_powerlessness\_desperation\_and\_submission
* intense\_pride\_dignity\_self\_confidence\_and\_honor
* intense\_sourness\_tartness\_and\_acidity

## verse

* intense\_interest\_fascination\_curiosity\_and\_intrigue
* intense\_fear\_dread\_apprehension\_and\_horror
* intense\_ecstasy\_pleasure\_bliss\_rapture\_and\_beatitude
* intense\_numbness\_detachment\_insensitivity\_and\_apathy
* intense\_contempt\_disdain\_loathing\_and\_detestation
* intense\_astonishment\_surprise\_amazement\_and\_shock
* intense\_helplessness\_powerlessness\_desperation\_and\_submission
* intense\_sourness\_tartness\_and\_acidity


---

## 📖 Citation

```bibtex
@misc{sirekist2025spanishTTS,
  author = {sirekist98},
  title = {Spanish TTS Model with Emotions and Multiple Voices},
  year = {2025},
  howpublished = {\url{https://huggingface.co/sirekist98/spanish_model}}
}
```

---

## ✨ Acknowledgements

* [Unsloth](https://github.com/unslothai/unsloth)
* [SNAC](https://huggingface.co/hubertsiuzdak/snac_24khz)
* [Hugging Face Datasets and Spaces](https://huggingface.co/)

---

## ❓ Questions or Contributions?

Open an issue or contact [@sirekist98](https://huggingface.co/sirekist98) on Hugging Face.

Thanks for checking out this model! 🚀