readme.md

---
license: apache-2.0
language:
- da
base_model: sesame/csm-1b
tags:
- text-to-speech
- tts
- danish
- lora
- csm
- audio-generation
- speech-synthesis
library_name: transformers
pipeline_tag: text-to-speech
datasets:
- mozilla-foundation/common_voice_17_0
- CoRal-project/coral-tts
---

# CSM-1B Danish Text-to-Speech (LoRA)

A natural-sounding Danish text-to-speech model based on CSM-1B, fine-tuned using LoRA (Low-Rank Adaptation) on a combination of Common Voice 17, CoRal-TTS, and private Danish speech data. Authored by [Nicolaj Reck](https://www.linkedin.com/in/nicolaj-reck-053aa38a/),

## Model Description

This model is a LoRA adapter for [`sesame/csm-1b`](https://huggingface.co/sesame/csm-1b) that enables natural Danish speech synthesis with optional voice control. The adapter was trained specifically for Danish TTS while preserving the multilingual capabilities of the base model.

- **Base Model**: [`sesame/csm-1b`](https://huggingface.co/sesame/csm-1b)
- **Language**: Danish (da)
- **Task**: Text-to-Speech
- **License**: Apache 2.0
- **Model Type**: LoRA Adapter
- **Precision**: FP16/BF16

## Key Features

- 🎯 **Natural Danish synthesis** with clear pronunciation and fluent prosody
- 🇬🇧 **Exceptional English with Danish accent** - Perfect for bilingual content
- 🔄 **Voice control** with male/female speaker selection
- ⚡ **Efficient fine-tuning** using LoRA (only ~16M parameters trained)
- 🛡️ **Voice leakage prevention** through frozen speaker/codec modules
- 📱 **Ready-to-use Gradio interface** included

## Quick Start

### Installation

```bash
pip install transformers torch torchaudio gradio
```

### Basic Usage

```python
import torch
from transformers import CsmForConditionalGeneration, AutoProcessor

# Load model and processor
model = CsmForConditionalGeneration.from_pretrained("nicolajreck/csm-1b-danish-tts")
processor = AutoProcessor.from_pretrained("nicolajreck/csm-1b-danish-tts")

# Generate speech
text = "[1]Hej! Velkommen til dansk tale syntese."  # [1] for female voice
inputs = processor(text, add_special_tokens=True).to("cuda")
audio = model.generate(**inputs, output_audio=True)

# Save audio
processor.save_audio(audio, "output.wav")
```

### Web Interface

Launch the included Gradio interface:

```bash
python danish_tts.py
```

Access at `http://localhost:7860` for an interactive TTS experience.

## Voice Control

The model supports two speaker voices:
- `[0]` - Male voice
- `[1]` - Female voice

Simply prefix your Danish text with the speaker token:
- `[0]God morgen! Hvordan har du det?` (Male)
- `[1]God morgen! Hvordan har du det?` (Female)

## Training Details

### Training Data

The model was trained on a carefully curated mix of Danish speech data:

- **[Common Voice 17 Danish](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0)**: ~10,224 validated samples
- **[CoRal-TTS Danish](https://huggingface.co/datasets/CoRal-project/coral-tts)**: ~16,547 filtered samples  
- **Private Extension**: ~8,644 additional samples

Total: ~35,415 Danish speech samples with balanced representation across datasets.

### Training Configuration

- **Method**: LoRA (Low-Rank Adaptation)
- **Rank**: 16, Alpha: 32, Dropout: 0.05
- **Target Modules**: `{q_proj, k_proj, v_proj, o_proj, out_proj, gate_proj, up_proj, down_proj, fc1, fc2}`
- **Hardware**: Single RTX 3090 (24GB)
- **Precision**: FP16 training, supports FP16/BF16 inference

### Data Processing

- Duration filtering: 0.6-16 seconds
- Text normalization: Quote stripping, terminal punctuation
- Equal-probability dataset mixing to prevent bias
- Chat-style formatting with Danish language cue

## Recommended Settings

For the most natural and fluent speech, use these generation parameters:

```python
# Natural speech settings
audio = model.generate(
    **inputs,
    output_audio=True,
    do_sample=True,
    temperature=0.96,
    depth_decoder_temperature=0.7, 
    top_k=50,
    top_p=0.9,
    repetition_penalty=1.0
)
```

## Example Outputs

The model handles various Danish text types effectively:

| Danish Text | Audio |
|-------------|-------|
| *"Husk at gemme arbejdet, før computeren genstarter, ellers risikerer du at miste både filer og vigtige ændringer."* | <audio controls><source src="./tts_examples/technical_instructions.wav" type="audio/wav"><source src="./tts_examples/technical_instructions.mp3" type="audio/mp3">Your browser does not support the audio element.</audio> |
| *"Pakken leveres i morgen mellem 9 og 12, og du får en SMS-besked, så snart den er klar til afhentning."* | <audio controls><source src="./tts_examples/service_message.wav" type="audio/wav"><source src="./tts_examples/service_message.mp3" type="audio/mp3">Your browser does not support the audio element.</audio> |
| *"Vi gør opmærksom på, at toget mod Københavns Hovedbanegård er forsinket med omkring 15 minutter. Vi undskylder ventetiden."* | <audio controls><source src="./tts_examples/announcement.wav" type="audio/wav"><source src="./tts_examples/announcement.mp3" type="audio/mp3">Your browser does not support the audio element.</audio> |
| *"Når du planlægger en rejse, kan det betale sig at undersøge, både transportmuligheder, overnatning og oplevelser inden da. Sådan får du mest muligt ud af tiden, og du slipper for unødvendig stress undervejs."* | <audio controls><source src="./tts_examples/travel_planning.wav" type="audio/wav"><source src="./tts_examples/travel_planning.mp3" type="audio/mp3">Your browser does not support the audio element.</audio> |

## Performance

Compared to the base CSM-1B model on Danish text:
- ✅ Pronunciation and word clarity
- ✅ Natural rhythm and speaking flow
- ✅ Speech with fewer dropped sounds
- ✅ Pleasant voice across different text types

## Gradio Interface Features

The included `danish_tts.py` provides a comprehensive web interface with:

- **Three-column layout**: Input settings, sampling controls, audio output
- **Auto max-length calculation** with adjustable multiplier
- **Advanced parameter control**: Dual temperatures, Top-K/Top-P, repetition penalty
- **Pre-configured examples** with optimized settings
- **Real-time generation** and audio playback

## Limitations

- Optimized specifically for Danish - other languages may have reduced quality
- Requires base model `sesame/csm-1b` to function
- Voice control limited to male/female binary selection
- Generated audio should be identified as synthetic in production use

## Technical Details

### Model Architecture
- **Base**: CSM-1B encoder-decoder with depth decoder
- **Audio Format**: 24kHz, generated via audio tokens
- **LoRA Integration**: Language projections only, speaker/codec frozen
- **Memory Requirements**: ~8GB VRAM for inference

### Files Included
- LoRA adapter weights
- Processor configuration  
- Gradio web interface (`danish_tts.py`)
- Training scripts and utilities

## Citation

If you use this model, please cite:

```bibtex
@misc{csm1b-danish-2024,
  title={High-Quality Danish Text-to-Speech with CSM-1B: Data Mixing, Voice Control, and LoRA Fine-Tuning},
  author={Nicolaj Reck},
  year={2024},
  howpublished={\\url{https://huggingface.co/nicolajreck/csm-1b-danish-tts}},
  note={LinkedIn: https://www.linkedin.com/in/nicolaj-reck-053aa38a/}
}
```

## Acknowledgments

**Authored by**: [Nicolaj Reck](https://www.linkedin.com/in/nicolaj-reck-053aa38a/) - 

Thanks to:
- **[Mozilla Foundation](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0)** for the Common Voice 17 dataset
- **[CoRal-TTS project](https://huggingface.co/datasets/CoRal-project/coral-tts)** for the Danish speech corpus  
- **[Sesame Research](https://huggingface.co/sesame/csm-1b)** for the base CSM-1B model
- The open-source community for tools and frameworks

## License

This model is released under the Apache 2.0 license. Please see the base model license for additional terms.