Update README.md
Browse files
README.md
CHANGED
@@ -16,10 +16,11 @@ Dia is a 1.6B parameter text to speech model created by Nari Labs.
|
|
16 |
|
17 |
Dia **directly generates highly realistic dialogue from a transcript**. You can condition the output on audio, enabling emotion and tone control. The model can also produce nonverbal communications like laughter, coughing, clearing throat, etc.
|
18 |
|
19 |
-
To accelerate research, we are providing access to pretrained model checkpoints and inference code. The model weights are hosted on [Hugging Face](https://huggingface.co/nari-labs/Dia-1.6B).
|
20 |
|
21 |
We also provide a [demo page](https://yummy-fir-7a4.notion.site/dia) comparing our model to [ElevenLabs Studio](https://elevenlabs.io/studio) and [Sesame CSM-1B](https://github.com/SesameAILabs/csm).
|
22 |
|
|
|
23 |
- Join our [discord server](https://discord.gg/pgdB5YRe) for community support and access to new features.
|
24 |
- Play with a larger version of Dia: generate fun conversations, remix content, and share with friends. 🔮 Join the [waitlist](https://tally.so/r/meokbo) for early access.
|
25 |
|
@@ -27,6 +28,13 @@ We also provide a [demo page](https://yummy-fir-7a4.notion.site/dia) comparing o
|
|
27 |
|
28 |
This will open a Gradio UI that you can work on.
|
29 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
30 |
```bash
|
31 |
git clone https://github.com/nari-labs/dia.git
|
32 |
cd dia
|
@@ -36,6 +44,16 @@ pip install uv
|
|
36 |
uv run app.py
|
37 |
```
|
38 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
39 |
## ⚙️ Usage
|
40 |
|
41 |
### As a Python Library
|
@@ -63,7 +81,7 @@ Dia has been tested on only GPUs (pytorch 2.0+, CUDA 12.6). CPU support is to be
|
|
63 |
The initial run will take longer as the Descript Audio Codec also needs to be downloaded.
|
64 |
|
65 |
On enterprise GPUs, Dia can generate audio in real-time. On older GPUs, inference time will be slower.
|
66 |
-
For reference, on a A4000 GPU, Dia
|
67 |
`torch.compile` will increase speeds for supported GPUs.
|
68 |
|
69 |
The full version of Dia requires around 10GB of VRAM to run. We will be adding a quantized version in the future.
|
@@ -76,7 +94,7 @@ This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENS
|
|
76 |
|
77 |
## ⚠️ Disclaimer
|
78 |
|
79 |
-
This project offers a high-fidelity speech generation model intended
|
80 |
|
81 |
- **Identity Misuse**: Do not produce audio resembling real individuals without permission.
|
82 |
- **Deceptive Content**: Do not use this model to generate misleading content (e.g. fake news)
|
@@ -99,5 +117,6 @@ Join our [Discord Server](https://discord.gg/pgdB5YRe) for discussions.
|
|
99 |
|
100 |
- We thank the [Google TPU Research Cloud program](https://sites.research.google/trc/about/) for providing computation resources.
|
101 |
- Our work was heavily inspired by [SoundStorm](https://arxiv.org/abs/2305.09636), [Parakeet](https://jordandarefsky.com/blog/2024/parakeet/), and [Descript Audio Codec](https://github.com/descriptinc/descript-audio-codec).
|
|
|
102 |
- "Nari" is a pure Korean word for lily.
|
103 |
-
- We thank Jason Y. for providing help with data filtering.
|
|
|
16 |
|
17 |
Dia **directly generates highly realistic dialogue from a transcript**. You can condition the output on audio, enabling emotion and tone control. The model can also produce nonverbal communications like laughter, coughing, clearing throat, etc.
|
18 |
|
19 |
+
To accelerate research, we are providing access to pretrained model checkpoints and inference code. The model weights are hosted on [Hugging Face](https://huggingface.co/nari-labs/Dia-1.6B). The model only supports English generation at the moment.
|
20 |
|
21 |
We also provide a [demo page](https://yummy-fir-7a4.notion.site/dia) comparing our model to [ElevenLabs Studio](https://elevenlabs.io/studio) and [Sesame CSM-1B](https://github.com/SesameAILabs/csm).
|
22 |
|
23 |
+
- (Update) We have a ZeroGPU Space running! Try it now [here](https://huggingface.co/spaces/nari-labs/Dia-1.6B). Thanks to the HF team for the support :)
|
24 |
- Join our [discord server](https://discord.gg/pgdB5YRe) for community support and access to new features.
|
25 |
- Play with a larger version of Dia: generate fun conversations, remix content, and share with friends. 🔮 Join the [waitlist](https://tally.so/r/meokbo) for early access.
|
26 |
|
|
|
28 |
|
29 |
This will open a Gradio UI that you can work on.
|
30 |
|
31 |
+
```bash
|
32 |
+
git clone https://github.com/nari-labs/dia.git
|
33 |
+
cd dia && uv run app.py
|
34 |
+
```
|
35 |
+
|
36 |
+
or if you do not have `uv` pre-installed:
|
37 |
+
|
38 |
```bash
|
39 |
git clone https://github.com/nari-labs/dia.git
|
40 |
cd dia
|
|
|
44 |
uv run app.py
|
45 |
```
|
46 |
|
47 |
+
Note that the model was not fine-tuned on a specific voice. Hence, you will get different voices every time you run the model.
|
48 |
+
You can keep speaker consistency by either adding an audio prompt (a guide coming VERY soon - try it with the second example on Gradio for now), or fixing the seed.
|
49 |
+
|
50 |
+
## Features
|
51 |
+
|
52 |
+
- Generate dialogue via `[S1]` and `[S2]` tag
|
53 |
+
- Generate non-verbal like `(laughs)`, `(coughs)`, etc.
|
54 |
+
- Voice cloning. See [`example/voice_clone.py`](example/voice_clone.py) for more information.
|
55 |
+
- In the Hugging Face space, you can upload the audio you want to clone and place its transcript before your script. Make sure the transcript follows the required format. The model will then output only the content of your script.
|
56 |
+
|
57 |
## ⚙️ Usage
|
58 |
|
59 |
### As a Python Library
|
|
|
81 |
The initial run will take longer as the Descript Audio Codec also needs to be downloaded.
|
82 |
|
83 |
On enterprise GPUs, Dia can generate audio in real-time. On older GPUs, inference time will be slower.
|
84 |
+
For reference, on a A4000 GPU, Dia roughly generates 40 tokens/s (86 tokens equals 1 second of audio).
|
85 |
`torch.compile` will increase speeds for supported GPUs.
|
86 |
|
87 |
The full version of Dia requires around 10GB of VRAM to run. We will be adding a quantized version in the future.
|
|
|
94 |
|
95 |
## ⚠️ Disclaimer
|
96 |
|
97 |
+
This project offers a high-fidelity speech generation model intended for research and educational use. The following uses are **strictly forbidden**:
|
98 |
|
99 |
- **Identity Misuse**: Do not produce audio resembling real individuals without permission.
|
100 |
- **Deceptive Content**: Do not use this model to generate misleading content (e.g. fake news)
|
|
|
117 |
|
118 |
- We thank the [Google TPU Research Cloud program](https://sites.research.google/trc/about/) for providing computation resources.
|
119 |
- Our work was heavily inspired by [SoundStorm](https://arxiv.org/abs/2305.09636), [Parakeet](https://jordandarefsky.com/blog/2024/parakeet/), and [Descript Audio Codec](https://github.com/descriptinc/descript-audio-codec).
|
120 |
+
- HuggingFace for providing the ZeroGPU Grant.
|
121 |
- "Nari" is a pure Korean word for lily.
|
122 |
+
- We thank Jason Y. for providing help with data filtering.
|