Cool stuff @fffiloni ! Happy to see how we could connect it to MusicLang (https://huggingface.co/musiclang) to add fine-grained control over the music generation for the next version :)
Mehdi Zatar
MehdiLeZ
AI & ML interests
None yet
Organizations
reacted to
fffiloni's
post with 👍
about 1 year ago
Post
I'm happy to announce that ✨ Image to Music v2 ✨ is ready for you to try and i hope you'll like it too ! 😌
This new version has been crafted with transparency in mind,
so you can understand the process of translating an image to a musical equivalent.
How does it works under the hood ? 🤔
First, we get a very literal caption from microsoft/kosmos-2-patch14-224; this caption is then given to a LLM Agent (currently HuggingFaceH4/zephyr-7b-beta )which task is to translate the image caption to a musical and inspirational prompt for the next step.
Once we got a nice musical text from the LLM, we can send it to the text-to-music model of your choice:
MAGNet, MusicGen, AudioLDM-2, Riffusion or Mustango
Instead of the previous version of Image to Music which used Mubert API, and could output curious and obscure combinations, we only provide open sourced models available on the hub, called via the gradio API.
Also i guess the music result should be more accurate to the atmosphere of the image input, thanks to the LLM Agent step.
Pro tip, you can adjust the inspirational prompt to match your expectations, according to the chosen model and specific behavior of each one 👌
Try it, explore different models and tell me which one is your favorite 🤗
—› fffiloni/image-to-music-v2
This new version has been crafted with transparency in mind,
so you can understand the process of translating an image to a musical equivalent.
How does it works under the hood ? 🤔
First, we get a very literal caption from microsoft/kosmos-2-patch14-224; this caption is then given to a LLM Agent (currently HuggingFaceH4/zephyr-7b-beta )which task is to translate the image caption to a musical and inspirational prompt for the next step.
Once we got a nice musical text from the LLM, we can send it to the text-to-music model of your choice:
MAGNet, MusicGen, AudioLDM-2, Riffusion or Mustango
Instead of the previous version of Image to Music which used Mubert API, and could output curious and obscure combinations, we only provide open sourced models available on the hub, called via the gradio API.
Also i guess the music result should be more accurate to the atmosphere of the image input, thanks to the LLM Agent step.
Pro tip, you can adjust the inspirational prompt to match your expectations, according to the chosen model and specific behavior of each one 👌
Try it, explore different models and tell me which one is your favorite 🤗
—› fffiloni/image-to-music-v2
- 9 replies
replied to
their
post
about 1 year ago
Hi @asigalov61
Thanks for your feedback! Any help or contribution of course welcomed, so don't hesitate to elaborate further :-) Happy to discuss this.
All the best,
Mehdi
reacted to
their
post with ❤️
over 1 year ago
Post
Dear music lovers 🕺,
MusicLang Space is now live: musiclang/README
MusicLang is a controllable model for music generation:
> 🦙 Discover the LLAMA2 architecture, trained from scratch for symbolic music generation, ensuring exceptional quality;
> 👨🎨 Unleash your creativity by extending an existing music, or create new ones from scratch;
> 🤖 Integrate MusicLang into your applications, with an inference optimized for CPUs written in C, other integrations and optimizations coming soon.
In the space, you’ll find :
1️⃣ MusicLang foundation model: our fondation model for creating and generating original midi soundtracks musiclang/musiclang-v2;
2️⃣ MusicLang predict: our AI prediction api of the MusicLang package https://github.com/musiclang/musiclang_predict?tab=readme-ov-file;
3️⃣ MusicLang Language:a new language for tonal music. This language allows composers to load, write, transform and predict symbolic music in a simple, condensed and high level manner https://github.com/MusicLang/musiclang;
4️⃣ MusicLang Demo Space: musiclang/musiclang-predict
5️⃣ Our Colab: https://colab.research.google.com/drive/1MA2mek826c05BjbWk2nRkVv2rW7kIU_S?usp=sharing
Help us share the future of music composition! Spread the word, show your support by adding a star or contribute to our project. ⭐️✨
Music Sounds Definitely Better with You 🎶 🖤
cc @floriangardin @MehdiLeZ @reach-vb
Thanks a lot,
The MusicLang team ❤️
MusicLang Space is now live: musiclang/README
MusicLang is a controllable model for music generation:
> 🦙 Discover the LLAMA2 architecture, trained from scratch for symbolic music generation, ensuring exceptional quality;
> 👨🎨 Unleash your creativity by extending an existing music, or create new ones from scratch;
> 🤖 Integrate MusicLang into your applications, with an inference optimized for CPUs written in C, other integrations and optimizations coming soon.
In the space, you’ll find :
1️⃣ MusicLang foundation model: our fondation model for creating and generating original midi soundtracks musiclang/musiclang-v2;
2️⃣ MusicLang predict: our AI prediction api of the MusicLang package https://github.com/musiclang/musiclang_predict?tab=readme-ov-file;
3️⃣ MusicLang Language:a new language for tonal music. This language allows composers to load, write, transform and predict symbolic music in a simple, condensed and high level manner https://github.com/MusicLang/musiclang;
4️⃣ MusicLang Demo Space: musiclang/musiclang-predict
5️⃣ Our Colab: https://colab.research.google.com/drive/1MA2mek826c05BjbWk2nRkVv2rW7kIU_S?usp=sharing
Help us share the future of music composition! Spread the word, show your support by adding a star or contribute to our project. ⭐️✨
Music Sounds Definitely Better with You 🎶 🖤
cc @floriangardin @MehdiLeZ @reach-vb
Thanks a lot,
The MusicLang team ❤️
-
- 8 replies
replied to
their
post
over 1 year ago
Thanks @reach-vb ! Next for us :
- Adding control over the instruments used in each bar and their properties (note density, pitch range, average velocity);
- Some performances improvements over the inference C script;
- A faster distilled model for real-time generation that can be embedded in plugins or mobile applications;
- Some specialized smaller models depending on our users' needs;
- & more, so stay tuned! ❤️
posted
an
update
over 1 year ago
Post
Dear music lovers 🕺,
MusicLang Space is now live: musiclang/README
MusicLang is a controllable model for music generation:
> 🦙 Discover the LLAMA2 architecture, trained from scratch for symbolic music generation, ensuring exceptional quality;
> 👨🎨 Unleash your creativity by extending an existing music, or create new ones from scratch;
> 🤖 Integrate MusicLang into your applications, with an inference optimized for CPUs written in C, other integrations and optimizations coming soon.
In the space, you’ll find :
1️⃣ MusicLang foundation model: our fondation model for creating and generating original midi soundtracks musiclang/musiclang-v2;
2️⃣ MusicLang predict: our AI prediction api of the MusicLang package https://github.com/musiclang/musiclang_predict?tab=readme-ov-file;
3️⃣ MusicLang Language:a new language for tonal music. This language allows composers to load, write, transform and predict symbolic music in a simple, condensed and high level manner https://github.com/MusicLang/musiclang;
4️⃣ MusicLang Demo Space: musiclang/musiclang-predict
5️⃣ Our Colab: https://colab.research.google.com/drive/1MA2mek826c05BjbWk2nRkVv2rW7kIU_S?usp=sharing
Help us share the future of music composition! Spread the word, show your support by adding a star or contribute to our project. ⭐️✨
Music Sounds Definitely Better with You 🎶 🖤
cc @floriangardin @MehdiLeZ @reach-vb
Thanks a lot,
The MusicLang team ❤️
MusicLang Space is now live: musiclang/README
MusicLang is a controllable model for music generation:
> 🦙 Discover the LLAMA2 architecture, trained from scratch for symbolic music generation, ensuring exceptional quality;
> 👨🎨 Unleash your creativity by extending an existing music, or create new ones from scratch;
> 🤖 Integrate MusicLang into your applications, with an inference optimized for CPUs written in C, other integrations and optimizations coming soon.
In the space, you’ll find :
1️⃣ MusicLang foundation model: our fondation model for creating and generating original midi soundtracks musiclang/musiclang-v2;
2️⃣ MusicLang predict: our AI prediction api of the MusicLang package https://github.com/musiclang/musiclang_predict?tab=readme-ov-file;
3️⃣ MusicLang Language:a new language for tonal music. This language allows composers to load, write, transform and predict symbolic music in a simple, condensed and high level manner https://github.com/MusicLang/musiclang;
4️⃣ MusicLang Demo Space: musiclang/musiclang-predict
5️⃣ Our Colab: https://colab.research.google.com/drive/1MA2mek826c05BjbWk2nRkVv2rW7kIU_S?usp=sharing
Help us share the future of music composition! Spread the word, show your support by adding a star or contribute to our project. ⭐️✨
Music Sounds Definitely Better with You 🎶 🖤
cc @floriangardin @MehdiLeZ @reach-vb
Thanks a lot,
The MusicLang team ❤️
-
-
-
- 8 replies