Update readme
Browse files- .gitattributes +1 -0
- README.md +92 -0
- teaser.png +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
*.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -1,3 +1,95 @@
|
|
| 1 |
---
|
| 2 |
license: mit
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
library_name: diffusers
|
| 6 |
+
tags:
|
| 7 |
+
- text-to-image
|
| 8 |
+
- personalization
|
| 9 |
+
- adapter
|
| 10 |
+
- stable-diffusion
|
| 11 |
+
- flux
|
| 12 |
+
- diffusers
|
| 13 |
+
base_model:
|
| 14 |
+
- runwayml/stable-diffusion-v1-5
|
| 15 |
+
- stabilityai/stable-diffusion-2-1
|
| 16 |
+
- stabilityai/stable-diffusion-xl-base-1.0
|
| 17 |
+
- stabilityai/stable-diffusion-3.5-large
|
| 18 |
+
- black-forest-labs/FLUX.1-dev
|
| 19 |
+
pipeline_tag: text-to-image
|
| 20 |
---
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
# DrUM (**D**raw **You**r **M**ind)
|
| 24 |
+
|
| 25 |
+
**DrUM** enables **personalized text-to-image (T2I) generation by integrating reference prompts** into T2I diffusion models. It works with **foundation T2I models such as Stable Diffusion v1/v2/XL/v3 and FLUX**, without requiring additional fine-tuning. DrUM leverages **condition-level modeling in the latent space using a transformer-based adapter**, and integrates seamlessly with **open-source text encoders such as OpenCLIP and Google T5**.
|
| 26 |
+
|
| 27 |
+
This repository provides the necessary components to run DrUM for **inference**. For the full source code, training scripts, and detailed documentation, please visit our official **[GitHub repository](https://github.com/Burf/DrUM)** and read the **[research paper](https://arxiv.org/abs/2508.03481)**.
|
| 28 |
+
|
| 29 |
+
<p align="center">
|
| 30 |
+
<img src="teaser.png" width="95%">
|
| 31 |
+
</p>
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
## Quickstart
|
| 35 |
+
|
| 36 |
+
This model is designed for easy use with the `diffusers` library as a custom pipeline.
|
| 37 |
+
|
| 38 |
+
### Installation
|
| 39 |
+
|
| 40 |
+
```bash
|
| 41 |
+
pip install torch torchvision diffusers transformers accelerate safetensors huggingface-hub
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
### Usage
|
| 45 |
+
|
| 46 |
+
```python
|
| 47 |
+
import torch
|
| 48 |
+
|
| 49 |
+
from diffusers import DiffusionPipeline
|
| 50 |
+
from pipeline import DrUM
|
| 51 |
+
|
| 52 |
+
# Load pipeline and attach DrUM
|
| 53 |
+
#drum = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", custom_pipeline = "Burf/DrUM", pipeline = "runwayml/stable-diffusion-v1-5", torch_dtype = torch.bfloat16, device = "cuda")
|
| 54 |
+
pipeline = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype = torch.bfloat16).to("cuda")
|
| 55 |
+
drum = DrUM(pipeline)
|
| 56 |
+
|
| 57 |
+
# Generate personalized images
|
| 58 |
+
images = drum(
|
| 59 |
+
prompt = "a photograph of an astronaut riding a horse",
|
| 60 |
+
ref = ["A retro-futuristic space exploration movie poster with bold, vibrant colors"],
|
| 61 |
+
weight = [1.0],
|
| 62 |
+
alpha = 0.3
|
| 63 |
+
)
|
| 64 |
+
|
| 65 |
+
images[0].save("personalized_image.png")
|
| 66 |
+
```
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
## Supported foundation T2I models
|
| 70 |
+
|
| 71 |
+
DrUM works with a wide variety of foundation T2I models that uses text encoders with same weights:
|
| 72 |
+
|
| 73 |
+
| Architecture | Pipeline | Text encoder | DrUM weight |
|
| 74 |
+
|--------------|----------------|-|-------------|
|
| 75 |
+
| Stable Diffusion v1 | `runwayml/stable-diffusion-v1-5`, `prompthero/openjourney-v4`,<br>`stablediffusionapi/realistic-vision-v51`,`stablediffusionapi/deliberate-v2`,<br>`stablediffusionapi/anything-v5`, `WarriorMama777/AbyssOrangeMix2`, ... | `openai/clip-vit-large-patch14` | `L.safetensors` |
|
| 76 |
+
| Stable Diffusion v2 | `stabilityai/stable-diffusion-2-1`, ... | `openai/clip-vit-huge-patch14` | `H.safetensors` |
|
| 77 |
+
| Stable Diffusion XL | `stabilityai/stable-diffusion-xl-base-1.0`, ... | `openai/clip-vit-large-patch14`,<br>`laion/CLIP-ViT-bigG-14-laion2B-39B-b160k` | `L.safetensors`,<br>`bigG.safetensors` |
|
| 78 |
+
| Stable Diffusion v3 | `stabilityai/stable-diffusion-3.5-large`<br>`stabilityai/stable-diffusion-3.5-medium`, ... | `openai/clip-vit-large-patch14`,<br>`laion/CLIP-ViT-bigG-14-laion2B-39B-b160k`,<br>`google/t5-v1_1-xxl` | `L.safetensors`,<br>`bigG.safetensors`,<br>`T5.safetensors` |
|
| 79 |
+
| FLUX | `black-forest-labs/FLUX.1-dev`, ... | `openai/clip-vit-large-patch14`,<br>`google/t5-v1_1-xxl` | `L.safetensors`<br>`T5.safetensors` |
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
## Citation
|
| 83 |
+
|
| 84 |
+
```
|
| 85 |
+
@inproceedings{kim2025drum,
|
| 86 |
+
title={Draw Your Mind: Personalized Generation via Condition-Level Modeling in Text-to-Image Diffusion Models},
|
| 87 |
+
author={Hyungjin Kim, Seokho Ahn, and Young-Duk Seo},
|
| 88 |
+
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
|
| 89 |
+
year={2025}
|
| 90 |
+
}
|
| 91 |
+
```
|
| 92 |
+
|
| 93 |
+
## License
|
| 94 |
+
|
| 95 |
+
This project is licensed under the MIT License.
|
teaser.png
ADDED
|
Git LFS Details
|