Burf commited on
Commit
4f75903
·
1 Parent(s): e600401

Update readme

Browse files
Files changed (3) hide show
  1. .gitattributes +1 -0
  2. README.md +92 -0
  3. teaser.png +3 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ *.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,3 +1,95 @@
1
  ---
2
  license: mit
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
+ language:
4
+ - en
5
+ library_name: diffusers
6
+ tags:
7
+ - text-to-image
8
+ - personalization
9
+ - adapter
10
+ - stable-diffusion
11
+ - flux
12
+ - diffusers
13
+ base_model:
14
+ - runwayml/stable-diffusion-v1-5
15
+ - stabilityai/stable-diffusion-2-1
16
+ - stabilityai/stable-diffusion-xl-base-1.0
17
+ - stabilityai/stable-diffusion-3.5-large
18
+ - black-forest-labs/FLUX.1-dev
19
+ pipeline_tag: text-to-image
20
  ---
21
+
22
+
23
+ # DrUM (**D**raw **You**r **M**ind)
24
+
25
+ **DrUM** enables **personalized text-to-image (T2I) generation by integrating reference prompts** into T2I diffusion models. It works with **foundation T2I models such as Stable Diffusion v1/v2/XL/v3 and FLUX**, without requiring additional fine-tuning. DrUM leverages **condition-level modeling in the latent space using a transformer-based adapter**, and integrates seamlessly with **open-source text encoders such as OpenCLIP and Google T5**.
26
+
27
+ This repository provides the necessary components to run DrUM for **inference**. For the full source code, training scripts, and detailed documentation, please visit our official **[GitHub repository](https://github.com/Burf/DrUM)** and read the **[research paper](https://arxiv.org/abs/2508.03481)**.
28
+
29
+ <p align="center">
30
+ <img src="teaser.png" width="95%">
31
+ </p>
32
+
33
+
34
+ ## Quickstart
35
+
36
+ This model is designed for easy use with the `diffusers` library as a custom pipeline.
37
+
38
+ ### Installation
39
+
40
+ ```bash
41
+ pip install torch torchvision diffusers transformers accelerate safetensors huggingface-hub
42
+ ```
43
+
44
+ ### Usage
45
+
46
+ ```python
47
+ import torch
48
+
49
+ from diffusers import DiffusionPipeline
50
+ from pipeline import DrUM
51
+
52
+ # Load pipeline and attach DrUM
53
+ #drum = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", custom_pipeline = "Burf/DrUM", pipeline = "runwayml/stable-diffusion-v1-5", torch_dtype = torch.bfloat16, device = "cuda")
54
+ pipeline = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype = torch.bfloat16).to("cuda")
55
+ drum = DrUM(pipeline)
56
+
57
+ # Generate personalized images
58
+ images = drum(
59
+ prompt = "a photograph of an astronaut riding a horse",
60
+ ref = ["A retro-futuristic space exploration movie poster with bold, vibrant colors"],
61
+ weight = [1.0],
62
+ alpha = 0.3
63
+ )
64
+
65
+ images[0].save("personalized_image.png")
66
+ ```
67
+
68
+
69
+ ## Supported foundation T2I models
70
+
71
+ DrUM works with a wide variety of foundation T2I models that uses text encoders with same weights:
72
+
73
+ | Architecture | Pipeline | Text encoder | DrUM weight |
74
+ |--------------|----------------|-|-------------|
75
+ | Stable Diffusion v1 | `runwayml/stable-diffusion-v1-5`, `prompthero/openjourney-v4`,<br>`stablediffusionapi/realistic-vision-v51`,`stablediffusionapi/deliberate-v2`,<br>`stablediffusionapi/anything-v5`, `WarriorMama777/AbyssOrangeMix2`, ... | `openai/clip-vit-large-patch14` | `L.safetensors` |
76
+ | Stable Diffusion v2 | `stabilityai/stable-diffusion-2-1`, ... | `openai/clip-vit-huge-patch14` | `H.safetensors` |
77
+ | Stable Diffusion XL | `stabilityai/stable-diffusion-xl-base-1.0`, ... | `openai/clip-vit-large-patch14`,<br>`laion/CLIP-ViT-bigG-14-laion2B-39B-b160k` | `L.safetensors`,<br>`bigG.safetensors` |
78
+ | Stable Diffusion v3 | `stabilityai/stable-diffusion-3.5-large`<br>`stabilityai/stable-diffusion-3.5-medium`, ... | `openai/clip-vit-large-patch14`,<br>`laion/CLIP-ViT-bigG-14-laion2B-39B-b160k`,<br>`google/t5-v1_1-xxl` | `L.safetensors`,<br>`bigG.safetensors`,<br>`T5.safetensors` |
79
+ | FLUX | `black-forest-labs/FLUX.1-dev`, ... | `openai/clip-vit-large-patch14`,<br>`google/t5-v1_1-xxl` | `L.safetensors`<br>`T5.safetensors` |
80
+
81
+
82
+ ## Citation
83
+
84
+ ```
85
+ @inproceedings{kim2025drum,
86
+ title={Draw Your Mind: Personalized Generation via Condition-Level Modeling in Text-to-Image Diffusion Models},
87
+ author={Hyungjin Kim, Seokho Ahn, and Young-Duk Seo},
88
+ booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
89
+ year={2025}
90
+ }
91
+ ```
92
+
93
+ ## License
94
+
95
+ This project is licensed under the MIT License.
teaser.png ADDED

Git LFS Details

  • SHA256: 3c8b8e866451821cef200a87ea37b7700218ffb8a803055e234fbe794e5fe6d5
  • Pointer size: 132 Bytes
  • Size of remote file: 1.31 MB