Sayak Paul's picture

Sayak Paul

sayakpaul

·

https://sayak.dev

AI & ML interests

Diffusion models, representation learning

Recent Activity

posted an update 1 day ago

Fast LoRA inference for Flux with Diffusers and PEFT 🚨 There are great materials that demonstrate how to optimize inference for popular image generation models, such as Flux. However, very few cover how to serve LoRAs fast, despite LoRAs being an inseparable part of their adoption. In our latest post, @BenjaminB and I show different techniques to optimize LoRA inference for the Flux family of models for image generation. Our recipe includes the use of: 1. `torch.compile` 2. Flash Attention 3 (when compatible) 3. Dynamic FP8 weight quantization (when compatible) 4. Hotswapping for avoiding recompilation during swapping new LoRAs 🤯 We have tested our recipe with Flux.1-Dev on both H100 and RTX 4090. We achieve at least a *2x speedup* in either of the GPUs. We believe our recipe is grounded in the reality of how LoRA-based use cases are generally served. So, we hope this will be beneficial to the community 🤗 Even though our recipe was tested primarily with NVIDIA GPUs, it should also work with AMD GPUs. Learn the details and the full code here: https://huggingface.co/blog/lora-fast

commented on their article 1 day ago

Fast LoRA inference for Flux with Diffusers and PEFT

new activity 2 days ago

black-forest-labs/FLUX.1-Kontext-dev:pipe.to("cuda") runs super slow, is it expected?

View all activity

Organizations

upvoted an article 2 days ago

Article

State of open video generation models in Diffusers

By

and 2 others •

Jan 27

• 57

upvoted 2 articles 15 days ago

Article

Building the Hugging Face MCP Server

By

and 3 others •

16 days ago

• 50

Article

Reachy Mini - The Open-Source Robot for Today's and Tomorrow's AI Builders

By

and 1 other •

17 days ago

• 608

upvoted a paper 24 days ago

Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generation

Paper • 2506.19852 • Published Jun 24 • 38

upvoted 2 articles about 1 month ago

Article

(LoRA) Fine-Tuning FLUX.1-dev on Consumer Hardware

By

and 4 others •

Jun 19

• 79

Article

Mixture of Experts Explained

By

and 5 others •

Dec 11, 2023

• 768

upvoted 2 articles 2 months ago

Article

How to train a new language model from scratch using Transformers and Tokenizers

By

•

Feb 14, 2020

• 42

Article

Exploring Quantization Backends in Diffusers

By

and 2 others •

May 21

• 39

upvoted 2 articles 3 months ago

Article

Welcoming Llama Guard 4 on Hugging Face Hub

By

and 3 others •

Apr 29

• 40

Article

CinePile 2.0 - making stronger datasets with adversarial refinement

By

and 3 others •

Oct 23, 2024

• 18

upvoted an article 4 months ago

Article

Welcome Llama 4 Maverick & Scout on Hugging Face!

By

and 6 others •

Apr 5

• 146

upvoted 2 papers 4 months ago

A Refined Analysis of Massive Activations in LLMs

Paper • 2503.22329 • Published Mar 28 • 14

ZClip: Adaptive Spike Mitigation for LLM Pre-Training

Paper • 2504.02507 • Published Apr 3 • 90

upvoted a collection 4 months ago

SANA-1.5

SANA-1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer • 6 items • Updated Apr 17 • 6

upvoted an article 4 months ago

Article

Don't repeat yourself - 🤗 Transformers Design Philosophy

By

•

Apr 5, 2022

• 37

upvoted 3 articles 5 months ago

Article

Welcome Gemma 3: Google's all new multimodal, multilingual, long context open LLM

By

and 3 others •

Mar 12

• 447

Article

Fine-tuning Florence-2 - Microsoft's Cutting-edge Vision Language Models

By

and 2 others •

Jun 24, 2024

• 199

Article

Distilling from Dialogues: Finding Meaning in LLM Interactions

By

•

Feb 25

• 4

upvoted a collection 5 months ago

Remote VAE Inference Endpoints

Models and handler code used in https://huggingface.co/blog/remote_vae • 5 items • Updated Mar 10 • 6

upvoted an article 5 months ago

Article

Remote VAEs for decoding with HF endpoints 🤗

By

and 1 other •

Feb 24

• 41