qwertyuiopasdfg/model002_16bit
2B
•
Updated
•
2
aaa
qwertyuiopasdfg
AI & ML interests
None yet
Recent Activity
updated
a model
4 days ago
qwertyuiopasdfg/model002_16bit
published
a model
4 days ago
qwertyuiopasdfg/model002_16bit
updated
a dataset
6 days ago
qwertyuiopasdfg/temp0001
Organizations
None yet
updated
a
model
4 days ago
published
a
model
4 days ago
updated
a
dataset
6 days ago
published
a
dataset
6 days ago
updated
a
model
6 days ago
published
a
model
6 days ago
liked
a
model
11 days ago
liked
a
model
14 days ago
liked
a
model
17 days ago
liked
a
model
18 days ago
liked
2
models
21 days ago
liked
a
Space
21 days ago
liked
a
model
23 days ago
liked
a
model
24 days ago
liked
a
model
25 days ago
reacted to
Ruurd's
post with 👀
25 days ago
Post
2264
The past year I have been trying to get diffusion models to work for language generation, without having to retrain a LLM from scratch. And recently, we finally succeeded:
We introduce "LAD: LoRA-Adapted Denoiser", a method to convert a LLaMA model into a text diffusion model using LoRA finetuning and structured input corruption.
🎯 Try the demo and read the write-up here!
https://ruurdkuiper.github.io/tini-lad/
Unlike autoregressive (word-for-word) models like ChatGPT, diffusion models iteratively refine a noised sequence. However, most current diffusion approaches rely on all-parameter retraining and repeatedly remasking tokens, which is costly and slow during both training and inference!
🧠 With LAD:
- We can finetune an autoregressive model for diffusive generation in just 10 hours on a single GPU.
- Test-time compute is fully adjustable: fewer steps means faster outputs while more steps improve output quality.
- Due to our unique noising schedule, remasking is not always needed during inference. All tokens are attended to in each iteration!
🔍 LAD is built using:
– A frozen LLaMA-8B backbone
– Structured noising: token swaps, duplications, replacements, span shifts
– Modified attention masks for bidirectional decoding
💡 We show that even small, fast-trained models can perform diffusive generation — with competitive benchmark performance, perplexity and more flexible test-time behavior than traditional transformers.
We introduce "LAD: LoRA-Adapted Denoiser", a method to convert a LLaMA model into a text diffusion model using LoRA finetuning and structured input corruption.
🎯 Try the demo and read the write-up here!
https://ruurdkuiper.github.io/tini-lad/
Unlike autoregressive (word-for-word) models like ChatGPT, diffusion models iteratively refine a noised sequence. However, most current diffusion approaches rely on all-parameter retraining and repeatedly remasking tokens, which is costly and slow during both training and inference!
🧠 With LAD:
- We can finetune an autoregressive model for diffusive generation in just 10 hours on a single GPU.
- Test-time compute is fully adjustable: fewer steps means faster outputs while more steps improve output quality.
- Due to our unique noising schedule, remasking is not always needed during inference. All tokens are attended to in each iteration!
🔍 LAD is built using:
– A frozen LLaMA-8B backbone
– Structured noising: token swaps, duplications, replacements, span shifts
– Modified attention masks for bidirectional decoding
💡 We show that even small, fast-trained models can perform diffusive generation — with competitive benchmark performance, perplexity and more flexible test-time behavior than traditional transformers.
- 2 replies
upvoted
a
paper
27 days ago
liked
2
models
28 days ago
