Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion

Paper | Website | Models (HuggingFace) | Code

Self Forcing trains autoregressive video diffusion models by simulating the inference process during training, performing autoregressive rollout with KV caching. It resolves the train-test distribution mismatch and enables real-time, streaming video generation on a single RTX 4090 while matching the quality of state-of-the-art diffusion models.

Requirements

We tested this repo on the following setup:

Nvidia GPU with at least 24 GB memory (RTX 4090, A100, and H100 are tested).
Linux operating system.
64 GB RAM.

Other hardware setup could also work but hasn't been tested.

Installation

Create a conda environment and install dependencies:

conda create -n self_forcing python=3.10 -y
conda activate self_forcing
pip install -r requirements.txt
pip install flash-attn --no-build-isolation
python setup.py develop

Quick Start

Download checkpoints

huggingface-cli download Wan-AI/Wan2.1-T2V-1.3B --local-dir-use-symlinks False --local-dir wan_models/Wan2.1-T2V-1.3B
huggingface-cli download gdhe17/Self-Forcing checkpoints/self_forcing_dmd.pt --local-dir .

GUI demo

python demo.py

Note:

Our model works better with long, detailed prompts since it's trained with such prompts. We will integrate prompt extension into the codebase (similar to Wan2.1) in the future. For now, it is recommended to use third-party LLMs (such as GPT-4o) to extend your prompt before providing to the model.
You may want to adjust FPS so it plays smoothly on your device.
The speed can be improved by enabling torch.compile, TAEHV-VAE, or using FP8 Linear layers, although the latter two options may sacrifice quality. It is recommended to use torch.compile if possible and enable TAEHV-VAE if further speedup is needed.

CLI Inference

Example inference script using the chunk-wise autoregressive checkpoint trained with DMD:

python inference.py \
    --config_path configs/self_forcing_dmd.yaml \
    --output_folder videos/self_forcing_dmd \
    --checkpoint_path checkpoints/self_forcing_dmd.pt \
    --data_path prompts/MovieGenVideoBench_extended.txt \
    --use_ema

Other config files and corresponding checkpoints can be found in configs folder and our huggingface repo.

Training

Download text prompts and ODE initialized checkpoint

huggingface-cli download gdhe17/Self-Forcing checkpoints/ode_init.pt --local-dir .
huggingface-cli download gdhe17/Self-Forcing vidprom_filtered_extended.txt --local-dir prompts

Note: Our training algorithm (except for the GAN version) is data-free (no video data is needed). For now, we directly provide the ODE initialization checkpoint and will add more instructions on how to perform ODE initialization in the future (which is identical to the process described in the CausVid repo).

Self Forcing Training with DMD

torchrun --nnodes=8 --nproc_per_node=8 --rdzv_id=5235 \
  --rdzv_backend=c10d \
  --rdzv_endpoint $MASTER_ADDR \
  train.py \
  --config_path configs/self_forcing_dmd.yaml \
  --logdir logs/self_forcing_dmd \
  --disable-wandb

Our training run uses 600 iterations and completes in under 2 hours using 64 H100 GPUs. By implementing gradient accumulation, it should be possible to reproduce the results in less than 16 hours using 8 H100 GPUs.

Acknowledgements

This codebase is built on top of the open-source implementation of CausVid by Tianwei Yin and the Wan2.1 repo.

Citation

If you find this codebase useful for your research, please kindly cite our paper:

@article{huang2025selfforcing,
  title={Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion},
  author={Huang, Xun and Li, Zhengqi and He, Guande and Zhou, Mingyuan and Shechtman, Eli},
  journal={arXiv preprint arXiv:2506.08009},
  year={2025}
}

gdhe17
/

Self-Forcing