[//]: # ()
[//]: # (
)
[//]: # (
If you like our project, please give us a star β on GitHub for the latest update.
This repository contains the official implementation of AvatarArtist, a method for generating 4D avatars from a single image.
π‘ We also have other avatar projects that may interest you β¨.
> **[HeadArtist: Text-conditioned 3D Head Generation with Self Score Distillation, SIGGRAPH 2024](https://arxiv.org/abs/2312.07539)**
> Hongyu Liu, Xuan Wang, Ziyu Wan, etc.
>
> 
>
> 
>
> **[Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation, SIGGRAPH Asia 2024](https://arxiv.org/abs/2406.01900)**
> Yue Ma, Hongyu Liu, Hongfa Wang, etc.
>
>
> 
## π¨ News
- [03/26/2025] Inference Code and pretrained models released!
## βοΈ Setup
### Environment
```bash
git clone --depth=1 https://github.com/ant-research/AvatarArtist
cd AvatarArtist
conda create -n avatarartist python=3.9.0
conda activate avatarartist
pip install -r requirements.txt
```
### Download Weights
The weights are available at [π€HuggingFace](https://huggingface.co/KumaPower/AvatarArtist), you can download it with the following commands. Please move the required files into the `pretrained_model` directory:
```bash
# if you are in china mainland, run this first: export HF_ENDPOINT=https://hf-mirror.com
huggingface-cli download --repo-type model \
KUMAPOWER/AvatarArtist \
--local-dir pretrained_model
```
## π€ Usage
### Inference
Our approach consists of two steps during the inference process. First, the DiT model generates a 4D representation based on the input image. Then, our Motion-Aware Cross-Domain Renderer takes this 4D representation as input and, guided by both the input image and driving signals, renders it into the final target image.
This is an example of inference using the demo data. The images used in this example are sourced from https://civitai.com/.
```python
python3 inference.py \
--img_file './demo_data/source_img/img_from_web/images512x512/final_ipimgs' \
--input_img_fvid './demo_data/source_img/img_from_web/coeffs/final_ipimgs' \
--input_img_motion './demo_data/source_img/img_from_web/motions/final_ipimgs' \
--video_name 'Obama' \
--target_path './demo_data/target_video/data_obama'
# --use_demo_cam (create a video like the teaser using predefined camera parameters)
```
This is an example of performing inference using the model. The images used in this example are diverse-domain images generated by a diffusion model, as described in our paper. You can use the --select_img option to specify a particular input image.
```python
python3 inference.py \
--img_file './demo_data/source_img/img_generate_different_domain/images512x512/demo_imgs' \
--input_img_fvid './demo_data/img_generate_different_domain/coeffs/demo_imgs' \
--input_img_motion './demo_data/source_img/img_generate_different_domain/motions/demo_imgs' \
--video_name "Obama" \
--target_path './demo_data/target_video/data_obama' \
--select_img 'your_selected_image.png in img_file'
```
### Custom Data Processing
We provide a set of scripts to process input images and videos for use with our model. These scripts ensure that the data is properly formatted and preprocessed, making it compatible with our inference pipeline. You can use them to prepare your own custom data for generating results with our model.
Please refer to [this guide](https://github.com/ant-research/AvatarArtist/tree/main/data_process) to learn how to obtain the inference data. You can also check the [demo data](https://github.com/ant-research/HeadArtist/tree/main/demo_data) for reference. The data structure is shown below.
The files in the `"dataset"` folder serve as the final input to the model, while the other files are intermediate outputs from the data processing pipelineοΌ
```
π¦ datasets/
βββ π dataset/
β βββ π coeffs/
β βββ π images512x512/
β βββ π uvRender256x256/
β βββ π orthRender256x256_face_eye/
β βββ π motions/
βββ π crop_fv_tracking/
βββ π realign_detections/
βββ π realign_detections/
βββ π realign/
βββ π raw_detection/
βββ π align_3d_landmark/
βββ π raw_frames/
```
### Different domain's input images generation
We provide a set of scripts to transfer the realistic domain's portrait to the other domain. Please refer to [this guide](https://github.com/ant-research/AvatarArtist/tree/main/different_domain_imge_gen).
## **π To-Do List**
### **Pending Tasks**
- [ ] Gradio demo
- [ ] Release training code
---
### **β
Completed Tasks**
- [x] Release inference code
- [x] Release data processing tools
- [x] Release the pipeline to generate input for different domains
## π Credits
We sincerely appreciate the contributions of the following open-source projects, which have significantly influenced our work:
- **DiT** builds upon [PixArt-alpha](https://github.com/PixArt-alpha/PixArt-alpha).
- **VAE** is based on [LVDM](https://github.com/YingqingHe/LVDM).
- **Motion-aware rendering** is inspired by [Portrait4D](https://github.com/YuDeng/Portrait-4D).
- **4D representation** in our paper is proposed in [Next3D](https://github.com/MrTornado24/Next3D) and [Next3D++](https://github.com/XChenZ/invertAvatar).
- We referenced [DATID3D](https://github.com/gwang-kim/DATID-3D) for domain-specific prompts.
## π License
* The majority of this project is released under the Apache 2.0 license as found in the [LICENSE](LICENSE) file.
## βοΈ Citation
If you make use of our work, please cite our paper.
```bibtex
@article{liu2025avatarartist,
title={AvatarArtist: Open-Domain 4D Avatarization},
author={Hongyu Liu, Xuan Wang, Ziyu Wan, Yue Ma, Jingye Chen, Yanbo Fan, Yujun Shen, Yibing Song, Qifeng Chen },
booktitle={CVPR},
year={2025}
}
```