Incompatible lora weight with ComfyUI

by Wakeme - opened 4 days ago

4 days ago

Hello, I'm trying to load the lora weight of the panoramic lora (image) with flux fill dev (fp8) using ComfyUI. However, I found that the lora does load at all, and I suspect that the name mapping is not compatible with the comfyui version as loaded_keys return an empty set. Is it possible to covert the lora weight into ComfyUI compatible naming / format?

The following are the lora key log

lora key not loaded: double_blocks.0.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.0.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.0.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.0.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.0.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.0.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.0.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.0.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.1.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.1.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.1.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.1.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.1.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.1.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.1.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.1.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.10.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.10.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.10.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.10.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.10.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.10.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.10.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.10.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.11.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.11.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.11.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.11.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.11.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.11.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.11.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.11.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.12.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.12.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.12.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.12.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.12.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.12.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.12.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.12.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.13.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.13.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.13.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.13.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.13.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.13.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.13.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.13.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.14.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.14.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.14.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.14.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.14.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.14.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.14.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.14.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.15.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.15.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.15.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.15.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.15.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.15.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.15.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.15.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.16.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.16.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.16.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.16.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.16.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.16.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.16.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.16.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.17.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.17.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.17.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.17.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.17.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.17.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.17.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.17.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.18.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.18.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.18.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.18.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.18.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.18.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.18.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.18.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.2.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.2.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.2.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.2.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.2.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.2.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.2.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.2.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.3.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.3.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.3.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.3.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.3.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.3.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.3.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.3.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.4.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.4.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.4.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.4.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.4.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.4.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.4.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.4.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.5.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.5.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.5.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.5.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.5.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.5.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.5.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.5.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.6.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.6.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.6.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.6.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.6.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.6.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.6.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.6.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.7.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.7.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.7.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.7.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.7.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.7.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.7.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.7.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.8.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.8.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.8.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.8.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.8.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.8.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.8.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.8.processor.qkv_lora2.up.weight
lora key not loaded: double_blocks.9.processor.proj_lora1.down.weight
lora key not loaded: double_blocks.9.processor.proj_lora1.up.weight
lora key not loaded: double_blocks.9.processor.proj_lora2.down.weight
lora key not loaded: double_blocks.9.processor.proj_lora2.up.weight
lora key not loaded: double_blocks.9.processor.qkv_lora1.down.weight
lora key not loaded: double_blocks.9.processor.qkv_lora1.up.weight
lora key not loaded: double_blocks.9.processor.qkv_lora2.down.weight
lora key not loaded: double_blocks.9.processor.qkv_lora2.up.weight
lora key not loaded: single_blocks.0.processor.proj_lora.down.weight
lora key not loaded: single_blocks.0.processor.proj_lora.up.weight
lora key not loaded: single_blocks.0.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.0.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.1.processor.proj_lora.down.weight
lora key not loaded: single_blocks.1.processor.proj_lora.up.weight
lora key not loaded: single_blocks.1.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.1.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.10.processor.proj_lora.down.weight
lora key not loaded: single_blocks.10.processor.proj_lora.up.weight
lora key not loaded: single_blocks.10.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.10.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.11.processor.proj_lora.down.weight
lora key not loaded: single_blocks.11.processor.proj_lora.up.weight
lora key not loaded: single_blocks.11.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.11.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.12.processor.proj_lora.down.weight
lora key not loaded: single_blocks.12.processor.proj_lora.up.weight
lora key not loaded: single_blocks.12.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.12.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.13.processor.proj_lora.down.weight
lora key not loaded: single_blocks.13.processor.proj_lora.up.weight
lora key not loaded: single_blocks.13.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.13.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.14.processor.proj_lora.down.weight
lora key not loaded: single_blocks.14.processor.proj_lora.up.weight
lora key not loaded: single_blocks.14.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.14.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.15.processor.proj_lora.down.weight
lora key not loaded: single_blocks.15.processor.proj_lora.up.weight
lora key not loaded: single_blocks.15.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.15.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.16.processor.proj_lora.down.weight
lora key not loaded: single_blocks.16.processor.proj_lora.up.weight
lora key not loaded: single_blocks.16.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.16.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.17.processor.proj_lora.down.weight
lora key not loaded: single_blocks.17.processor.proj_lora.up.weight
lora key not loaded: single_blocks.17.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.17.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.18.processor.proj_lora.down.weight
lora key not loaded: single_blocks.18.processor.proj_lora.up.weight
lora key not loaded: single_blocks.18.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.18.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.19.processor.proj_lora.down.weight
lora key not loaded: single_blocks.19.processor.proj_lora.up.weight
lora key not loaded: single_blocks.19.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.19.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.2.processor.proj_lora.down.weight
lora key not loaded: single_blocks.2.processor.proj_lora.up.weight
lora key not loaded: single_blocks.2.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.2.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.20.processor.proj_lora.down.weight
lora key not loaded: single_blocks.20.processor.proj_lora.up.weight
lora key not loaded: single_blocks.20.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.20.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.21.processor.proj_lora.down.weight
lora key not loaded: single_blocks.21.processor.proj_lora.up.weight
lora key not loaded: single_blocks.21.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.21.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.22.processor.proj_lora.down.weight
lora key not loaded: single_blocks.22.processor.proj_lora.up.weight
lora key not loaded: single_blocks.22.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.22.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.23.processor.proj_lora.down.weight
lora key not loaded: single_blocks.23.processor.proj_lora.up.weight
lora key not loaded: single_blocks.23.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.23.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.24.processor.proj_lora.down.weight
lora key not loaded: single_blocks.24.processor.proj_lora.up.weight
lora key not loaded: single_blocks.24.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.24.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.25.processor.proj_lora.down.weight
lora key not loaded: single_blocks.25.processor.proj_lora.up.weight
lora key not loaded: single_blocks.25.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.25.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.26.processor.proj_lora.down.weight
lora key not loaded: single_blocks.26.processor.proj_lora.up.weight
lora key not loaded: single_blocks.26.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.26.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.27.processor.proj_lora.down.weight
lora key not loaded: single_blocks.27.processor.proj_lora.up.weight
lora key not loaded: single_blocks.27.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.27.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.28.processor.proj_lora.down.weight
lora key not loaded: single_blocks.28.processor.proj_lora.up.weight
lora key not loaded: single_blocks.28.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.28.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.29.processor.proj_lora.down.weight
lora key not loaded: single_blocks.29.processor.proj_lora.up.weight
lora key not loaded: single_blocks.29.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.29.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.3.processor.proj_lora.down.weight
lora key not loaded: single_blocks.3.processor.proj_lora.up.weight
lora key not loaded: single_blocks.3.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.3.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.30.processor.proj_lora.down.weight
lora key not loaded: single_blocks.30.processor.proj_lora.up.weight
lora key not loaded: single_blocks.30.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.30.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.31.processor.proj_lora.down.weight
lora key not loaded: single_blocks.31.processor.proj_lora.up.weight
lora key not loaded: single_blocks.31.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.31.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.32.processor.proj_lora.down.weight
lora key not loaded: single_blocks.32.processor.proj_lora.up.weight
lora key not loaded: single_blocks.32.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.32.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.33.processor.proj_lora.down.weight
lora key not loaded: single_blocks.33.processor.proj_lora.up.weight
lora key not loaded: single_blocks.33.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.33.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.34.processor.proj_lora.down.weight
lora key not loaded: single_blocks.34.processor.proj_lora.up.weight
lora key not loaded: single_blocks.34.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.34.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.35.processor.proj_lora.down.weight
lora key not loaded: single_blocks.35.processor.proj_lora.up.weight
lora key not loaded: single_blocks.35.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.35.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.36.processor.proj_lora.down.weight
lora key not loaded: single_blocks.36.processor.proj_lora.up.weight
lora key not loaded: single_blocks.36.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.36.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.37.processor.proj_lora.down.weight
lora key not loaded: single_blocks.37.processor.proj_lora.up.weight
lora key not loaded: single_blocks.37.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.37.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.4.processor.proj_lora.down.weight
lora key not loaded: single_blocks.4.processor.proj_lora.up.weight
lora key not loaded: single_blocks.4.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.4.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.5.processor.proj_lora.down.weight
lora key not loaded: single_blocks.5.processor.proj_lora.up.weight
lora key not loaded: single_blocks.5.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.5.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.6.processor.proj_lora.down.weight
lora key not loaded: single_blocks.6.processor.proj_lora.up.weight
lora key not loaded: single_blocks.6.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.6.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.7.processor.proj_lora.down.weight
lora key not loaded: single_blocks.7.processor.proj_lora.up.weight
lora key not loaded: single_blocks.7.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.7.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.8.processor.proj_lora.down.weight
lora key not loaded: single_blocks.8.processor.proj_lora.up.weight
lora key not loaded: single_blocks.8.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.8.processor.qkv_lora.up.weight
lora key not loaded: single_blocks.9.processor.proj_lora.down.weight
lora key not loaded: single_blocks.9.processor.proj_lora.up.weight
lora key not loaded: single_blocks.9.processor.qkv_lora.down.weight
lora key not loaded: single_blocks.9.processor.qkv_lora.up.weight

AINeural

4 days ago

use this to convert the keys for the comfyui version you are using.

https://github.com/RamonGuthrie/ComfyUI-RBG-LoraConverter

Wakeme

4 days ago

•

edited 4 days ago

@AINeural No luck
No convertible keys found in fill-HunyuanWorld-PanoDiT-Image-lora.safetensors. Saving as is.

I think it had to be converted manually. The lora structure is also very odd. There are 2 types of weight *.proj_lora*, and .qkv_lora`

*.proj_lora1.down - shape [128, 3 072], similar to proj_lora2
*.proj_lora1.up - shape [3 072, 128], similar to proj_lora2
*.qkv_lora1.down - shape [128, 3 072], similar to qkv_lora2
*.qkv_lora1.up - shape [9 216, 128], similar to qkv_lora2

The 128 is the lora rank, and for qkv up weight it looks like concatenate qkv projection (9216 = 3*3072), and in ComfyUI compatible format, the attention part is split into q, k, and v individually. For the qkv_lora1.up part I think it just the slice in QKV order, but I have no idea for the qkv_lora1.down

It turns out that it is possible to convert to PeFT format (ComfyUI compatible)

import torch
from diffusers import FluxFillPipeline
from safetensors.torch import save_file
from diffusers.utils.state_dict_utils import convert_all_state_dict_to_peft, convert_state_dict_to_kohya
import os

model_path = "black-forest-labs/FLUX.1-Fill-dev"
lora_repo_id = "tencent/HunyuanWorld-1"
lora_subfolder = "HunyuanWorld-PanoDiT-Image"
lora_filename = "lora.safetensors"
output_path = "./HunyuanWorld-PanoDiT-Image-peft.safetensors"

print(f"Loading base model: {model_path}...")
pipe = FluxFillPipeline.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16
)

print(f"Loading LoRA '{lora_filename}' from {lora_repo_id}...")
pipe.load_lora_weights(
    lora_repo_id,
    subfolder=lora_subfolder,
    weight_name=lora_filename,
    torch_dtype=torch.bfloat16
)

print("Extracting LoRA state dictionary from the pipeline...")
lora_state_dict = pipe.lora_state_dict(
    lora_repo_id,
    subfolder=lora_subfolder,
    weight_name=lora_filename,
    torch_dtype=torch.bfloat16
)

print("Converting LoRA keys to PEFT format...")
peft_state_dict = convert_all_state_dict_to_peft(lora_state_dict)

print("Resolving shared tensor memory by cloning...")
final_state_dict = {key: tensor.clone() for key, tensor in peft_state_dict.items()}

print("Saving the final .safetensors file...")
save_file(final_state_dict, output_path)

pipe.unload_lora_weights()

Now the lora loaded without problem

I have uploaded the converted lora here. But loading with nunchaku seems like still not working properly

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment