--- license: other base_model: Lightricks/LTX-Video tags: - ltx-video - ltx-video-diffusers - text-to-video - image-to-video - diffusers - simpletuner - not-for-all-audiences - lora - template:sd-lora - video-to-video - lycoris pipeline_tag: video-to-video inference: true widget: - text: unconditional (blank prompt) parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_0_0.gif - text: At midnight in a neon-lit cityscape, an agile anime protagonist leaps between rooftops, evading shadowy figures with fluid, acrobatic action in a striking black and white animated style. parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_1_0.gif - text: At dawn in a rustic village kitchen, a dedicated sous-chef passionately chops and stirs ingredients, crafting a traditional dish with precision and care in a refined black and white animated scene. parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_2_0.gif - text: In the soft morning light of a quiet garden, a reflective individual gazes into the distance, their subtle expressions and gentle actions captured in a delicate black and white animated portrait. parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_3_0.gif - text: Under the glow of a full moon at an abandoned urban theater, a solitary figure delivers a dramatic monologue, their every movement and emotion rendered in cinematic black and white detail. parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_4_0.gif - text: At twilight in a sophisticated art gallery, a poised subject glides through elegant corridors, each graceful step and refined gesture illuminated in a timeless black and white animated scene. parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_5_0.gif - text: In the early morning mist on a bustling city street, a daring adventurer sprints through narrow alleys, navigating obstacles with dynamic energy in a high-octane black and white animated sequence. parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_6_0.gif - text: During a foggy night on a deserted pier, an enigmatic figure silently watches the dark waters, their slow, deliberate actions and shadowy presence unfolding in a mysterious black and white animated narrative. parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_7_0.gif - text: At the break of dawn in a historic train station, a character in period attire rushes across the platform, their hurried actions and nostalgic surroundings brought to life in classic black and white animation. parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_8_0.gif - text: In the fading light of an urban alley adorned with abstract murals, a creative soul sketches fleeting impressions, their experimental actions and imaginative process captured in an artistic black and white animated style. parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_9_0.gif - text: At twilight in a bustling high-tech metropolis, a futuristic agent navigates a maze of holographic displays and sleek architecture, their rapid movements and cutting-edge surroundings depicted in a bold black and white animated scene. parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_10_0.gif - text: At dusk in a serene lakeside park, a graceful woman performs a fluid dance, her elegant movements and gentle expressions perfectly timed with the shifting light in a refined black and white animated sequence. parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_11_0.gif - text: At noon in a busy city square, a determined man strides purposefully through the crowd, his assertive actions and commanding presence captured in a dynamic black and white animated scene. parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_12_0.gif - text: In the afternoon at a lively playground, a spirited boy races joyfully across a field, his energetic play and mischievous antics rendered in a vibrant black and white animated style. parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_13_0.gif - text: At sunrise on a quaint village street, a curious girl skips along the pavement, her bright energy and animated expressions shining through in a charming black and white animated scene. parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_14_0.gif - text: During a cozy evening in a warmly lit living room, a family gathers around a table for a shared meal, their tender interactions and joyful connections animated in a heartfelt black and white scene. parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_15_0.gif - text: A black and white disney scene in the style of Steamboat Willie parameters: negative_prompt: ugly, cropped, blurry, low-quality, mediocre average output: url: ./assets/image_16_0.gif --- # ltxvideo-disney This is a LyCORIS adapter derived from [Lightricks/LTX-Video](https://huggingface.co/Lightricks/LTX-Video). The main validation prompt used during training was: ``` A black and white disney scene in the style of Steamboat Willie ``` ## Validation settings - CFG: `3.8` - CFG Rescale: `0.0` - Steps: `25` - Sampler: `FlowMatchEulerDiscreteScheduler` - Seed: `42` - Resolution: `768x512` Note: The validation settings are not necessarily the same as the [training settings](#training-settings). You can find some example images in the following gallery: The text encoder **was not** trained. You may reuse the base model text encoder for inference. ## Training settings - Training epochs: 2666 - Training steps: 8000 - Learning rate: 5e-05 - Learning rate schedule: cosine - Warmup steps: 400000 - Max grad value: 0.0 - Effective batch size: 24 - Micro-batch size: 8 - Gradient accumulation steps: 1 - Number of GPUs: 3 - Gradient checkpointing: True - Prediction type: flow-matching (extra parameters=['training_scheduler_timestep_spacing=trailing', 'inference_scheduler_timestep_spacing=trailing']) - Optimizer: adamw_bf16 - Trainable parameter precision: Pure BF16 - Base model precision: `int8-quanto` - Caption dropout probability: 10.0% ### LyCORIS Config: ```json { "bypass_mode": true, "algo": "lokr", "multiplier": 1.0, "full_matrix": true, "linear_dim": 10000, "linear_alpha": 1, "factor": 4, "apply_preset": { "target_module": [ "Attention", "FeedForward" ], "module_algo_map": { "FeedForward": { "factor": 4 }, "Attention": { "factor": 2 } } } } ``` ## Datasets ### disney-black-and-white - Repeats: 0 - Total number of images: ~69 - Total number of aspect buckets: 1 - Resolution: 0.2304 megapixels - Cropped: False - Crop style: None - Crop aspect: None - Used for regularisation data: No ## Inference ```python import torch from diffusers import DiffusionPipeline from lycoris import create_lycoris_from_weights def download_adapter(repo_id: str): import os from huggingface_hub import hf_hub_download adapter_filename = "pytorch_lora_weights.safetensors" cache_dir = os.environ.get('HF_PATH', os.path.expanduser('~/.cache/huggingface/hub/models')) cleaned_adapter_path = repo_id.replace("/", "_").replace("\\", "_").replace(":", "_") path_to_adapter = os.path.join(cache_dir, cleaned_adapter_path) path_to_adapter_file = os.path.join(path_to_adapter, adapter_filename) os.makedirs(path_to_adapter, exist_ok=True) hf_hub_download( repo_id=repo_id, filename=adapter_filename, local_dir=path_to_adapter ) return path_to_adapter_file model_id = 'Lightricks/LTX-Video' adapter_repo_id = 'bghira/ltxvideo-disney' adapter_filename = 'pytorch_lora_weights.safetensors' adapter_file_path = download_adapter(repo_id=adapter_repo_id) pipeline = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.bfloat16) # loading directly in bf16 lora_scale = 1.0 wrapper, _ = create_lycoris_from_weights(lora_scale, adapter_file_path, pipeline.transformer) wrapper.merge_to() prompt = "A black and white disney scene in the style of Steamboat Willie" negative_prompt = 'ugly, cropped, blurry, low-quality, mediocre average' ## Optional: quantise the model to save on vram. ## Note: The model was quantised during training, and so it is recommended to do the same during inference time. from optimum.quanto import quantize, freeze, qint8 quantize(pipeline.transformer, weights=qint8) freeze(pipeline.transformer) pipeline.to('cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu') # the pipeline is already in its target precision level model_output = pipeline( prompt=prompt, negative_prompt=negative_prompt, num_inference_steps=25, generator=torch.Generator(device='cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu').manual_seed(42), width=768, height=512, guidance_scale=3.8, ).frames[0] from diffusers.utils.export_utils import export_to_gif export_to_gif(model_output, "output.gif", fps=25) ``` ## Exponential Moving Average (EMA) SimpleTuner generates a safetensors variant of the EMA weights and a pt file. The safetensors file is intended to be used for inference, and the pt file is for continuing finetuning. The EMA model may provide a more well-rounded result, but typically will feel undertrained compared to the full model as it is a running decayed average of the model weights.