InternVL-MoonViT-V2-LFM2.5-2.6B-HF

This model combines the MoonViT V2 vision encoder from Kimi K3 with the LFM2.5-2.6B language model.

The multimodal projector (MLP) was not trained. Please keep this in mind when evaluating or using the model.

Model architecture

• Vision encoder: MoonViT V2 from Kimi K3 (401M)
• Language model: LFM2.5-2.6B (2.7B, hybrid conv/attention; lm_head tied to the input embeddings)
• Multimodal projector (MLP): not trained (13M)

Both towers are copied bit-for-bit from their source checkpoints. The tokenizer adds the InternVL special tokens (<img>, </img>, <IMG_CONTEXT>, <quad>, </quad>, <ref>, </ref>, <box>, </box>), and the chat template renders one <IMG_CONTEXT> placeholder per image, which the processor expands to that image's token count. MoonViT performs the 2x2 patch merge itself, so an image of h x w patches becomes (h // 2) * (w // 2) image tokens, capped by the processor's in_token_limit (25600 patches).

Loading the model

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "PerRing/InternVL-MoonViT-V2-LFM2.5-2.6B-HF"

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
).eval()

The vision tower also supports FlashAttention-2, either from the flash-attn package (attn_implementation="flash_attention_2") or from the Hub kernel (attn_implementation="kernels-community/flash-attn2").

Inference

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"},
        ],
    },
]
inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Note: This repository uses custom modeling and processing code, so trust_remote_code=True is required.