naver-hyperclovax

/

HyperCLOVAX-SEED-Vision-Instruct-3B

like 219

Follow

HyperCLOVA X 942

First of all, thank you for sharing.

While studying multimodal llm models, the way to call model.generate() is different for each, but recently, it seems to follow a specific method, so I am writing this so that the model can be modified in that direction.

First, in the case of Qwen2.5-VL and Phi-4-multimodal, model.generate() is called in a way roughly like below.

The important thing is that it goes in the form of **inputs.

However, the model you shared is not like that, and if it is not difficult, I would like to ask if it is possible to modify it.

Qwen2.5-VL

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

# Preparation for inference
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=128)

phi-4-multimodal

chat = [
    {'role': 'user', 'content': f'<|image_1|>What is shown in this image?'},
    {
        'role': 'assistant',
        'content': "The image depicts a street scene with a prominent red stop sign in the foreground. The background showcases a building with traditional Chinese architecture, characterized by its red roof and ornate decorations. There are also several statues of lions, which are common in Chinese culture, positioned in front of the building. The street is lined with various shops and businesses, and there's a car passing by.",
    },
    {'role': 'user', 'content': 'What is so special about this image'},
]
url = 'https://www.ilankelman.org/stopsigns/australia.jpg'
image = Image.open(requests.get(url, stream=True).raw)
prompt = processor.tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
# need to remove last <|endoftext|> if it is there, which is used for training, not inference. For training, make sure to add <|endoftext|> in the end.
if prompt.endswith('<|endoftext|>'):
    prompt = prompt.rstrip('<|endoftext|>')

print(f'>>> Prompt\n{prompt}')

inputs = processor(prompt, [image], return_tensors='pt').to('cuda:0')
generate_ids = model.generate(
    **inputs,
    max_new_tokens=1000,
    generation_config=generation_config,
)