Qwen

/

Qwen2.5-VL-32B-Instruct

like 352

Follow

Qwen 29.5k

Hi Qwen team,

First of all, thank you for your impressive work on the Qwen2.5-VL-32B-Instruct model!

I’ve been testing the model using the open-source version available on Hugging Face, running it locally. However, I’ve noticed a significant difference in outputs between my local deployment, the API, and the official demo on https://chat.qwen.ai/.

Test Image

My Code

import re

import PIL.Image as Image
import torch
from qwen_vl_utils import process_vision_info

from transformers import (
    AutoProcessor,
    Qwen2_5_VLForConditionalGeneration,
)


if __name__ == "__main__":
    # We recommend enabling flash_attention_2 for better acceleration and memory saving, especially in multi-image and video scenarios.
    model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
        "Qwen/Qwen2.5-VL-32B-Instruct",
        torch_dtype=torch.bfloat16,
        attn_implementation="flash_attention_2",
        device_map="auto",
    )
    # default processer
    processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-32B-Instruct")

    prompt = """
    I will provide you with a picture as well as a question that you will need to think about before answering this question. Output as Json e.g. {'think': content of think, 'answer': content of answer}. For example: {'think': According to the recipe, the customer ordered two roast chicken and the price for each roast chicken is 2.781, so the total price for this meal is 2.781x2 =5.562 . 'answer': 5.562}. Note that for complicate question you can think longer and for easy question you can think shorter or {'think': I can see two pigs and three chickens in this image, so there are a total of 5 animals.' answer': 5}

    Question: What was the percentage of foreign visitors to GDP in Hungary in 2017?\nAnswer the question using a single word or phrase.
    """

    image_path = "1.png"
    image = Image.open(image_path)

    messages = [
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "image": image,
                },
                {"type": "text", "text": prompt},
            ],
        },
    ]

    # Preparation for inference
    text = processor.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    image_inputs, video_inputs = process_vision_info(messages)
    inputs = processor(
        text=[text],
        images=image_inputs,
        videos=video_inputs,
        padding=True,
        return_tensors="pt",
    )
    inputs = inputs.to("cuda")

    generated_ids = model.generate(**inputs)
    generated_ids_trimmed = [
        out_ids[len(in_ids) :]
        for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
    ]
    output_text = processor.batch_decode(
        generated_ids_trimmed,
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False,
    )
    print(output_text[0])

My Output

json
{
  "answer": "73.2%"
}

Output From https://chat.qwen.ai/ and also the Hugginface API Demo https://huggingface.co/spaces/Qwen/Qwen2.5-VL-32B-Instruct

{
  "think": "The image is a bar chart showing the share of GDP contribution from foreign visitor spending and domestic spending in Hungary. The bar labeled 'Foreign visitor spending' shows a value of 73.2%, indicating that this percentage represents the contribution of foreign visitors to the GDP.",
  "answer": "73.2%"
}

Looking forward to you reply.