Seg2Click-Qwen2.5-VL-3B

Seg2Click predicts a pixel-level action mask from a screenshot and an instruction, then selects a click at the centroid of the largest connected foreground component.

Paper: https://arxiv.org/abs/2610.16234

Code and installation instructions: https://github.com/jcwang0602/Seg2Click

Contents

This checkpoint includes the merged backbone, the FP32 action-mask decoder, and the tokenizer and processor required for inference. LoRA is already merged; no separate adapter should be applied.

Inference

Install Seg2Click and its dependencies, then download the public checkpoint:

from huggingface_hub import snapshot_download
from PIL import Image
from seg2click.predictor import Seg2ClickPredictor

checkpoint = snapshot_download("jcwang0602/Seg2Click-Qwen2.5-VL-3B")
predictor = Seg2ClickPredictor.from_pretrained(checkpoint)
prediction = predictor.predict(Image.open("screenshot.png"), "close this window")
print(prediction.point)  # normalized (x, y) in the original screenshot

The custom decoder requires the Seg2Click inference code. Loading the backbone alone with a generic Transformers generation pipeline does not perform mask-to-click prediction.

License

This model is fine-tuned from Qwen/Qwen2.5-VL-3B-Instruct. The backbone license is included in LICENSE; it is distinct from the MIT license of the Seg2Click source code.