---
language:
- en
license: mit
metrics:
- accuracy
pipeline_tag: image-text-to-text
tags:
- VLM
- Computer-Use-Agent
- OS-Agent
- GUI
- Grounding
library_name: transformers
---
# Introduction (ADDED TikToken3 support)
Reinforcement learning (RL) (e.g., GRPO) helps with grounding because of its inherent objective alignment—rewarding successful clicks—rather than encouraging long textual Chain-of-Thought (CoT) reasoning. Unlike approaches that rely heavily on verbose CoT reasoning, GRPO directly incentivizes actionable and grounded responses. Based on findings from our [blog](https://huggingface.co/blog/HelloKKMe/grounding-r1), we share state-of-the-art GUI grounding models trained using GRPO.
# Grounding Performance
We follow the standard evaluation protocol and benchmark our model on three challenging datasets. Our method consistently achieves the best results among all open-source model families. Below are the comparative results:
| **Model** | **Size** | **Open Source** | **ScreenSpot-V2** | **ScreenSpotPro** | **OSWORLD-G** | **OSWORLD-G-Refined** |
|-------------------|:--------:|:---------------:|:-----------------:|:-----------------:|:-----------------:|:-----------------:|
| OpenAI CUA | — | ❌ | 87.9 | 23.4 | — | — |
| Claude 3.7 | — | ❌ | 87.6 | 27.7 | — | — |
| JEDI-7B | 7B | ✅ | 91.7 | 39.5 | 54.1 | — |
| SE-GUI | 7B | ✅ | 90.3 | 47.0 | — | — |
| UI-TARS | 7B | ✅ | 91.6 | 35.7 | 47.5 | — |
| UI-TARS-1.5* | 7B | ✅ | 89.7* | 42.0* | 52.8* | 64.2* |
| UGround-v1-7B | 7B | ✅ | — | 31.1 | — | 36.4 |
| Qwen2.5-VL-32B-Instruct | 32B | ✅ | 91.9* | 48.0 | 46.5 | 59.6* |
| UGround-v1-72B | 72B | ✅ | — | 34.5 | — | — |
| Qwen2.5-VL-72B-Instruct | 72B | ✅ | 94.00* | 53.3 | — | 62.2* |
| UI-TARS | 72B | ✅ | 90.3 | 38.1 | — | — |
| OpenCUA | 7B | ✅ | 92.3 | 50.0 | 55.3 | 68.3* |
| OpenCUA | 32B | ✅ | 93.4 | 55.3 | 59.6 | 70.2* |
| GTA1-2507 (Ours) | 7B | ✅ | 92.4 *(∆ +2.7)* | 50.1*(∆ +8.1)* | 55.1 *(∆ +2.3)* | 67.7 *(∆ +3.5)* |
| GTA1 (Ours) | 7B | ✅ | 93.4 *(∆ +0.1)* | 55.5*(∆ +5.5)* | 60.1*(∆ +4.8)* | 68.8*(∆ +0.5)* |
| GTA1 (Ours) | 32B | ✅ | 95.2 *(∆ +1.8)* | 63.6*(∆ +8.3)* | 65.2 *(∆ +5.6)* | 72.2*(∆ +2.0)* |
> **Note:**
> - Model size is indicated in billions (B) of parameters.
> - A dash (—) denotes results that are currently unavailable.
> - A superscript asterisk (﹡) denotes our evaluated result.
> - UI-TARS-1.5 7B, OpenCUA-7B, and OpenCUA-32B are applied as our baseline models.
> - ∆ indicates the performance improvement (∆) of our model compared to its baseline.
# Agent Performance
## OSWorld and OSWorld-Verified Benchmarks
We evaluate our models on the OSWorld and OSWorld-Verified benchmarks following the standard evaluation protocol. The results demonstrate strong performance across both datasets.
| **Agent Model** | **Step** | **OSWorld** | **OSWorld-Verified** |
|-----------------|:--------:|:-----------:|:-------------------:|
| **Proprietary Models** |
| Claude 3.7 Sonnet | 100 | 28.0 | — |
| OpenAI CUA 4o | 200 | 38.1 | — |
| UI-TARS-1.5 | 100 | 42.5 | 41.8 |
| OpenAI CUA o3 | 200 | 42.9 | — |
| **Open-Source Models** |
| Aria-UI w/ GPT-4o | 15 | 15.2 | — |
| Aguvis-72B w/ GPT-4o | 15 | 17.0 | — |
| UI-TARS-72B-SFT | 50 | 18.8 | — |
| Agent S w/ Claude-3.5-Sonnet | 15 | 20.5 | — |
| Agent S w/ GPT-4o | 15 | 20.6 | — |
| UI-TARS-72B-DPO | 15 | 22.7 | — |
| UI-TARS-72B-DPO | 50 | 24.6 | — |
| UI-TARS-1.5-7B | 100 | 26.9 | 27.4 |
| Jedi-7B w/ o3 | 100 | — | 51.0 |
| Jedi-7B w/ GPT-4o | 100 | 27.0 | — |
| Agent S2 w/ Claude-3.7-Sonnet | 50 | 34.5 | — |
| Agent S2 w/ Gemini-2.5-Pro | 50 | 41.4 | 45.8 |
| Agent S2.5 w/ o3 | 100 | — | 56.0 |
| Agent S2.5 w/ GPT-5 | 100 | — | 58.4 |
| CoAct-1 w/o3 & o4mini & OpenAI CUA 4o | 150 | — | 60.8 |
| GTA1-7B-2507 w/ o3 | 100 | 45.2 | 53.1 |
| GTA1-7B-2507 w/ GPT-5 | 100 | — | 61.0 |
| GTA1-32B w/ o3 | 100 | — | 55.4 |
| GTA1-32B w/ GPT-5 | 100 | — | 63.4 |
> **Note:** A dash (—) indicates unavailable results.
## WindowsAgentArena Benchmark
We also evaluate our models on the WindowsAgentArena benchmark, demonstrating strong performance in Windows-specific GUI automation tasks.
| **Agent Model** | **Step** | **Success Rate** |
|-----------------|:--------:|:---------------:|
| Kimi-VL | 15 | 10.4 |
| WAA | — | 19.5 |
| Jedi w/ GPT-4o | 100 | 33.7 |
| GTA1-7B-2507 w/ o3 | 100 | 47.9 |
| GTA1-7B-2507 w/ GPT-5 | 100 | 49.2 |
| GTA1-32B w/ o3 | 100 | 51.2 |
| GTA1-32B w/ GPT-5 | 100 | 50.6 |
> **Note:** A dash (—) indicates unavailable results.
# Inference
Below is a code snippet demonstrating how to run inference using a trained model.
```python
from transformers import AutoTokenizer, AutoImageProcessor
from transformers.models.qwen2_vl.image_processing_qwen2_vl_fast import smart_resize
from PIL import Image
from io import BytesIO
import base64
import re
from vllm import LLM, SamplingParams
instruction="click start"
image_path="example.png"
CLICK_REGEXES = [
# pyautogui.click(x=123, y=456)
re.compile(r"click\s*\(\s*x\s*=\s*(\d+)\s*,\s*y\s*=\s*(\d+)\s*\)", re.IGNORECASE),
# pyautogui.click(123, 456) or click(123,456)
re.compile(r"click\s*\(\s*(\d+)\s*,\s*(\d+)\s*\)", re.IGNORECASE),
]
def format_message(image_path,instruction):
SYSTEM_PROMPT = (
"You are a GUI agent. You are given a task and a screenshot of the screen. "
"You need to perform a series of pyautogui actions to complete the task."
)
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": [
{"type": "image", "image": image_path},
{"type": "text", "text": instruction},
]},
]
text = prompt_tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
text2, n = re.subn(
r"<\|media_begin\|>.*?<\|media_end\|>",
"<|vision_start|><|image_pad|><|vision_end|>",
text,
flags=re.S
)
if n == 0:
raise RuntimeError("Cannot find <|media_begin|>...<|media_end|> token.")
return text2
def parse_xy_from_text(text: str):
if "click" not in text.lower():
return [-1, -1]
for rx in CLICK_REGEXES:
m = rx.search(text)
if m:
try:
return int(m.group(1)), int(m.group(2))
except Exception:
continue
return [-1,-1]
def convert_pil_image_to_base64(image):
buffered = BytesIO()
image.save(buffered, format="PNG")
return base64.b64encode(buffered.getvalue()).decode()
llm = LLM(
model="Salesforce/GTA1-32B",
tokenizer="Salesforce/GTA1-32B",
tokenizer_mode="slow",
trust_remote_code=True,
dtype="bfloat16",
limit_mm_per_prompt={"image": 1},
tensor_parallel_size=1,
)
prompt_tok = AutoTokenizer.from_pretrained("Salesforce/GTA1-32B", trust_remote_code=True)
sp = SamplingParams(max_tokens=512, temperature=0.0)
tokenizer = llm.get_tokenizer()
processor=AutoImageProcessor.from_pretrained("Salesforce/GTA1-32B", trust_remote_code=True)
image = Image.open(image_path).convert('RGB')
resized_height, resized_width = smart_resize(
image.height,
image.width,
factor=processor.patch_size * processor.merge_size,
min_pixels=processor.min_pixels,
max_pixels=processor.max_pixels,
)
resized_image = image.resize((resized_width, resized_height))
messages = format_message(image_path, instruction)
response = llm.generate(
[{"prompt": messages, "multi_modal_data": {"image": [resized_image]}}],
sampling_params=sp
)[0].outputs[0].text
coordinates = parse_xy_from_text(response)
print(coordinates[0]/resized_width*image.width, coordinates[1]/resized_height*image.height)
```
# Model Serving
Below is an example script for serving the model.
```python
import torch
import os
# -------------------------
# System / Torch defaults
# -------------------------
os.environ.setdefault("TOKENIZERS_PARALLELISM", "false") # avoid CPU oversubscription
os.environ.setdefault("VLLM_USE_V1", "1")
os.environ.setdefault("VLLM_ENGINE_IN_BACKGROUND_THREAD", "0")
import base64
import re
from typing import Dict, List, Union
from PIL import Image
from io import BytesIO
import traceback
import argparse
import asyncio
import requests
import ray
from ray import serve
from fastapi import FastAPI
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
import uuid
N_REPLICAS = 2
try:
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
except Exception:
pass
# -------------------------
# IO helpers
# -------------------------
def pil_to_base64(img: Image.Image, format: str = "PNG") -> str:
buffer = BytesIO()
img.save(buffer, format=format)
img_bytes = buffer.getvalue()
img_b64 = base64.b64encode(img_bytes).decode("utf-8")
return img_b64
def data_uri_to_pil(data_uri: str) -> Image.Image:
header, b64_str = data_uri.split(",", 1)
img_data = base64.b64decode(b64_str)
buffer = BytesIO(img_data)
img = Image.open(buffer)
return img
def extract_images(messages: List[Dict]) -> List[Image.Image]:
images = []
for msg in messages:
if msg.get("role") == "user":
for content in msg.get("content", []):
if content.get("type") in ["image", "image_url"]:
if content["type"] == "image":
images.append(data_uri_to_pil(content["image"]).convert("RGB"))
else:
images.append(data_uri_to_pil(content["image_url"]["url"]).convert("RGB"))
return images
# -------------------------
# Prompt builder
# -------------------------
def build_prompt_with_template(tokenizer: AutoTokenizer, messages: List[Dict]) -> str:
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
text2, n = re.subn(
r"<\|media_begin\|>.*?<\|media_end\|>",
"<|vision_start|><|image_pad|><|vision_end|>",
text,
flags=re.S,
)
if n == 0:
raise RuntimeError("Did not find <|media_begin|>...<|media_end|> block in template.")
return text2
# -------------------------
# Deployment
# -------------------------
def build_app(model_path: str, num_replicas: int, port: int):
api = FastAPI(title="GTA1-32B Multi-GPU Service (High-throughput)")
@serve.deployment(
num_replicas=num_replicas,
ray_actor_options={"num_gpus": 1, "num_cpus": 4},
max_ongoing_requests=16,
)
class GTA1Model:
def __init__(self, model_path: str):
gpu_ids = ray.get_gpu_ids()
self.gpu_id = gpu_ids[0] if gpu_ids else 0
print(f"🔍 Ray assigned GPU IDs: {gpu_ids}")
# Initialize vLLM within this replica (Ray sets CUDA_VISIBLE_DEVICES)
print(f"🔄 Initializing vLLM on GPU {self.gpu_id}[ray id] from {model_path}")
if not torch.cuda.is_available():
raise RuntimeError("CUDA is not available")
self.llm = LLM(
model=model_path,
tokenizer=model_path,
tokenizer_mode="slow",
trust_remote_code=True,
dtype="bfloat16",
limit_mm_per_prompt={"image": 1},
max_model_len=32768,
tensor_parallel_size=1,
)
self.vllm_tokenizer = self.llm.get_tokenizer()
self.hf_tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
self.model_path = model_path
self.dtype = "bfloat16"
print(f"✅ vLLM initialized successfully (Ray GPU Id: {self.gpu_id})")
# ------------ batching core ------------
@serve.batch(max_batch_size=8, batch_wait_timeout_s=0.1) # increase if GPU allows
async def _generate_batch(self, payload: Union[Dict, List[Dict]]):
"""Build prompts, enforce single image, and call vLLM.generate."""
if isinstance(payload, dict):
list_of_payloads = [payload]
else:
list_of_payloads = payload
request_id = uuid.uuid4().hex[:8]
# --- Build per-sample prompt/image ---
prompts: List[str] = []
images_per_req: List[Image.Image] = []
error_results = []
early_exit = False
for p in list_of_payloads:
try:
messages = p["messages"]
imgs = extract_images(messages)
if len(imgs) != 1:
raise RuntimeError(f"Exactly one image is required, got {len(imgs)}")
prompt_text = build_prompt_with_template(self.hf_tokenizer, messages)
# Sanity check on tokens: 1 <|image_pad|>, no <|media_placeholder|>
tok = self.vllm_tokenizer
id_imgpad = tok.encode("<|image_pad|>", add_special_tokens=False)[0]
id_media = tok.encode("<|media_placeholder|>", add_special_tokens=False)[0]
ids = tok.encode(prompt_text, add_special_tokens=False)
if sum(i == id_imgpad for i in ids) != 1 or any(i == id_media for i in ids):
raise RuntimeError("Prompt media tokens invalid after conversion")
prompts.append(prompt_text)
images_per_req.append(imgs[0])
except Exception as e:
early_exit = True
trace = traceback.format_exc()
error_results.append(
{
"response": "",
"error": {
"message": str(e),
"trace": trace,
'type_of_payload': str(type(payload)),
'type_of_list_of_payloads': str(type(list_of_payloads)),
'type_of_p': str(type(p)),
'p_keys': str(p.keys()) if isinstance(p, dict) else str(p),
},
"usage": {},
"gpu_id": self.gpu_id
}
)
if early_exit:
return error_results
# --- vLLM generation ---
args_base = list_of_payloads[0]
sp = SamplingParams(
max_tokens=args_base.get("max_new_tokens", 512),
temperature=args_base.get("temperature", 0.0),
top_p=args_base.get("top_p", 0.9),
)
requests_list = [
{"prompt": pr, "multi_modal_data": {"image": [im]}}
for pr, im in zip(prompts, images_per_req)
]
outs = self.llm.generate(requests_list, sampling_params=sp)
tok = self.vllm_tokenizer
results: List[Dict] = []
for pr, o in zip(prompts, outs):
text = o.outputs[0].text if o.outputs else ""
gen_tokens = len(o.outputs[0].token_ids) if (o.outputs and hasattr(o.outputs[0], 'token_ids')) else None
prompt_tokens = len(tok.encode(pr, add_special_tokens=False))
usage = {
"prompt_tokens": prompt_tokens,
"generated_tokens": gen_tokens if gen_tokens is not None else None,
"total_tokens": (prompt_tokens + gen_tokens) if gen_tokens is not None else None,
}
results.append({
"response": text,
"error": "",
"usage": usage,
"gpu_id": self.gpu_id,
'bs_size_in_this_request': f"{request_id}:{len(list_of_payloads)}"
})
return results
# Exposed single-call entry that joins the batch
async def call_llm(self, payload: Dict):
try:
res = await self._generate_batch(payload)
return res
except Exception as e:
trace = traceback.format_exc()
return {"response": "", "error": {"message": str(e), "trace": trace}, "usage": {}, "gpu_id": self.gpu_id}
def health(self):
return {
"status": "ok",
"gpu_id": self.gpu_id,
"dtype": self.dtype,
"model_path": self.model_path,
}
model = GTA1Model.bind(model_path)
@serve.deployment(max_ongoing_requests=96)
@serve.ingress(api)
class GTA1App:
def __init__(self, model_handle):
self.model_deployment = model_handle
@api.get("/health")
async def health_all(self):
# Calling the same Serve handle N times does not guarantee each call hits a different replica
attempts = max(8, N_REPLICAS * 4) # oversample
calls = [self.model_deployment.health.remote() for i in range(attempts)]
replies = await asyncio.gather(*calls)
# dedupe by replica_id (or by tuple(gpu_id))
seen = {}
for r in replies:
seen[r.get("gpu_id", f"unknown-{len(seen)}")] = r
if len(seen) >= N_REPLICAS:
break
return {"replicas": list(seen.values())}
@api.post("/call_llm")
async def call_llm(self, req: Dict):
return await self.model_deployment.call_llm.remote(req)
return GTA1App.bind(model)
# -------------------------
# Main
# -------------------------
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--model_path", type=str, default="Salesforce/GTA1-32B")
parser.add_argument("--host", type=str, default="0.0.0.0")
parser.add_argument("--port", type=int, default=3005)
parser.add_argument("--num_replicas", type=int, default=2)
args = parser.parse_args()
N_REPLICAS = args.num_replicas
ray.init(ignore_reinit_error=True)
print(f"🚀 Starting GTA1-32B service on {args.host}:{args.port}")
serve.start(detached=True, http_options={"host": args.host, "port": args.port})
app = build_app(args.model_path, args.num_replicas, args.port)
serve.run(app, name="GTA1-32B", route_prefix="/")
# Quick health sample
try:
r = requests.get(f"http://0.0.0.0:{args.port}/health", timeout=5)
print(r.json())
except Exception as e:
print("Health probe failed:", e)
```
Here is the example usage,
```python
import argparse
import base64
import concurrent.futures
import json
import os
import re
from typing import Dict, List, Tuple
from gui_agent.agent.gta1.format_message import encode_numpy_image_to_base64, encode_image_bytes, smart_resize
import requests
from PIL import Image, ImageDraw
def image_file_to_data_uri(image_path: str) -> str:
if not os.path.exists(image_path):
raise FileNotFoundError(f"Image not found: {image_path}")
with open(image_path, "rb") as f:
b64 = base64.b64encode(f.read()).decode("utf-8")
# default to png; serverside only requires a data URI header then comma
return f"data:image/png;base64,{b64}"
def build_messages(image_path: str, instruction: str, system_prompt: str) -> List[Dict]:
return [
{"role": "system", "content": system_prompt},
{
"role": "user",
"content": [
{"type": "image", "image": image_file_to_data_uri(image_path)},
{"type": "text", "text": instruction},
],
},
]
def call_health(base_url: str, timeout: float = 10.0) -> Dict:
r = requests.get(f"{base_url}/health", timeout=timeout)
r.raise_for_status()
return r.json()
def call_single(
base_url: str,
image_path: str,
instruction: str,
system_prompt: str,
max_new_tokens: int = 512,
temperature: float = 0.0,
top_p: float = 0.9,
timeout: float = 120.0,
) -> List[Dict]:
payload = {
"messages": build_messages(image_path, instruction, system_prompt),
"max_new_tokens": max_new_tokens,
"temperature": temperature,
"top_p": top_p,
}
r = requests.post(f"{base_url}/call_llm", json=payload, timeout=timeout)
r.raise_for_status()
resp = r.json()
if isinstance(resp, dict):
return [resp]
return resp
def call_many_concurrent(
base_url: str,
image_path: str,
instruction: str,
system_prompt: str,
num_requests: int,
concurrency: int,
max_new_tokens: int = 512,
temperature: float = 0.0,
top_p: float = 0.9,
timeout: float = 120.0,
) -> List[List[Dict]]:
results: List[List[Dict]] = []
def _one(i: int) -> List[Dict]:
# Vary instruction slightly so you can trace requests
instr = f"{instruction} [req {i+1}/{num_requests}]"
return call_single(
base_url,
image_path,
instr,
system_prompt,
max_new_tokens,
temperature,
top_p,
timeout,
)
with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as pool:
futures = [pool.submit(_one, i) for i in range(num_requests)]
for fut in concurrent.futures.as_completed(futures):
results.append(fut.result())
return results
def pretty_print_response(batch_results: List[Dict]) -> None:
if isinstance(batch_results, dict):
batch_results = [batch_results]
for idx, item in enumerate(batch_results):
if item.get("error"):
print(f"[#{idx}] ERROR: {json.dumps(item['error'], ensure_ascii=False)})")
else:
usage = item.get("usage", {})
print(f"[#{idx}] gpu={item.get('gpu_id')} tokens={usage} text=\n{item.get('response','').strip()}\n")
CLICK_KWARGS_REGEX = re.compile(r"pyautogui\.click\(\s*x\s*=\s*(\d+)\s*,\s*y\s*=\s*(\d+)\s*\)")
CLICK_POSARGS_REGEX = re.compile(r"pyautogui\.click\(\s*(\d+)\s*,\s*(\d+)\s*\)")
def extract_clicks_from_text(text: str) -> List[Tuple[int, int]]:
clicks: List[Tuple[int, int]] = []
for x, y in CLICK_KWARGS_REGEX.findall(text or ""):
clicks.append((int(x), int(y)))
for x, y in CLICK_POSARGS_REGEX.findall(text or ""):
clicks.append((int(x), int(y)))
return clicks
def extract_clicks_from_results(result_items: List[Dict]) -> List[Tuple[int, int]]:
clicks: List[Tuple[int, int]] = []
if isinstance(result_items, dict):
result_items = [result_items]
for item in result_items:
if item.get("error"):
continue
clicks.extend(extract_clicks_from_text(item.get("response", "")))
return clicks
def compute_resized_dims_for_server_mapping(image_path: str) -> Tuple[int, int, int, int]:
with Image.open(image_path) as im:
width, height = im.size
resized_H, resized_W = smart_resize(
height,
width,
factor=28,
min_pixels=1000,
max_pixels=1000000000000,
)
return width, height, int(resized_W), int(resized_H)
def map_clicks_to_original(clicks_resized: List[Tuple[int, int]],
original_w: int,
original_h: int,
resized_w: int,
resized_h: int) -> List[Tuple[int, int]]:
if resized_w == 0 or resized_h == 0:
return []
scale_x = original_w / float(resized_w)
scale_y = original_h / float(resized_h)
mapped: List[Tuple[int, int]] = []
for x, y in clicks_resized:
mapped_x = int(round(x * scale_x))
mapped_y = int(round(y * scale_y))
mapped.append((mapped_x, mapped_y))
return mapped
def draw_circles_on_image(image_path: str,
points: List[Tuple[int, int]],
output_path: str,
radius: int = 8,
color: Tuple[int, int, int] = (255, 0, 0),
width: int = 3) -> None:
if not points:
return
with Image.open(image_path).convert("RGB") as img:
drawer = ImageDraw.Draw(img)
for (x, y) in points:
left = x - radius
top = y - radius
right = x + radius
bottom = y + radius
drawer.ellipse([(left, top), (right, bottom)], outline=color, fill=(0,255,0), width=width)
img.save(output_path)
print(f"Annotated image saved to: {output_path} (points drawn: {len(points)})")
SYSTEM_PROMPT = (
"You are a GUI agent. You are given a task and a screenshot of the screen. "
"You need to perform a series of pyautogui actions to complete the task."
)
def main():
parser = argparse.ArgumentParser(description="Examples: single and batched inference against GTA1-32B Ray Serve.")
parser.add_argument("--host", type=str, default="http://localhost", help="Ray Serve host, e.g. http://localhost or http://IP")
parser.add_argument("--port", type=int, default=3005, help="Ray Serve port")
parser.add_argument("--image", type=str, required=False, default="example.jpg", help="Path to input image")
parser.add_argument("--instruction", type=str, default="click the icon in the bottom row, third from the left", help="User instruction")
parser.add_argument("--system", type=str, default=SYSTEM_PROMPT)
parser.add_argument("--mode", type=str, choices=["single", "batch", "health"], default="batch")
parser.add_argument("--num_requests", type=int, default=8, help="Number of requests in batch mode")
parser.add_argument("--concurrency", type=int, default=8, help="Max concurrent HTTP calls in batch mode")
parser.add_argument("--max_new_tokens", type=int, default=512)
parser.add_argument("--temperature", type=float, default=0.0)
parser.add_argument("--top_p", type=float, default=0.9)
parser.add_argument("--timeout", type=float, default=180.0)
args = parser.parse_args()
base_url = f"{args.host}:{args.port}"
if args.mode == "health":
info = call_health(base_url, timeout=10.0)
print(json.dumps(info, indent=2))
return
if args.mode == "single":
result_list = call_single(
base_url=base_url,
image_path=args.image,
instruction=args.instruction,
system_prompt=args.system,
max_new_tokens=args.max_new_tokens,
temperature=args.temperature,
top_p=args.top_p,
timeout=args.timeout,
)
print(result_list)
pretty_print_response(result_list)
clicks_resized = extract_clicks_from_results(result_list)
if clicks_resized:
orig_w, orig_h, resized_w, resized_h = compute_resized_dims_for_server_mapping(args.image)
mapped_clicks = map_clicks_to_original(clicks_resized, orig_w, orig_h, resized_w, resized_h)
out_path = f"ray_serve/annotated.png"
draw_circles_on_image(args.image, mapped_clicks, out_path)
return
if args.mode == "batch":
print(f"Submitting {args.num_requests} requests with concurrency={args.concurrency}...")
batch_outs = call_many_concurrent(
base_url=base_url,
image_path=args.image,
instruction=args.instruction,
system_prompt=args.system,
num_requests=args.num_requests,
concurrency=args.concurrency,
max_new_tokens=args.max_new_tokens,
temperature=args.temperature,
top_p=args.top_p,
timeout=args.timeout,
)
for i, one_result in enumerate(batch_outs):
print(f"===== Result for request {i+1} =====")
pretty_print_response(one_result)
all_clicks_resized: List[Tuple[int, int]] = []
for one_result in batch_outs:
all_clicks_resized.extend(extract_clicks_from_results(one_result))
if all_clicks_resized:
orig_w, orig_h, resized_w, resized_h = compute_resized_dims_for_server_mapping(args.image)
mapped_clicks = map_clicks_to_original(all_clicks_resized, orig_w, orig_h, resized_w, resized_h)
out_path = f"ray_serve/annotated.png"
draw_circles_on_image(args.image, mapped_clicks, out_path)
return
if __name__ == "__main__":
main()
```
## Ethical Considerations
This model is released for research and educational purposes. While our model demonstrates strong performance on GUI benchmarks, users should carefully evaluate its suitability for their specific use cases.
**Important Considerations:**
- **Accuracy Limitations:** Like all AI systems, this model may produce incorrect outputs or fail to accurately identify GUI elements in certain scenarios.
- **Safety and Security:** Exercise caution when deploying GUI automation agents, especially in production environments where incorrect actions could affect system integrity or data security.
- **Human Oversight:** We recommend maintaining appropriate human supervision when using this model for automated GUI interactions.
- **Compliance:** Users are responsible for ensuring their use of this model complies with applicable laws, regulations, and organizational policies.
**Recommended Best Practices:**
- Thoroughly test the model in controlled environments before production deployment
- Implement safeguards and error handling mechanisms
- Consider the potential impact of automated actions on user systems and data
- Regularly monitor and validate model performance in your specific domain
For further guidance on use cases, refer to our AUP and AI AUP.
## Citation
If you're using any GTA model or find it helpful in your research, please cite it as follows:
```markdown
@article{yang2025gta1guitesttimescaling,
title={GTA1: GUI Test-time Scaling Agent},
author={Yan Yang and Dongxu Li and Yutong Dai and Yuhao Yang and Ziyang Luo and Zirui Zhao and Zhiyuan Hu and Junzhe Huang and Amrita Saha and Zeyuan Chen and Ran Xu and Liyuan Pan and Silvio Savarese and Caiming Xiong and Junnan Li},
year={2025},
eprint={2507.05791},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2507.05791},
}
```