GUI grounding

Grounding Open-Domain Instructions to Automate Web Support Tasks

Nancy Xu, Sam Masling et al.

Introduces RUSS, which translates support instructions into a domain-specific language and grounds them in actionable web elements.

ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces

Zecheng He, Srinivas Sunkara et al.

Pretrains ActionBert on user interaction traces to combine visual, linguistic, and UI-structure features. It evaluates the resulting UI representations on tasks including icon classification and description-based component retrieval.

Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements

Yang Li, Gang Li et al.

Introduces crowdsourced descriptions of mobile UI elements and studies multimodal models that generate useful widget captions.

Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?

Jieshan Chen, Mulong Xie et al.

Compares seven traditional and learned GUI element detectors on more than 50,000 interface images. The analysis motivates a coarse-to-fine non-text detector combined with a learned text detector to address GUI-specific localization requirements.

Mapping Natural Language Instructions to Mobile UI Action Sequences

Yang Li, Jiacong He et al.

Introduces PIXELHELP for predicting mobile UI action sequences from instructions, together with action-phrase annotations and synthetic UI action descriptions for training. Separate Transformers extract instruction steps and ground object descriptions using interface content and screen position.

Mapping Natural Language Commands to Web Elements

Panupong Pasupat, Tian-Shun Jiang et al.

Defines language grounding on webpages as choosing the UI element targeted by a natural-language command. A collection of more than 50,000 commands and three baseline models study functional references, relationships between elements, and visual position cues.

APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions

Toby Jia-Jun Li, Igor Labutov et al.

Introduces APPINITE, a conversational interface for specifying the intended targets of demonstrated Android actions. It constructs semantic UI representations and uses an on-screen overlay to preview and refine executable data descriptions.

GUI Testing Using Computer Vision

Tsung-Hsiang Chang, Tom Yeh et al.

Presents visual GUI test scripts that use images to specify actions and expected screen feedback. Recording both input events and screenshots can generate tests from demonstrations, supporting unit, regression, and test-driven workflows.

Sikuli: Using GUI screenshots for search and automation

Tom Yeh, Tsung-Hsiang Chang et al.

Introduces screenshot-based GUI search and visual scripting, using image patterns to locate interface targets and direct mouse and keyboard events. It evaluates screenshot queries and demonstrates automation and interactive-help tasks.

MMBench-GUI

Four-level GUI-agent benchmark across six platforms, from content understanding and element grounding to task automation and collaboration, scored with an efficiency-quality area metric.

OSWorld-G

Computer-use grounding benchmark with original and refined instructions, released with the Jedi synthetic data pipeline, a 4-million-example Jedi dataset and Jedi-3B and Jedi-7B models.

Precision CUA Bench

Evaluation resources for pixel-precise cursor grounding in code editors.

Screen Point-and-Read / Tree-of-Lens

Screen-reading benchmark and Tree-of-Lens implementation for interpreting user-indicated regions and their layout context.

ScreenSpot

GUI grounding test set of 1,272 screenshot instructions from iOS, Android, macOS, Windows and web interfaces, each labeled with the target element's bounding box.

ScreenSpot Pro

GUI grounding benchmark focused on professional high-resolution interfaces.

ScreenSpot v2

Second version of the ScreenSpot grounding test set, published by OS-Copilot as separate desktop, mobile and web annotation files with a shared screenshot archive.

ScreenSpot-v2 instruction variants

Instruction variants over ScreenSpot-v2 screenshots that distinguish original commands, explicit actions, target descriptions and impossible operations.

Web Context Bench (Python)

Local-page benchmark comparing six browser representations on token use, factual reading and target-element selection, with deterministic answer scoring.

WebClick

A web-interface localization benchmark with screenshot instructions and target boxes, including agent browsing, human browsing and calendar cases.

Aguvis stage 1 grounding data

The first-stage Aguvis training collection, packaging GUI element-grounding annotations and image archives from multiple interface-data sources.

Aria-UI Context-aware Data

Episode-based mobile GUI grounding data released with Aria-UI for incorporating interaction history into target localization.

Aria-UI Data

GUI element descriptions and grounding instructions spanning web, mobile, and Ubuntu desktop interfaces.

AutoGUI v1 training data

Screenshot-based grounding and captioning samples with UI functionality descriptions, target coordinates, and bounding boxes.

DragOn

Drag-grounding training and evaluation data pairing each screenshot instruction with start and end boxes, across text highlighting, spreadsheet cells, slide resizing and sliders.

GroundGUI RL Dataset

Desktop grounding prompts selected from GroundCUA and formatted for click-based reinforcement learning with target-box rewards.

Grounding-Anything-GUI

The public training archive linked by the Phi-Ground project for its Phi-Ground-Any model.

GUI-360

Desktop GUI trajectory data supporting multiple interface understanding tasks.

GUI-Actor data

A processed screenshot and bounding-box collection released for training GUI-Actor visual grounding models.

GUI-Drag

Data for learning text-dragging interactions in graphical interfaces.

GUI-Lasagne L1

The published GUI-Lasagne training-data collection used for SpiritSight, distributed as dated archive shards for visual interface learning.

GUI-Net-1M

A TongUI-collected training corpus from multimodal web tutorials, distributed as annotation files and split screenshot archives for GUI trajectory supervision.

GUI-R1 data bundle

Training and evaluation Parquet bundles accompanying GUI-R1, including GUI grounding and mobile/web action-prediction data.

GUIAct

Web and Android GUI action-instruction data, with single-step, multi-step and smartphone partitions.

GUIChat

Conversational GUI supervision with text-rich images and bounding boxes, organized as single-turn questions and multi-turn dialogues.

GUIEnv-local

The released local-supervision portion of GUIEnv, pairing webpage regions and text for OCR and GUI grounding.

Jedi

Synthetic GUI grounding data associated with UI decomposition research and the OSWorld-G benchmark.

OS-Atlas Data

GUI grounding pre-training data for OS-Atlas in desktop (Linux, macOS, Windows), mobile and web domains, with element boxes stored as relative left, top, right and bottom ratios.

ScreenDrag

GUI dragging data with screen observations, prior state, and continuous pointer actions, provided as trajectory files and Parquet representations.

ScreenRef

GUI screenshots paired with localization instructions and targets, including absent-target examples; image access is gated under publisher terms.

ScreenRef Annotations

Annotation files, task recipes, and image manifests for ScreenRef; screenshots reside in the separately gated image repository.

ScreenSpot-Pro data

High-resolution professional interface screenshots and target annotations for evaluating instruction-based GUI localization.

ShowUI-web

Web-interface grounding examples with instructions, element boxes and points, paired with a separately packaged screenshot archive.

ShowUI-π training data

Drag-and-click episode data for the ShowUI-π model, spanning desktop editing, presentation, handwriting and other interface tasks.

UGround V1 Data

Gated web grounding training data for UGround in Web-Hybrid and Web-Direct subsets, with a bounding-box version whose coordinates are normalized to 0-999.

UI-AGILE-Data

UI-AGILE training annotations and screenshots together with converted GUI-grounding and AndroidControl evaluation files.

UI-R1 3B training data

The released mobile training subset for the original UI-R1 3B model, with action annotations and individual screenshot files.

UI-Vision

Desktop GUI data for interface understanding and interaction research.

UIBert AppSim and RefExp annotations

Two Rico-derived UI component retrieval datasets: AppSim pairs semantically similar elements across screens, while RefExp links referring expressions to target elements.

VideoCAD data

CAD screen recordings paired with action logs and target model images for studying precise interface control and long-horizon visual imitation.

Wave UI

Screenshot-instruction records with target boxes and UI metadata such as OCR, source, element type and platform.

WebSRC v1.0

Webpage segments, questions, answers, HTML, screenshots and layout metadata for structural reading comprehension.

AgentCPM-GUI

8B on-device smartphone agent built on MiniCPM-V that operates Chinese and English apps from screenshots, trained with reinforcement fine-tuning and a compact JSON action space.

AgentCPM-GUI 8B checkpoint

An eight-billion-parameter MiniCPM-V-based model that consumes smartphone screenshots and predicts compact actions for Chinese and English Android app tasks.

Aguvis

Vision-based GUI agent models with unified action representations.

Aria-UI

Mixture-of-experts GUI grounding model with 3.9B activated parameters that can use text or interleaved image history, with released training data and an AndroidWorld agent.

Aria-UI context-aware checkpoint

The context-aware Aria-UI release that uses interaction context to support grounding in dynamic Android and desktop agent tasks.

Claude Computer Use

Claude tool capability for screenshot-based computer interaction.

CogAgent

Visual language model for GUI understanding and interaction.

CogAgent 9B 20241220

A bilingual GLM-4V-based execution model that uses screenshots, task instructions and action history to predict GUI operations.

DiMo-GUI

GUI grounding research using modality-aware visual reasoning and test-time scaling.