Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systems
Combines rapid target prediction with deliberate interface analysis, adaptively selecting the grounding process according to task complexity.
Cross-platform
GUI grounding
Source-reviewed research and assets tagged with GUI grounding.
Membership follows reviewed tags; it does not establish a dependency between neighboring records. Dates preserve source precision. Annotations retain their original language.
SpiritSight Agent: Advanced GUI Agent with One Look
Introduces SpiritSight and GUI-Lasagne data, combining multilevel GUI annotations with visual block parsing for high-resolution interface navigation.
Cross-platform
Smoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot Tasks
Uses inference of possible user queries from screenshots and target coordinates as a training bridge between GUI grounding and action reasoning.
Cross-platform
TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents
Combines hierarchical screen parsing with spatially enriched element descriptions to support both locating GUI targets and describing indicated elements.
Cross-platform
AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs
Infers element functionality from UI changes before and after interaction to produce verified annotations for grounding-model training.
Cross-platform
WinClick: GUI Grounding with Multimodal Large Language Models
Introduces a screenshot-based Windows GUI agent and the WinSpot benchmark for evaluating localization of actionable interface regions.
DesktopWindows
GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
Collects environment-specific interaction data through autonomous exploration to adapt grounding models to previously unseen GUIs.
Cross-platform
UI-TARS: Pioneering Automated GUI Interaction with Native Agents
Studies a screenshot-driven native GUI model with unified actions, deliberate reasoning, and iterative training on interaction traces.
Cross-platform
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection
Uses staged supervised training to combine GUI understanding and grounding with hierarchical reasoning and expectation-based reflection.
Cross-platform
Aria-UI: Visual Grounding for GUI Instructions
Introduces a vision-based grounding model trained with diverse synthetic instructions and action histories for context-sensitive GUI targeting.
Cross-platform
Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning
Locates GUI targets by identifying and aggregating attention maps in pretrained multimodal models without task-specific fine-tuning.
Cross-platform
WEPO: Web Element Preference Optimization for LLM-based Web Navigation
Uses web-element preference optimization with sampled negative HTML elements to align high-level user intent with navigation actions.
Web
Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining
Combines information-sensitive image cropping with a learning loop between element description and grounding for complex interface understanding.
WebDesktop
Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
Uses screenshots, a unified action space, and staged grounding and reasoning training for cross-platform GUI interaction.
Cross-platform
Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
Separates interpretation of user instructions from precise GUI localization using general-purpose and specialized visual-language components.
Cross-platformWebDesktop
ShowUI: One Vision-Language-Action Model for GUI Visual Agent
Combines visual token selection, interleaved visual-action history, and curated instruction data in a compact GUI action model.
Cross-platform
Improved GUI Grounding via Iterative Narrowing
Uses iterative visual narrowing to improve localization of requested GUI elements with general or specially trained vision-language models.
Cross-platform
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
Combines cross-platform GUI grounding data synthesis with action-model training to study generalization to unfamiliar interfaces.
Cross-platform
AutoGLM: Autonomous Foundation Agents for GUIs
Presents AutoGLM agents for browser and phone control, separating planning from grounding and training through an evolving interaction curriculum.
WebAndroid
EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data
Synthesizes GUI training examples at multiple levels of detail from webpages and studies transfer of learned understanding to other interface environments.
WebCross-platform
Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms
Extends Ferret-UI across device platforms with adaptive visual scaling and synthetic data for referring, grounding, and user-centered UI tasks.
iOSAndroidWebCross-platform
Harnessing Webpage UIs for Text-Rich Visual Understanding
Builds MultiUI by generating instructions from webpage accessibility trees and pairing them with screenshots for multimodal training. Evaluation examines web-interface understanding, including VisualWebBench and Mind2Web, and transfer to other text-rich visual tasks.
Web
ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents
Separates multimodal reasoning and action planning from a dedicated UI localization model, with evaluation on Android devices.
Android
TinyClick: Single-Turn Agent for Empowering GUI Automation
Adapts a compact Florence-2 model to map user instructions to screen coordinates using multitask training and augmented data.
Cross-platform
Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
Introduces UGround, a visual target-localization model trained with large-scale synthetic references to elements in GUI screenshots.
Cross-platform
The Impact of Element Ordering on LM Agent Performance
Studies how the ordering of UI elements in an agent observation affects web and desktop performance. A pixel-based element detector and dimensionality-reduction ordering are evaluated on OmniACT when document hierarchy is unavailable.
WebDesktop
Grounded GUI Understanding for Vision-Based Spatial Intelligent Agent: Exemplified by Extended Reality Apps
Orienter detects interactable GUI elements in XR applications by interpreting scene context and revisiting candidate elements through a validation and reflection loop. It addresses open-vocabulary element types and context-dependent interactability.
From Grounding to Planning: Benchmarking Bottlenecks in Web Agents
Separates web-agent planning and grounding evaluations on Mind2Web to analyze which components limit task performance.
Web
OmniParser for Pure Vision Based GUI Agent
Parses screenshots into interactable regions and semantic element descriptions to support visually grounded agent actions.
Cross-platform
AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
Introduces Android screenshots with element locations, functionality descriptions, and instructions linked to sequences of GUI actions.
Android
Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding
Introduces Screen Point-and-Read and a hierarchical layout approach that interprets screen regions indicated by a user-selected point.
Cross-platform
VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning
Uses image-dependent question-answer data and staged fine-tuning to improve GUI comprehension and reduce unsupported visual answers.
Cross-platform
GUICourse: From General Vision Language Models to Versatile GUI Agents
Introduces GUIEnv, GUIAct, and GUIChat datasets to develop OCR, grounding, component knowledge, and interaction abilities in visual-language models.
Cross-platform
V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel Multimodal LLM
Describes V-Zen, a multimodal GUI model with dual-resolution image encoders for grounding and next-action prediction, fine-tuned using GUIDE interface elements and task sequences.
Visual Grounding Methods for Efficient Interaction with Desktop Graphical User Interfaces
Compares modular OCR-based and end-to-end methods for locating desktop GUI targets from natural-language instructions.
Desktop
GUIDE: Graphical User Interface Data for Execution
Introduces annotated website task trajectories with screenshots, prior actions, reasoning, and grounded next actions for process automation.
Web
VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
Separately evaluates web-page reading, understanding, and element grounding across tasks drawn from real websites.
Web
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
Adapts multimodal modeling to mobile interfaces through detailed screen crops, region annotations, and UI reasoning tasks.
ReALM: Reference Resolution As Language Modeling
Frames conversational and on-screen reference resolution as language modeling, allowing textual representations of screen entities to support user-reference interpretation. Evaluation separately examines on-screen references against an existing resolver and language-model baselines.
Adjacent research
AgentStudio: A Toolkit for Building General Virtual Agents
AgentStudio combines an interactive GUI/API environment with tools for creating tasks, annotating interface elements, and labeling actions in videos. Its online suite and GroundUI, IDMBench, and CriticBench datasets examine interaction, grounding, learning from video, and success detection.
UI Semantic Group Detection: Grouping UI Elements with Similar Semantics in Mobile Graphical User Interface
UISCGD detects semantic groups of adjacent text and non-text UI elements. It learns from Android and iOS screenshots to support interface retrieval, UI-to-code structure and screen-reader accessibility.
AndroidiOS
OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
Introduces a dataset and benchmark for generating executable task scripts from screenshots and visually grounded desktop or web instructions.
DesktopWeb
UFO: A UI-Focused Agent for Windows OS Interaction
UFO uses two agents to inspect Windows application interfaces and their control metadata, then grounds and executes user requests within and across applications. The study evaluates this UI-focused architecture on nine Windows applications.
WindowsDesktop
Dual-View Visual Contextualization for Web Navigation
Uses screenshot geometry and visual content to enrich HTML element representations for web navigation. Nearby elements contribute textual and visual context, with evaluation on cross-task, cross-website, and cross-domain Mind2Web splits.
Web
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Introduces visually grounded website tasks for evaluating multimodal agents that must interpret both images and text.
Web
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
Trains a screenshot-driven GUI agent with grounding pretraining and introduces ScreenSpot for evaluating interface target localization.
Cross-platform
GPT-4V(ision) is a Generalist Web Agent, if Grounded
Introduces SeeAct and live-website evaluation, examining how visual planning and element grounding affect web-agent task completion.
Web
CogAgent: A Visual Language Model for GUI Agents
Introduces a visual-language model with low- and high-resolution image encoders for screenshot understanding and GUI navigation.
DesktopAndroidWeb
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation
Studies MM-Navigator, a multimodal agent that interprets smartphone screenshots and predicts localized actions without task-specific training.
iOSAndroid
Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
Overlays labeled image regions to help multimodal models refer to and reason about visual targets; a general visual-prompting method relevant to GUI grounding.
Adjacent research
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations
ILuvUI generates 335,000 conversational training examples paired with UI screenshots, covering questions, descriptions, and plans. The resulting vision-language model is assessed on element detection, response quality, and examples of multi-step UI navigation.
Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API
Trains a screenshot-based instruction-grounding model to predict target coordinates, using reinforcement learning to improve spatial decoding.
Cross-platform
From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
Studies instruction-following agents that observe screenshots and act through a generic mouse-and-keyboard action space, evaluating pixel-based pretraining for GUI tasks on MiniWoB++.
Web
Lexi: Self-Supervised Learning of the UI Language
Introduces Lexi and UICaption to learn UI image-text representations from descriptions of interface functionality. Evaluation covers UI action entailment, instruction-based retrieval, referring-expression grounding, and UI entity recognition.
UGIF: UI Grounded Instruction Following
Introduces multilingual smartphone instruction-following data and a pipeline that retrieves help instructions and maps them to UI actions.
Towards Better Semantic Understanding of Mobile Interfaces
Augments RICO screens and view hierarchies with human labels for icon shape and meaning and for links between UI elements and text labels. It compares image-only and multimodal models on semantic understanding of previously unseen apps.
MUG: Interactive Multimodal Grounding on User Interfaces
Studies iterative UI grounding in which users refine or correct an agent's interpretation over multiple conversational turns.
Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus
Introduces Spotlight, a vision-language model conditioned on a UI screenshot and a selected region of interest. It evaluates screenshot-based mobile UI understanding and explores multitask learning and few-shot prompting.
VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling
The Versatile UI Transformer jointly models interface screenshots, structure, and language for five UI tasks. Its encoder supports element detection when structural metadata is absent, while an autoregressive component handles UI question answering and command grounding.
Understanding Mobile GUI: from Pixel-Words to Screen-Sentences
Models mobile screens as atomic text and graphic components whose relationships are learned by a Screen Transformer. The work introduces RICO-PW annotations and evaluates metadata-free screen understanding on relation, clickability, retrieval, and app-classification tasks.
Showing 181–240 of 380 resources.
Get full records and evidence through the topic export manifest. Source records can be passed to the standard RAG and citation exporters after cloning the catalog.