Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systems
Combines rapid target prediction with deliberate interface analysis, adaptively selecting the grounding process according to task complexity.
跨平台
GUI 定位
已审阅并标注「GUI 定位」的研究与资产。
条目依据已审阅标签归类，同一主题不表示资源之间存在依赖关系。日期保留来源精度，注释保留原语言。
SpiritSight Agent: Advanced GUI Agent with One Look
Introduces SpiritSight and GUI-Lasagne data, combining multilevel GUI annotations with visual block parsing for high-resolution interface navigation.
跨平台
Smoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot Tasks
Uses inference of possible user queries from screenshots and target coordinates as a training bridge between GUI grounding and action reasoning.
跨平台
TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents
Combines hierarchical screen parsing with spatially enriched element descriptions to support both locating GUI targets and describing indicated elements.
跨平台
AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs
Infers element functionality from UI changes before and after interaction to produce verified annotations for grounding-model training.
跨平台
WinClick: GUI Grounding with Multimodal Large Language Models
Introduces a screenshot-based Windows GUI agent and the WinSpot benchmark for evaluating localization of actionable interface regions.
桌面Windows
GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
Collects environment-specific interaction data through autonomous exploration to adapt grounding models to previously unseen GUIs.
跨平台
UI-TARS: Pioneering Automated GUI Interaction with Native Agents
Studies a screenshot-driven native GUI model with unified actions, deliberate reasoning, and iterative training on interaction traces.
跨平台
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection
Uses staged supervised training to combine GUI understanding and grounding with hierarchical reasoning and expectation-based reflection.
跨平台
Aria-UI: Visual Grounding for GUI Instructions
Introduces a vision-based grounding model trained with diverse synthetic instructions and action histories for context-sensitive GUI targeting.
跨平台
Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning
Locates GUI targets by identifying and aggregating attention maps in pretrained multimodal models without task-specific fine-tuning.
跨平台
WEPO: Web Element Preference Optimization for LLM-based Web Navigation
Uses web-element preference optimization with sampled negative HTML elements to align high-level user intent with navigation actions.
网页
Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining
Combines information-sensitive image cropping with a learning loop between element description and grounding for complex interface understanding.
网页桌面
Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
Uses screenshots, a unified action space, and staged grounding and reasoning training for cross-platform GUI interaction.
跨平台
Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
Separates interpretation of user instructions from precise GUI localization using general-purpose and specialized visual-language components.
跨平台网页桌面
ShowUI: One Vision-Language-Action Model for GUI Visual Agent
Combines visual token selection, interleaved visual-action history, and curated instruction data in a compact GUI action model.
跨平台
Improved GUI Grounding via Iterative Narrowing
Uses iterative visual narrowing to improve localization of requested GUI elements with general or specially trained vision-language models.
跨平台
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
Combines cross-platform GUI grounding data synthesis with action-model training to study generalization to unfamiliar interfaces.
跨平台
AutoGLM: Autonomous Foundation Agents for GUIs
Presents AutoGLM agents for browser and phone control, separating planning from grounding and training through an evolving interaction curriculum.
网页Android
EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data
Synthesizes GUI training examples at multiple levels of detail from webpages and studies transfer of learned understanding to other interface environments.
网页跨平台
Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms
Extends Ferret-UI across device platforms with adaptive visual scaling and synthetic data for referring, grounding, and user-centered UI tasks.
iOSAndroid网页跨平台
Harnessing Webpage UIs for Text-Rich Visual Understanding
Builds MultiUI by generating instructions from webpage accessibility trees and pairing them with screenshots for multimodal training. Evaluation examines web-interface understanding, including VisualWebBench and Mind2Web, and transfer to other text-rich visual tasks.
网页
ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents
Separates multimodal reasoning and action planning from a dedicated UI localization model, with evaluation on Android devices.
Android
TinyClick: Single-Turn Agent for Empowering GUI Automation
Adapts a compact Florence-2 model to map user instructions to screen coordinates using multitask training and augmented data.
跨平台
Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
Introduces UGround, a visual target-localization model trained with large-scale synthetic references to elements in GUI screenshots.
跨平台
The Impact of Element Ordering on LM Agent Performance
Studies how the ordering of UI elements in an agent observation affects web and desktop performance. A pixel-based element detector and dimensionality-reduction ordering are evaluated on OmniACT when document hierarchy is unavailable.
网页桌面
Grounded GUI Understanding for Vision-Based Spatial Intelligent Agent: Exemplified by Extended Reality Apps
Orienter detects interactable GUI elements in XR applications by interpreting scene context and revisiting candidate elements through a validation and reflection loop. It addresses open-vocabulary element types and context-dependent interactability.
From Grounding to Planning: Benchmarking Bottlenecks in Web Agents
Separates web-agent planning and grounding evaluations on Mind2Web to analyze which components limit task performance.
网页
OmniParser for Pure Vision Based GUI Agent
Parses screenshots into interactable regions and semantic element descriptions to support visually grounded agent actions.
跨平台
AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
Introduces Android screenshots with element locations, functionality descriptions, and instructions linked to sequences of GUI actions.
Android
Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding
Introduces Screen Point-and-Read and a hierarchical layout approach that interprets screen regions indicated by a user-selected point.
跨平台
VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning
Uses image-dependent question-answer data and staged fine-tuning to improve GUI comprehension and reduce unsupported visual answers.
跨平台
GUICourse: From General Vision Language Models to Versatile GUI Agents
Introduces GUIEnv, GUIAct, and GUIChat datasets to develop OCR, grounding, component knowledge, and interaction abilities in visual-language models.
跨平台
V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel Multimodal LLM
Describes V-Zen, a multimodal GUI model with dual-resolution image encoders for grounding and next-action prediction, fine-tuned using GUIDE interface elements and task sequences.
Visual Grounding Methods for Efficient Interaction with Desktop Graphical User Interfaces
Compares modular OCR-based and end-to-end methods for locating desktop GUI targets from natural-language instructions.
桌面
GUIDE: Graphical User Interface Data for Execution
Introduces annotated website task trajectories with screenshots, prior actions, reasoning, and grounded next actions for process automation.
网页
VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
Separately evaluates web-page reading, understanding, and element grounding across tasks drawn from real websites.
网页
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
Adapts multimodal modeling to mobile interfaces through detailed screen crops, region annotations, and UI reasoning tasks.
ReALM: Reference Resolution As Language Modeling
Frames conversational and on-screen reference resolution as language modeling, allowing textual representations of screen entities to support user-reference interpretation. Evaluation separately examines on-screen references against an existing resolver and language-model baselines.
相关研究
AgentStudio: A Toolkit for Building General Virtual Agents
AgentStudio combines an interactive GUI/API environment with tools for creating tasks, annotating interface elements, and labeling actions in videos. Its online suite and GroundUI, IDMBench, and CriticBench datasets examine interaction, grounding, learning from video, and success detection.
UI Semantic Group Detection: Grouping UI Elements with Similar Semantics in Mobile Graphical User Interface
UISCGD detects semantic groups of adjacent text and non-text UI elements. It learns from Android and iOS screenshots to support interface retrieval, UI-to-code structure and screen-reader accessibility.
AndroidiOS
OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
Introduces a dataset and benchmark for generating executable task scripts from screenshots and visually grounded desktop or web instructions.
桌面网页
UFO: A UI-Focused Agent for Windows OS Interaction
UFO uses two agents to inspect Windows application interfaces and their control metadata, then grounds and executes user requests within and across applications. The study evaluates this UI-focused architecture on nine Windows applications.
Windows桌面
Dual-View Visual Contextualization for Web Navigation
Uses screenshot geometry and visual content to enrich HTML element representations for web navigation. Nearby elements contribute textual and visual context, with evaluation on cross-task, cross-website, and cross-domain Mind2Web splits.
网页
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Introduces visually grounded website tasks for evaluating multimodal agents that must interpret both images and text.
网页
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
Trains a screenshot-driven GUI agent with grounding pretraining and introduces ScreenSpot for evaluating interface target localization.
跨平台
GPT-4V(ision) is a Generalist Web Agent, if Grounded
Introduces SeeAct and live-website evaluation, examining how visual planning and element grounding affect web-agent task completion.
网页
CogAgent: A Visual Language Model for GUI Agents
Introduces a visual-language model with low- and high-resolution image encoders for screenshot understanding and GUI navigation.
桌面Android网页
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation
Studies MM-Navigator, a multimodal agent that interprets smartphone screenshots and predicts localized actions without task-specific training.
iOSAndroid
Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
Overlays labeled image regions to help multimodal models refer to and reason about visual targets; a general visual-prompting method relevant to GUI grounding.
相关研究
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations
ILuvUI generates 335,000 conversational training examples paired with UI screenshots, covering questions, descriptions, and plans. The resulting vision-language model is assessed on element detection, response quality, and examples of multi-step UI navigation.
Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API
Trains a screenshot-based instruction-grounding model to predict target coordinates, using reinforcement learning to improve spatial decoding.
跨平台
From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
Studies instruction-following agents that observe screenshots and act through a generic mouse-and-keyboard action space, evaluating pixel-based pretraining for GUI tasks on MiniWoB++.
网页
Lexi: Self-Supervised Learning of the UI Language
Introduces Lexi and UICaption to learn UI image-text representations from descriptions of interface functionality. Evaluation covers UI action entailment, instruction-based retrieval, referring-expression grounding, and UI entity recognition.
UGIF: UI Grounded Instruction Following
Introduces multilingual smartphone instruction-following data and a pipeline that retrieves help instructions and maps them to UI actions.
Towards Better Semantic Understanding of Mobile Interfaces
Augments RICO screens and view hierarchies with human labels for icon shape and meaning and for links between UI elements and text labels. It compares image-only and multimodal models on semantic understanding of previously unseen apps.
MUG: Interactive Multimodal Grounding on User Interfaces
Studies iterative UI grounding in which users refine or correct an agent's interpretation over multiple conversational turns.
Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus
Introduces Spotlight, a vision-language model conditioned on a UI screenshot and a selected region of interest. It evaluates screenshot-based mobile UI understanding and explores multitask learning and few-shot prompting.
VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling
The Versatile UI Transformer jointly models interface screenshots, structure, and language for five UI tasks. Its encoder supports element detection when structural metadata is absent, while an autoregressive component handles UI question answering and command grounding.
Understanding Mobile GUI: from Pixel-Words to Screen-Sentences
Models mobile screens as atomic text and graphic components whose relationships are learned by a Screen Transformer. The work introduces RICO-PW annotations and evaluates metadata-free screen understanding on relation, clickability, retrieval, and app-classification tasks.
第 181–240 条，共 380 条资源。
通过主题导出清单获取完整记录和证据。克隆目录后，可将主题记录直接交给标准 RAG 与引用导出工具。