Screen understanding

VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning

Ziyang Meng, Yu Dai et al.

Uses image-dependent question-answer data and staged fine-tuning to improve GUI comprehension and reduce unsupported visual answers.

GUI Action Narrator: Where and When Did That Action Take Place?

Qinchen Wu, Difei Gao et al.

Introduces Act2Cap and a cursor-guided method for describing GUI actions from video, with temporal and spatial localization challenges.

GUICourse: From General Vision Language Models to Versatile GUI Agents

Wentong Chen, Junbo Cui et al.

Introduces GUIEnv, GUIAct, and GUIChat datasets to develop OCR, grounding, component knowledge, and interaction abilities in visual-language models.

GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding

Dongping Chen, Yue Huang et al.

Introduces video-based GUI questions and annotations for studying temporal interface understanding across varied interaction settings.

Practical, Automated Scenario-based Mobile App Testing

Shengcheng Yu, Chunrong Fang et al.

ScenTest builds scenario-specific event knowledge graphs from crowdsourced test reports containing GUI and action information. It uses the current app state and testing context to generate scenario tests across applications.

Tell Me What's Next: Textual Foresight for Generic UI Representations

Andrea Burns, Kate Saenko et al.

Textual Foresight learns UI representations by describing the future screen expected after an action on the current interface. The OpenApp dataset supports pretraining and downstream evaluation of representations that connect individual elements with whole-screen changes.

CAAP: Context-Aware Action Planning Prompting to Solve Computer Tasks with Front-End UI Only

Junhee Cho, Jihoon Kim et al.

CAAP converts screenshots into textual observations and plans keyboard-and-mouse actions using a prompt that examines task context from multiple perspectives. The agent is evaluated on MiniWoB++ and WebShop using only front-end visual observations.

V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel Multimodal LLM

Abdur Rahman, Rajat Chawla et al.

Describes V-Zen, a multimodal GUI model with dual-resolution image encoders for grounding and next-action prediction, fine-tuned using GUIDE interface elements and task sequences.

SmartFlow: Robotic Process Automation using LLMs

Arushi Jain, Shubham Paliwal et al.

Presents SmartFlow, which converts visible GUI elements into text for language-model action planning and executes the plan through a scripting engine. A collection of enterprise applications evaluates adaptation to different layouts and tasks.

MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling

Sidong Feng, Suyu Ma et al.

MUD combines LLM-guided Android app exploration, noise filtering and human annotation to collect modern UI screenshots. The study evaluates the resulting data for element detection and UI retrieval.

VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?

Junpeng Liu, Yifan Song et al.

Separately evaluates web-page reading, understanding, and element grounding across tasks drawn from real websites.

Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

Keen You, Haotian Zhang et al.

Adapts multimodal modeling to mobile interfaces through detailed screen crops, region annotations, and UI reasoning tasks.

ReALM: Reference Resolution As Language Modeling

Joel Ruben Antony Moniz, Soundarya Krishnan et al.

Frames conversational and on-screen reference resolution as language modeling, allowing textual representations of screen entities to support user-reference interpretation. Evaluation separately examines on-screen references against an existing resolver and language-model baselines.

Computer User Interface Understanding. A New Dataset and a Learning Framework

Andrés Muñoz, Daniel Borrajo

Introduces desktop interaction videos and a framework combining synthetic examples with hierarchical contrastive learning for interface classification.

UI Semantic Group Detection: Grouping UI Elements with Similar Semantics in Mobile Graphical User Interface

Shuhong Xiao, Yunnong Chen et al.

UISCGD detects semantic groups of adjacent text and non-text UI elements. It learns from Android and iOS screenshots to support interface retrieval, UI-to-code structure and screen-reader accessibility.

Cradle: Empowering Foundation Agents Towards General Computer Control

Weihao Tan, Wentao Zhang et al.

Uses screenshot observations, planning modules, and generated low-level keyboard and mouse control for general computer interaction.

Improving Language Understanding from Screenshots

Tianyu Gao, Zirui Wang et al.

Studies language understanding from text-rendered screenshots using joint image-patch and text prediction; a perception foundation relevant to screen-reading models.

CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation

Xinbei Ma, Zhuosheng Zhang et al.

Combines screenshots, layout information, and action history with a factored action predictor for smartphone automation.

ScreenAI: A Vision-Language Model for UI and Infographics Understanding

Gilles Baechler, Srinivas Sunkara et al.

Introduces ScreenAI and screen-annotation training that supports UI question answering, navigation, and summarization data generation.

Dual-View Visual Contextualization for Web Navigation

Jihyung Kil, Chan Hee Song et al.

Uses screenshot geometry and visual content to enrich HTML element representations for web navigation. Nearby elements contribute textual and visual context, with evaluation on cross-task, cross-website, and cross-domain Mind2Web splits.

VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning

Yunpeng Song, Yiheng Bian et al.

VisionTasker converts screenshots into textual UI interpretations and presents one interface at a time to an LLM planner for action selection. It evaluates visual representations and execution of 147 tasks on an Android smartphone, including integration with programming by demonstration.

CogAgent: A Visual Language Model for GUI Agents

Wenyi Hong, Weihan Wang et al.

Introduces a visual-language model with low- and high-resolution image encoders for screenshot understanding and GUI navigation.

UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model

Jiabo Ye, Anwen Hu et al.

Introduces UReader, which combines unified instruction tuning, auxiliary text-reading tasks and shape-adaptive image crops for OCR-free visual language understanding. Its evaluation includes webpage screenshots alongside documents, tables, charts and natural images.

ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations

Yue Jiang, Eldon Schoop et al.

ILuvUI generates 335,000 conversational training examples paired with UI screenshots, covering questions, descriptions, and plans. The resulting vision-language model is assessed on element detection, response quality, and examples of multi-step UI navigation.

WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics

Jason Wu, Siyan Wang et al.

Collects rendered webpages and structural metadata for visual UI modeling, studying how web semantics transfer to interface-understanding tasks.

Lexi: Self-Supervised Learning of the UI Language

Pratyay Banerjee, Shweti Mahajan et al.

Introduces Lexi and UICaption to learn UI image-text representations from descriptions of interface functionality. Evaluation covers UI action entailment, instruction-based retrieval, referring-expression grounding, and UI entity recognition.

Understanding HTML with Large Language Models

Izzeddin Gur, Ofir Nachum et al.

Studies language models for HTML element classification, page description and autonomous web navigation. Fine-tuned models are evaluated on MiniWoB, and a distilled, automatically labeled CommonCrawl HTML corpus supports the training approach.

Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding

Kenton Lee, Mandar Joshi et al.

Pix2Struct pretrains an image-to-text model by reconstructing simplified HTML from masked webpage screenshots. Its variable-resolution visual representation is evaluated across several domains, including user-interface understanding.

Towards Better Semantic Understanding of Mobile Interfaces

Srinivas Sunkara, Maria Wang et al.

Augments RICO screens and view hierarchies with human labels for icon shape and meaning and for links between UI elements and text labels. It compares image-only and multimodal models on semantic understanding of previously unseen apps.

Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus

Gang Li, Yang Li

Introduces Spotlight, a vision-language model conditioned on a UI screenshot and a selected region of interest. It evaluates screenshot-based mobile UI understanding and explores multitask learning and few-shot prompting.

Enabling Conversational Interaction with Mobile UI using Large Language Models

Bryan Wang, Gang Li et al.

Studies few-shot prompting of a single language model for four mobile-UI interaction tasks, aiming to support conversational interface use without a separate trained model for each task.

ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots

Yu-Chung Hsiao, Fedir Zubach et al.

Introduces screenshot question answering over mobile interfaces, linking answers to screen content and bounding boxes.

Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale

Gang Li, Gilles Baechler et al.

Introduces CLAY, a pipeline that repairs noisy mobile UI layouts by removing nodes without valid visual counterparts and assigning semantic element types, supported by human-annotated layouts for evaluation.

VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling

Yang Li, Gang Li et al.

The Versatile UI Transformer jointly models interface screenshots, structure, and language for five UI tasks. Its encoder supports element detection when structural metadata is absent, while an autoregressive component handles UI question answering and command grounding.

Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning

Bryan Wang, Gang Li et al.

Studies mobile-screen summarization using visual, textual, and structural UI information, with a dataset of human-written descriptions.

UIBert: Learning Generic Multimodal Representations for UI Understanding

Chongyang Bai, Xiaoxue Zang et al.

Learns shared image, text, and structural UI representations through self-supervised tasks based on relationships among interface components.

Understanding Mobile GUI: from Pixel-Words to Screen-Sentences

Jingwen Fu, Xiaoyi Zhang et al.

Models mobile screens as atomic text and graphic components whose relationships are learned by a Screen Transformer. The work introduces RICO-PW annotations and evaluates metadata-free screen understanding on relation, clickability, retrieval, and app-classification tasks.

WebSRC: A Dataset for Web-Based Structural Reading Comprehension

Xingyu Chen, Zihan Zhao et al.

Introduces question answering over webpage text and structure, with aligned HTML, screenshots, and metadata for structural reading comprehension.

Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

Xiaoyi Zhang, Lilian de Greef et al.

Infers UI elements and accessibility metadata from mobile screenshots to support screen-reader interaction with iOS applications.

ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces

Zecheng He, Srinivas Sunkara et al.

Pretrains ActionBert on user interaction traces to combine visual, linguistic, and UI-structure features. It evaluates the resulting UI representations on tasks including icon classification and description-based component retrieval.

Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements

Yang Li, Gang Li et al.

Introduces crowdsourced descriptions of mobile UI elements and studies multimodal models that generate useful widget captions.

Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?

Jieshan Chen, Mulong Xie et al.

Compares seven traditional and learned GUI element detectors on more than 50,000 interface images. The analysis motivates a coarse-to-fine non-text detector combined with a learned text detector to address GUI-specific localization requirements.

Rico: A Mobile App Dataset for Building Data-Driven Design Applications

Biplab Deka, Zifeng Huang et al.

Introduces Rico, collecting Android interface designs and interaction traces through crowdsourced use followed by automated exploration. The paper demonstrates UI-layout retrieval using learned representations and describes visual, textual, structural, and interaction data.

Sikuli: Using GUI screenshots for search and automation

Tom Yeh, Tsung-Hsiang Chang et al.

Introduces screenshot-based GUI search and visual scripting, using image patterns to locate interface targets and direct mouse and keyboard events. It evaluates screenshot queries and demonstrates automation and interactive-help tasks.

OS-Critic Bench

A cross-platform GUI-critic benchmark with mobile, desktop and web JSONL splits and screenshot assets, accompanied by public inference and accuracy-scoring scripts.

Screen Point-and-Read / Tree-of-Lens

Screen-reading benchmark and Tree-of-Lens implementation for interpreting user-indicated regions and their layout context.

VisualWebArena

Visually grounded web tasks extending the WebArena evaluation setting.

VisualWebBench

Evaluation of multimodal models on visual web understanding tasks.

Web Context Bench (Python)

Local-page benchmark comparing six browser representations on token use, factual reading and target-element selection, with deterministic answer scoring.

WONDERBREAD

An offline benchmark of workflow documentation, quality assessment and knowledge transfer using human web-navigation demonstrations, action traces and written procedures.

AndroidCode

HTML and annotation release used for Code2World’s action-conditioned GUI-state prediction training, constructed from mobile GUI trajectories.

Aria-UI Data

GUI element descriptions and grounding instructions spanning web, mobile, and Ubuntu desktop interfaces.

AutoGUI v1 training data

Screenshot-based grounding and captioning samples with UI functionality descriptions, target coordinates, and bounding boxes.

GUI-Lasagne L1

The published GUI-Lasagne training-data collection used for SpiritSight, distributed as dated archive shards for visual interface learning.

GUI-World

GUI video benchmark and training data covering six scenarios, from Android and iOS to websites and XR, with eight question types for image and video LLMs and the GUI-Vid model.

GUIMid training data

A released GUI-agent mid-training mixture and supporting domain datasets, together with a post-training trajectory file and image archives for web and mobile interaction.

Mobile3M

Pre-training corpus for Xiaomi's MobileVLM, distributed as per-app graph archives for mobile apps such as Baidu Map, QQ Music, Ctrip and Vipshop.

MobileViews

Over 600,000 Android screenshot and view-hierarchy pairs from more than 20,000 Google Play apps, collected with an optimized DroidBot, with complete per-app traces.

MultiUI

7.3 million training samples built from webpage UIs and structured with enhanced accessibility trees, for text-rich visual understanding, released with UIX-Qwen2 models.

Rico

Mobile app screenshots, view hierarchies, and interaction traces for UI research.