Screen understanding
Source-reviewed research and assets tagged with Screen understanding.
Membership follows reviewed tags; it does not establish a dependency between neighboring records. Dates preserve source precision. Annotations retain their original language.
ScreenQA
About 86,000 human-written question-answer pairs over 35,000 Rico mobile screenshots, plus ScreenQA Short and ComplexQA variants, for benchmarking screen content understanding.
ScreenRef
GUI screenshots paired with localization instructions and targets, including absent-target examples; image access is gated under publisher terms.
UIBert AppSim and RefExp annotations
Two Rico-derived UI component retrieval datasets: AppSim pairs semantically similar elements across screens, while RefExp links referring expressions to target elements.
UICrit
Mobile-interface critiques and quality ratings with region coordinates and RICO screen references, including a larger three-annotator release than the original paper dataset.
Wave UI
Screenshot-instruction records with target boxes and UI metadata such as OCR, source, element type and platform.
WebSRC v1.0
Webpage segments, questions, answers, HTML, screenshots and layout metadata for structural reading comprehension.
Code2World-8B checkpoint
Released GUI world-model checkpoint that turns a screenshot and proposed action into HTML for rendering a predicted next screen.
CogAgent 9B 20241220
A bilingual GLM-4V-based execution model that uses screenshots, task instructions and action history to predict GUI operations.
GroundGUI-8B
Screenshot grounding checkpoint that emits a single click in a normalized coordinate space.
OmniParser v2 model bundle
A screen-parsing weight bundle containing icon-region detectors and an icon captioner for converting screenshots into agent-readable interface elements.
POINTS-GUI-G 8B
A specialized interface-grounding checkpoint that returns normalized point coordinates for a requested element in a screenshot.
SpiritSight-Agent 8B base
An InternVL2-based visual GUI checkpoint distributed for grounding and interface navigation, with the publisher recommending task-specific fine-tuning of the base release.
UIX-Qwen2
A MultiUI-trained visual-language checkpoint for text-rich webpage understanding and interface tasks.
ZonUI-3B checkpoint
Qwen2.5-VL-3B-based GUI grounding checkpoint that predicts click locations from screenshots and text instructions across desktop, mobile and web screens.
CAAP Agent
Screenshot-driven agent implementation for MiniWoB++ and WebShop, using a visual observer and context-aware prompts to issue mouse and keyboard actions.
CoCo-Agent
Research implementation combining screen and layout perception with conditional action prediction for smartphone automation.
Mobilerun Portal (formerly Droidrun Portal)
An Android accessibility service that exposes UI-element inspection, device-control interfaces and screen feedback to mobile automation agents.
OmniParser
Screenshot parser that detects interactable regions and captions icons so vision-language models can ground actions, released as V1.5 and V2 weights alongside OmniTool for driving a Windows 11 VM.
Seg2Click
Training and inference implementation that predicts an executable-region mask and derives a GUI click from that mask.
Sikuli legacy research implementation
The original Sikuli source distribution combines screenshot targets with a Jython scripting library and IDE for visual GUI automation and testing.
Tarsier
Perception utilities for web agents that tag interactable elements with typed IDs an LLM can act on, and use OCR to render page screenshots as whitespace-structured text for models without vision.
UI Defect Mocker
Synthesizes visual UI defects from mobile screenshots and Android layout metadata to support GUI-testing experiments.
VisionTasker
An Android automation implementation that translates screenshot-based UI understanding into language descriptions for step-by-step LLM planning and action execution.
WebMarker
A browser library that labels interactive elements and their boxes for visual prompting, preserving element mappings that automation code can use to execute selected actions.
Windows-MCP
A Windows MCP bridge exposing application, window, keyboard and mouse operations plus native UI-state inspection for agents.
Yunkai
Android phone assistant combining model-driven tool use with accessibility-tree reading, screenshots, and tap, swipe and text-entry primitives. Its documented interaction flow presents operation plans for user confirmation.
Web Browser Automation with Agents (smolagents tutorial)
Versioned tutorial building a vision-guided CodeAgent with Selenium/Helium browser tools, screenshot callbacks and example web-navigation tasks.
Showing 181–212 of 212 resources.
Get full records and evidence through the topic export manifest. Source records can be passed to the standard RAG and citation exporters after cloning the catalog.