GroundGUI training and evaluation code
Code for screenshot grounding through supervised training, teacher-guided distillation, reinforcement learning, and benchmark evaluation.
DesktopWebCross-platform
GUI grounding
Source-reviewed research and assets tagged with GUI grounding.
Membership follows reviewed tags; it does not establish a dependency between neighboring records. Dates preserve source precision. Annotations retain their original language.
GUI Grounding Evaluation Harness
ScreenSpot-Pro evaluation harness with explicit coordinate transforms, prompt and crop-based inference comparisons, and per-sample grounding accuracy and cost records.
GUI-RC / GUI-RCPO
GUI-grounding research code for spatial voting across sampled predictions and test-time policy optimization using region-consistency rewards.
Cross-platformDesktopWeb
GUIPivot
Query-inference training tools connecting screenshot grounding with GUI action reasoning, including preprocessing, query refinement and offline single-step evaluation on AndroidControl and AITZ.
Android
MAI-UI inference and evaluation code
Published MAI-UI grounding and navigation-agent implementations with inference demos and evaluation scripts.
Android
OculiX (SikuliX continuation)
A maintained continuation of SikuliX that locates on-screen targets with image matching and OCR, drives desktop or Android interfaces, and exposes scripting and optional MCP interfaces.
DesktopWindowsmacOSLinuxAndroid
OmniParser
Screenshot parser that detects interactable regions and captions icons so vision-language models can ground actions, released as V1.5 and V2 weights alongside OmniTool for driving a Windows 11 VM.
Cross-platform
ScreenSuite
A shared evaluation harness for GUI-model perception, offline action prediction and multi-step interaction, using simple smolagents-based agents across existing benchmarks.
WebAndroidDesktop
SeeAct
Generalist web agent framework for large multimodal models such as GPT-4V, with a Playwright tool that runs agents on live websites, text-choice and set-of-mark grounding, and a crawler mode.
Web
Seg2Click
Training and inference implementation that predicts an executable-region mask and derives a GUI click from that mask.
Cross-platform
Self Operating Computer
Framework, released in November 2023, in which a multimodal model views the screen and picks mouse and keyboard actions toward a goal, with backends such as GPT-4o, Claude 3, Gemini and LLaVA.
Desktop
Set of Mark
Visual-prompting toolbox that labels image regions for multimodal reference and grounding, including GUI applications.
Adjacent research
Sikuli legacy research implementation
The original Sikuli source distribution combines screenshot targets with a Jython scripting library and IDE for visual GUI automation and testing.
DesktopmacOSLinuxWindows
TAG attention-driven GUI grounding
Grounding and evaluation code that aggregates a pretrained vision-language model’s attention maps to localize requested interface elements without fine-tuning.
Web
TongUI training and inference framework
Training configurations, preprocessing and inference examples for tutorial-trained GUI models, with a LLaMA-Factory fine-tuning workflow and screenshot-grounding interfaces.
Cross-platformDesktopWeb
VideoCAD training and evaluation code
VideoCADFormer training, action-log preprocessing and evaluation code for learning CAD interface action sequences from screen recordings of browser-based Onshape.
Web
WebMarker
A browser library that labels interactive elements and their boxes for visual prompting, preserving element mappings that automation code can use to execute selected actions.
Web
WebVLN research implementation
Training and evaluation code for question-guided website navigation that combines visual observations with web-specific HTML information.
Web
WinDOM code
GUI-grounding training and evaluation code with same-family distillation, GRPO recipes and point-in-box scoring, using a Windows-style web environment.
Web
Showing 361–380 of 380 resources.
Get full records and evidence through the topic export manifest. Source records can be passed to the standard RAG and citation exporters after cloning the catalog.