GroundGUI training and evaluation code
Code for screenshot grounding through supervised training, teacher-guided distillation, reinforcement learning, and benchmark evaluation.
桌面网页跨平台
GUI 定位
已审阅并标注「GUI 定位」的研究与资产。
条目依据已审阅标签归类，同一主题不表示资源之间存在依赖关系。日期保留来源精度，注释保留原语言。
GUI Grounding Evaluation Harness
ScreenSpot-Pro evaluation harness with explicit coordinate transforms, prompt and crop-based inference comparisons, and per-sample grounding accuracy and cost records.
GUI-RC / GUI-RCPO
GUI-grounding research code for spatial voting across sampled predictions and test-time policy optimization using region-consistency rewards.
跨平台桌面网页
GUIPivot
Query-inference training tools connecting screenshot grounding with GUI action reasoning, including preprocessing, query refinement and offline single-step evaluation on AndroidControl and AITZ.
Android
MAI-UI inference and evaluation code
Published MAI-UI grounding and navigation-agent implementations with inference demos and evaluation scripts.
Android
OculiX (SikuliX continuation)
A maintained continuation of SikuliX that locates on-screen targets with image matching and OCR, drives desktop or Android interfaces, and exposes scripting and optional MCP interfaces.
桌面WindowsmacOSLinuxAndroid
OmniParser
Screenshot parser that detects interactable regions and captions icons so vision-language models can ground actions, released as V1.5 and V2 weights alongside OmniTool for driving a Windows 11 VM.
跨平台
ScreenSuite
A shared evaluation harness for GUI-model perception, offline action prediction and multi-step interaction, using simple smolagents-based agents across existing benchmarks.
网页Android桌面
SeeAct
Generalist web agent framework for large multimodal models such as GPT-4V, with a Playwright tool that runs agents on live websites, text-choice and set-of-mark grounding, and a crawler mode.
网页
Seg2Click
Training and inference implementation that predicts an executable-region mask and derives a GUI click from that mask.
跨平台
Self Operating Computer
Framework, released in November 2023, in which a multimodal model views the screen and picks mouse and keyboard actions toward a goal, with backends such as GPT-4o, Claude 3, Gemini and LLaVA.
桌面
Set of Mark
Visual-prompting toolbox that labels image regions for multimodal reference and grounding, including GUI applications.
相关研究
Sikuli legacy research implementation
The original Sikuli source distribution combines screenshot targets with a Jython scripting library and IDE for visual GUI automation and testing.
桌面macOSLinuxWindows
TAG attention-driven GUI grounding
Grounding and evaluation code that aggregates a pretrained vision-language model’s attention maps to localize requested interface elements without fine-tuning.
网页
TongUI training and inference framework
Training configurations, preprocessing and inference examples for tutorial-trained GUI models, with a LLaMA-Factory fine-tuning workflow and screenshot-grounding interfaces.
跨平台桌面网页
VideoCAD training and evaluation code
VideoCADFormer training, action-log preprocessing and evaluation code for learning CAD interface action sequences from screen recordings of browser-based Onshape.
网页
WebMarker
A browser library that labels interactive elements and their boxes for visual prompting, preserving element mappings that automation code can use to execute selected actions.
网页
WebVLN research implementation
Training and evaluation code for question-guided website navigation that combines visual observations with web-specific HTML information.
网页
WinDOM code
GUI-grounding training and evaluation code with same-family distillation, GRPO recipes and point-in-box scoring, using a Windows-style web environment.
网页
第 361–380 条，共 380 条资源。
通过主题导出清单获取完整记录和证据。克隆目录后，可将主题记录直接交给标准 RAG 与引用导出工具。