VideoCAD: A Dataset and Model for Learning Long-Horizon 3D CAD UI Interactions from Video
Introduces synthetic CAD interaction videos, a model for learning interface actions from video, and evaluation tasks for spatial and long-horizon reasoning.
网页
数据生成
已审阅并标注「数据生成」的研究与资产。
条目依据已审阅标签归类，同一主题不表示资源之间存在依赖关系。日期保留来源精度，注释保留原语言。
ZeroGUI: Automating Online GUI Learning at Zero Human Cost
Combines generated tasks, estimated success rewards, and online reinforcement learning to automate GUI-agent interaction training.
跨平台
UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents
UI-Genie jointly improves a mobile GUI agent and a reward model that processes screenshots with interaction history and supplies action- and task-level feedback. Its data pipeline uses rule-based verification, corrupted trajectories, and hard negatives to support reward-guided exploration and iterative training.
BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism
BacktrackAgent adds verification, judgment, and reflection modules to detect erroneous GUI actions and guide recovery. It develops backtracking-oriented training data that includes post-action pages and evaluates task completion and step accuracy on Mobile3M and Auto-UI.
WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollback
WebCoT turns reflection, look-ahead, branching and rollback procedures into chain-of-thought training rationales. Fine-tuning on these reconstructed trajectories is evaluated for web-agent self-improvement and transfer across navigation tasks.
网页
ScreenExplorer: Training a Vision-Language Model for Diverse Exploration in Open GUI World
Trains ScreenExplorer with reinforcement learning in dynamic GUI environments, using a world-model curiosity reward and experience-stream distillation to support sustained exploration.
ProgRM: Build Better GUI Agents with Progress Rewards
ProgRM predicts per-step task completion progress to provide intermediate rewards during online GUI-agent training. It derives progress labels by using longest common subsequences to identify key trajectory steps, allowing useful partial progress to contribute even when a trajectory ultimately fails.
GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent
GUI-explorer generates exploration goals from screenshots and interface structure, then mines observation-action-outcome triples for app-specific transition knowledge. The resulting training-free agent uses this knowledge during mobile task execution, evaluated on SPA-Bench and AndroidWorld.
Android
Web-Shepherd: Advancing PRMs for Reinforcing Web Agents
Web-Shepherd is a process reward model for assessing individual steps of web-navigation trajectories. It introduces a collection of 40,000 step-level preference pairs with checklists and a separate benchmark for comparing reward models, then evaluates verification during WebArena-lite navigation.
网页
Efficient Agent Training for Computer Use
PC Agent-E augments 312 human computer-use trajectories with alternative action decisions synthesized by Claude 3.7 Sonnet. It studies training with this combination and introduces WindowsAgentArena-V2 as an updated evaluation benchmark.
Windows桌面
Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis
Introduces OSWorld-G grounding evaluation and the Jedi synthesis pipeline for examples spanning layout reasoning and fine-grained interaction.
桌面
Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents
Introduces MONDAY and an automated pipeline that extracts mobile-navigation training data from instructional videos across operating-system platforms.
跨平台
GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning
GUI-Shift trains an inverse-dynamics task that predicts the initial action connecting two GUI states, using trajectories without natural-language instructions. It combines rule-based reinforcement learning with data filtering and evaluates transfer to Android automation and screenshot grounding.
Android
MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learning
MobileIPL samples trees of action-planning thoughts, propagates rule-based leaf rewards backward, and forms preference pairs for reasoning-level optimization. It also expands screenshot-based instruction data through staged evolution before evaluation on mobile GUI-agent benchmarks.
ScaleTrack: Scaling and back-tracking Automated GUI Agents
ScaleTrack combines a unified collection of GUI grounding samples with a planning objective that predicts the next action while reconstructing prior actions. This backward prediction links the current screenshot to interaction history during agent training.
WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model
WebEvolver trains a co-evolving world model to predict web observations and generate additional agent-training trajectories. The same model supports look-ahead action selection at inference, with evaluation on live web-task benchmarks.
网页
PixelWeb: The First Web GUI Dataset with Pixel-Wise Labels
Introduces PixelWeb, a webpage GUI dataset with element images, contours, and pixel masks. Its annotation pipeline combines bitmap channel information with DOM-based visibility and stacking analysis to address occlusion and overlapping elements.
网页
TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents
TongUI converts multimodal online tutorials, including videos and articles, into GUI-agent trajectories. Its GUI-Net dataset contains 143,000 trajectories spanning five operating systems and more than 200 applications, and supports fine-tuning for grounding and navigation.
跨平台
UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis
UI-E2I-Synth uses GPT-4o to synthesize GUI grounding instructions with varied complexity, addressing small elements, imbalanced element types, and implicit requests. The accompanying UI-I2E-Bench evaluates instruction-to-element localization across these annotation challenges.
Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
Studies whether mid-training on GUI perception, multimodal reasoning, and text reasoning transfers to GUI-agent planning. Experiments compare 11 task sources and curated mixtures using WebArena and AndroidWorld to investigate training-data choices when GUI trajectories are scarce.
网页Android
Explorer: Robust Collection of Interactable GUI Elements
Explorer collects interactable GUI elements from a live target application and records user sessions for application-specific model training. It links repeated interface states into a navigation map that supports path planning and voice-directed interaction.
Android网页
GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration
Introduces GUI-Xplore exploration videos and downstream tasks, with an agent that combines action-aware modeling and graph-based environment reasoning.
跨平台
MP-GUI: Modality Perception with MLLMs for GUI Understanding
Uses specialized graphical, textual, and spatial perception components with a data-collection pipeline to improve GUI understanding.
跨平台
STEVE: A Step Verification Pipeline for Computer-use Agent Training
Labels individual computer-use actions by comparing surrounding screenshots, then trains agents from both successful and unsuccessful steps.
桌面Windows
DeskVision: Large Scale Desktop Region Captioning for Advanced GUI Agents
Introduces AutoCaptioner, DeskVision, and GUIExplorer for generating desktop region descriptions and training interface-understanding models.
桌面
Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks
Separates planning from execution and creates synthetic plan annotations for training agents on extended web-navigation tasks.
网页
SpiritSight Agent: Advanced GUI Agent with One Look
Introduces SpiritSight and GUI-Lasagne data, combining multilevel GUI annotations with visual block parsing for high-resolution interface navigation.
跨平台
ATLaS: Agent Tuning via Learning Critical Steps
Selects critical steps from expert interaction trajectories for targeted agent fine-tuning, studying efficiency and generalization across tasks.
网页相关研究
Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents
Synthesizes multimodal web trajectories through exploration and refinement, then trains a browser agent to study the effect of scaling interaction data.
网页
Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMs
Couples large- and small-model web-agent exploration with trajectory synthesis and distillation to improve both model types iteratively.
网页
AppVLM: A Lightweight Vision Language Model for Online App Control
AppVLM first learns app actions from AndroidControl and then refines its policy through data collected in AndroidWorld. The study compares offline action prediction and online task completion for a lightweight vision-language smartphone agent.
Android
MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users
Studies automatic annotation and federated training of mobile GUI agents from decentralized user interaction data with heterogeneous distributions.
ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation
Splits mobile tasks into reaching the required page and operating on it, using subtask data and preferred GUI trajectories for training.
AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs
Infers element functionality from UI changes before and after interaction to produce verified annotations for grounding-model training.
跨平台
GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
Collects environment-specific interaction data through autonomous exploration to adapt grounding models to previously unseen GUIs.
跨平台
UI-TARS: Pioneering Automated GUI Interaction with Native Agents
Studies a screenshot-driven native GUI model with unified actions, deliberate reasoning, and iterative training on interaction traces.
跨平台
Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training
Builds self-critique training examples from searched recovery paths to improve agents' ability to detect and correct errors during interaction.
网页相关研究
OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis
Builds GUI trajectories by exploring environments first and deriving tasks retrospectively, with reward-model filtering of generated trajectories.
跨平台
PC Agent: While You Sleep, AI Works -- A Cognitive Journey into Digital World
Combines PC Tracker interaction capture, completion of action and reasoning annotations, and planning and grounding agents for desktop workflows.
桌面
Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents
Combines autonomous task proposal, web interaction, and visual success evaluation to discover skills and improve browser-agent policies.
网页
Large Action Models: From Inception to Implementation
Uses a Windows-agent case study to describe a development workflow for large action models, covering data collection, training, grounding, environment integration, and evaluation.
桌面Windows
AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials
Converts public tutorials into task specifications and executes them with an agent to synthesize multimodal interaction trajectories.
网页
Falcon-UI: Understanding GUI Before Following User Instructions
Pretrains GUI-context understanding with instruction-free web interactions before fine-tuning Falcon-UI on instructed GUI tasks.
网页Android跨平台
Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
Uses screenshots, a unified action space, and staged grounding and reasoning training for cross-platform GUI interaction.
跨平台
DroidCall: A Dataset for LLM-powered Android Intent Invocation
Introduces data and a generation pipeline for translating natural-language requests into Android intent invocations using compact language models.
Android相关研究
AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations
Adapts multimodal web agents to unfamiliar websites using a small number of human demonstrations through prompting or meta-learning.
网页
AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents
Provides a shared Android action environment, reproducible tasks, and instruction data for training and comparing language and multimodal agents.
Android
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
Combines cross-platform GUI grounding data synthesis with action-model training to study generalization to unfamiliar interfaces.
跨平台
OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization
Develops multimodal browser agents through repeated cycles of real-web exploration, trajectory feedback, and policy optimization.
网页
EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data
Synthesizes GUI training examples at multiple levels of detail from webpages and studies transfer of learned understanding to other interface environments.
网页跨平台
Harnessing Webpage UIs for Text-Rich Visual Understanding
Builds MultiUI by generating instructions from webpage accessibility trees and pairing them with screenshots for multimodal training. Evaluation examines web-interface understanding, including VisualWebBench and Mind2Web, and transfer to other text-rich visual tasks.
网页
Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
Introduces UGround, a visual target-localization model trained with large-scale synthetic references to elements in GUI screenshots.
跨平台
NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wild
Generates browser-agent demonstrations through autonomous exploration and retrospective task labeling, pruning trajectories that lack meaningful subtasks.
网页
DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation
Generates labeled slides and interface visuals through code to support visual element detection, description, and classification with less manual annotation.
跨平台
Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale
Synatra converts indirect procedural knowledge, including human-oriented tutorials, into direct demonstrations for digital agents. It trains a 7B CodeLlama on 100,000 synthesized demonstrations and evaluates the resulting policy on Mind2Web, MiniWoB++, and WebArena.
网页
MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding
MobileVLM adds two pretraining stages and four UI tasks to learn element detail and relationships between successive screens. Its Mobile3M dataset contains three million Chinese mobile UI pages connected by recorded transition actions.
MobileViews: A Million-scale and Diverse Mobile GUI Dataset
Builds mobile screenshot and view-hierarchy pairs through automated application traversal and evaluates their usefulness for GUI grounding.
Android
VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents
Benchmarks visual agents across GUI, embodied, and design tasks, with training trajectories collected through solvers, agents, and human demonstrations.
跨平台相关研究
GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices
Introduces annotated cross-application mobile trajectories and studies history-aware navigation over long task sequences.
Android
Tell Me What's Next: Textual Foresight for Generic UI Representations
Textual Foresight learns UI representations by describing the future screen expected after an action on the current interface. The OpenApp dataset supports pretraining and downstream evaluation of representations that connect individual elements with whole-screen changes.
第 181–240 条，共 347 条资源。
通过主题导出清单获取完整记录和证据。克隆目录后，可将主题记录直接交给标准 RAG 与引用导出工具。