Human in the loop

OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agents

Pengzhou Cheng, Zheng Wu et al.

Uses step-level confidence estimates to decide when a GUI agent should act autonomously or request human intervention in complex situations.

Towards Computer-Using Personal Agents

Piero A. Bonatti, John Domingue et al.

Proposes computer-using personal agents connected to user-controlled data repositories, discussing privacy, interoperability, and collaborative tasks.

CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation

Faria Huq, Zora Zhiruo Wang et al.

Introduces CowPilot, a web-navigation framework where agents suggest actions and users can pause, reject, override, or resume them. It evaluates autonomous and collaborative use through task success and human effort.

Interactive Speculative Planning: Enhance Agent Efficiency through Co-design of System and User Interface

Wenyue Hua, Mengting Wan et al.

Studies agent-system and interface co-design that incorporates user interruptions and intermediate feedback to reduce planning latency.

IDA: Breaking Barriers in No-code UI Automation Through Large Language Models and Human-Centric Design

Segev Shlomov, Avi Yaeli et al.

IDA helps business users create web UI automation through guided demonstrations, a semantic programming model, and a teaching-oriented interface. A prototype user study examines automation creation with enterprise applications by users without programming backgrounds.

Unveiling Disparities in Web Task Handling Between Human and Web Agent

Kihoon Son, Jinhyeon Kwon et al.

Compares human think-aloud web-task behavior with web-agent planning, action, and reflection. The analysis highlights differences in updating plans, handling ambiguity, and investigating failures.

Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts

Tian Huang, Chun Yu et al.

Prompt2Task converts textual goals or procedures into smartphone UI automation using agents for intent interpretation, external-information handling, and execution. User feedback and accumulated task knowledge support subsequent task generation and operation.

GPTVoiceTasker: Advancing Multi-step Mobile Task Efficiency Through Dynamic Interface Exploration and Learning

Minh Duc Vu, Han Wang et al.

Introduces GPTVoiceTasker, a mobile assistant that interprets user commands, executes device interactions, and learns from prior commands to streamline later task execution. The work includes a user study of task efficiency.

MobileAgent: enhancing mobile control via human-machine interaction and SOP integration

Tinghe Ding

Combines operating-procedure context with user interaction to guide mobile task automation and handle personalized or sensitive requests.

MUG: Interactive Multimodal Grounding on User Interfaces

Tao Li, Gang Li et al.

Studies iterative UI grounding in which users refine or correct an agent's interpretation over multiple conversational turns.

META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI

Liangtai Sun, Xingyu Chen et al.

Introduces GUI-based task-oriented dialogue and a dataset pairing mobile-interface operations with conversational responses.

A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility

Andrea Burns, Deniz Arsan et al.

Introduces MoTIF mobile-app interactions with feasible and infeasible commands, feasibility explanations, and follow-up questions for ambiguous requests.

Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations

Toby Jia-Jun Li, Marissa Radensky et al.

Introduces PUMICE, an agent that learns mobile-app procedures and conditional concepts through natural-language conversation and GUI demonstrations, recursively resolving ambiguous instructions with users.

Kite: Building Conversational Bots from Mobile Apps

Toby Jia-Jun Li, Oriana Riva

Derives task graphs from demonstrated Android app interactions and layers a conversational question-answer interface over those graphs. The evaluation examines task-model extraction and developer use of the resulting bot templates.

APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions

Toby Jia-Jun Li, Igor Labutov et al.

Introduces APPINITE, a conversational interface for specifying the intended targets of demonstrated Android actions. It constructs semantic UI representations and uses an on-screen overlay to preview and refine executable data descriptions.

SUGILITE: Creating Multimodal Smartphone Automation by Demonstration

Toby Jia-Jun Li, Amos Azaria et al.

Introduces SUGILITE, which combines spoken instructions, demonstrated Android actions, and UI hierarchies to generate reusable automation scripts. Its interface includes context checking and repairs for unfamiliar situations and changing app interfaces.

PLOW: A Collaborative Task Learning Agent

James Allen, Nathanael Chambers et al.

Describes PLOW, which learns executable web-task models from a session combining demonstrations, natural-language explanations, and dialogue. Browser instrumentation connects user actions with DOM structure to support task generalization.

Eager: Programming Repetitive Tasks by Example

Allen Cypher

Describes Eager, which detects repeated HyperCard interactions, generalizes them into loops, and previews predicted actions by highlighting interface objects. Users can then ask it to execute the inferred repetitive task.

MobileWorld (Tongyi-MAI)

A containerized Android evaluation environment with GUI-only, agent-user interaction and MCP-augmented tasks, plus checks on application and backend state.

COLA demo

Research demonstration of collaborative Windows GUI automation with dynamic task scheduling and recovery mechanisms.

MagenticLite / Magentic UI

Microsoft agent application combining orchestration with browser interaction; successor to Magentic UI.

META-GUI baseline

Training and evaluation pipeline for conversational mobile GUI agents, with action prediction, response generation, and dataset processing.

MobileAgent (Alipay)

Mobile-control research implementation combining operating-procedure context with human interaction for personalized task execution.

Nanobrowser

A browser-extension agent that reads pages, interacts with elements and moves between tabs from a side-panel task interface.

OS-Kairos

Android GUI-agent research implementation that uses confidence-guided interaction to choose between autonomous actions and human assistance.

VeriOS

Research code for proactive human–GUI-agent verification, including offline interaction-loop and dual-agent evaluation scripts plus LLaMA-Factory fine-tuning configurations.

Yunkai

Android phone assistant combining model-driven tool use with accessibility-tree reading, screenshots, and tap, swipe and text-entry primitives. Its documented interaction flow presents operation plans for user confirmation.