Computer-Use Agents: Methods, Evaluation, Human Agency, and Research Assets

Scope, historical foundations, and relation to prior surveys

Computer use as interaction with software

Computer-use agents connect a user's goal to changes in software. Their distinguishing problem is not simply producing an answer about an interface: they must select and execute actions, observe what happened, and continue toward the goal. We use computer-use agent for a system that carries out tasks in a software environment through an observation–action loop. Within that broader category, this survey centers on graphical user interfaces (GUIs) on web, desktop, and mobile platforms, together with the representations and infrastructure that make those interactions possible. This operational definition follows the agent–environment perspective developed by Sager et al. and the environment, observation, and action distinctions used in the OS-agent survey. It does not require a particular model architecture or unrestricted autonomy. [1, 2]

An interface is more than its pixels. A browser agent may observe a screenshot, the document object model (DOM), an accessibility tree, or a combination of these. A desktop agent may also need information about open windows and files; a mobile agent must translate decisions into touches, gestures, and navigation operations. Consequently, a screenshot-only definition would exclude substantial interface research. Conversely, generic tool calling is not sufficient evidence of computer use within our scope. A method belongs in the central discussion when it performs software-interface actions or provides a concrete component for understanding, grounding, or evaluating those actions. Hybrid systems that combine interface operations with code or application programming interfaces (APIs) are relevant when that relationship is documented. Zhang et al. explicitly discuss this combination, rather than treating GUI and API execution as mutually exclusive designs. [3, 2]

This boundary also separates an enabling capability from a complete agent. Recognizing a button, describing a screen, predicting a demonstrated next action, and accomplishing a task in a changing application are related but distinct achievements. Pix2Struct, for example, evaluates interface understanding through referring-expression selection and screen or widget captioning; those experiments support its relevance to perception, without establishing autonomous task completion. A paper about physical navigation, a generic language-only planner, or a document question-answering model is not brought into the core merely because its techniques might transfer. The same distinction applies to autonomy: asking for clarification, yielding control, or requiring confirmation can be part of a useful agent's behavior. [4, 3]

Several historical threads, rather than one succession

The motivation to automate interfaces predates foundation models. Koala, published in 2007, recorded browser interactions as editable, approximately natural-language instructions and shared them through a wiki. It combined demonstration, contextual interpretation of page elements, and personalization through user data. Its worked example also included steps left to the user and mechanisms for correcting an interpretation. Sikuli's 2009 institutional abstract describes a complementary approach: screenshots could identify interface elements for search and serve as patterns in scripts that direct mouse and keyboard events. These works established different ways to connect a user's recognizable interface to executable behavior. They should not be retrospectively described as general-purpose language-model agents. [5, 6]

Mobile research developed both task automation and reusable interface corpora. SUGILITE combined verbal commands, user demonstrations, and Android accessibility hierarchies to generalize recorded procedures, including procedures spanning applications. The paper describes context checks and error handling, but also limitations of its string-based generalization and accessibility-dependent implementation. Rico addressed a different need: it combined human and automated exploration of Android applications to collect screenshots, view hierarchies, interaction traces, and interface representations for data-driven design applications. Rico is therefore an important interface-data resource, not evidence that an autonomous agent had mastered those applications. Keeping the original paper, its collected data, and later distributions distinct avoids transferring claims between different releases. [7, 8]

A parallel thread made web interaction an environment for learning. World of Bits exposed rendered pixels and DOM information, accepted keyboard and mouse operations, and addressed the difficulty of obtaining rewards while reproducing changing websites. Its settings included small controlled tasks in MiniWoB and approximations of websites reconstructed from recorded HTTP traffic. Workflow-Guided Exploration subsequently used demonstrations to constrain exploration and trained a separate policy from successful experience. That work also introduced MiniWoB++ tasks involving additional language, layout, and horizon challenges. MiniWoB and MiniWoB++ are related contributions, but the later extension should not be silently attributed to the original platform paper. [9, 10]

Interface representation and behavioral data remained distinct routes toward broader capability. Pix2Struct pretrained an image-to-text model to recover simplified HTML from masked webpage screenshots, then adapted it to visually situated language tasks, including mobile-interface understanding. Android in the Wild collected natural-language goals and demonstrations containing screenshots, typed text, navigation actions, and coordinate-based gestures across Android versions and device configurations. The former supplies a representation-learning approach; the latter supplies demonstrated behavior and generalization settings. Neither resource alone establishes how reliably an agent will recover after its own erroneous action. Android in the Wild explicitly distinguishes its reproducible offline action-matching evaluation from more costly online validation. [4, 11]

Language-conditioned web tasks broadened the relationship between instructions and software state. WebShop introduced a simulated shopping environment built from real product information, with search and selection actions and a reward tied to the requested product properties. Its reported model experiments used a simplified observation mode; its preliminary transfer experiment permitted real-site navigation while retaining consequential form actions inside the evaluation interface. Mind2Web instead recorded tasks and interaction traces from real websites, including webpage snapshots, to study generalization across tasks, websites, and domains. Its authors acknowledge that evaluation against cached actions can reject valid alternative paths. These are different experimental contracts, even when both are described as web-agent research. [12, 13]

Interactive environments then made task outcomes a more direct object of study. WebArena supplied standalone, functional websites and task-specific checks, allowing agents to produce states beyond a single recorded trajectory. OSWorld extended this emphasis to real desktop applications, files, and workflows involving multiple applications within controlled operating-system environments. AndroidWorld constructed parameterized tasks in an Android emulator, with initialization, success checking, and cleanup tied to application or system state. These environments support different forms of repeatable interaction; they do not make every deployment condition identical. WebArena deliberately controls its websites, OSWorld's benchmark and infrastructure have distinct platform coverage, and AndroidWorld fixes operating-system and application versions for reproducibility. [14, 15, 16]

The history thus contains overlapping lines of work: end-user automation, interface representation, demonstration collection, policy learning, and executable evaluation. Foundation models connect these lines through flexible language and visual processing, but the earlier questions remain recognizable: what does an instruction mean in the current interface, which information is observable, when should a procedure adapt, and what counts as completion? Shared scripts and demonstrations remain useful even when an agent can plan; structured interface access remains useful even when it can see. This continuity motivates a survey organized around relationships among capabilities and evidence, rather than a chronology that assumes each new method replaces its predecessors. [5, 10, 1]

What existing surveys already organize

Recent surveys provide several useful organizing perspectives. Broad GUI surveys cover data, architectures, training, evaluation, and applications; platform-focused reviews explain the constraints of browser or phone interaction; specialized reviews examine trust and efficiency. Table 1 compares the documented emphasis of eight surveys whose relevant full-text sections were inspected for this discussion. The entries describe their organizing choices, not exclusive coverage boundaries. A topic not named in a cell should not be interpreted as absent from that survey. The versions matter: several first appeared in 2024 or 2025 and were subsequently revised, so the comparison refers to the identified revision rather than assuming a fixed publication-year snapshot.

These perspectives overlap productively. Data and benchmarks are already central to broad GUI reviews, while operational concerns also appear in platform surveys. Trust and efficiency are not independent afterthoughts: verification can introduce runtime cost, and a successful sequence may still have unacceptable side effects. The trustworthy-GUI survey examines this through the relationship between perception, reasoning, and interaction; the efficiency survey follows resource use across the same loop. We therefore do not present the inclusion of assets, safety, or systems concerns as an unprecedented feature. Our emphasis is to make the connection from a research statement to its supporting artifact and evaluation setting explicit. [17, 20, 21]

That emphasis changes the questions asked of a contribution. A released repository may contain an agent, a dataset, an evaluator, or several components with different versions and permissions. A dataset paper and a later distribution can describe the same research lineage without identifying the same bytes. An environment may support multiple operating systems while a particular experiment covers only one. Accordingly, the later sections distinguish paper-level contributions from artifact-level properties and attach claims to the relevant setting. This is a complementary reading of the literature, intended to help readers identify what can be learned, tested, or reused from a work without inferring guarantees from its name or its presence in a catalog.

Boundary of this contextual reading

This section is an evidence-traced narrative synthesis, not an exhaustive or systematic review. All 15 survey records in the frozen inventory received primary-page metadata and abstract checks; eight were selected for deeper comparison. Relevant sections of those eight surveys and 12 foundational papers were read in retained primary HTML or PDF. Sikuli's narrower historical description uses its complete institutional abstract and author project page; the linked full paper was unavailable during retrieval. These reading statuses are recorded separately from catalog inclusion. The inventory is a discovery aid, not a claim that every listed paper's full text or every reported experiment has been verified. The search and evidence protocol elsewhere in this article describes the broader retrieval process and its limits.

Review design and evidence base

This article is a structured narrative survey with a reproducible search trace and a deliberately selected primary-reading set. Its organizing questions are: RQ1, how do observation and action interfaces constrain agent design; RQ2, what do demonstrations, synthetic data, and reinforcement learning teach; RQ3, which capabilities do current evaluations actually measure; and RQ4, how can research assets remain useful to both people and software agents? The aim is to connect methods, measurement, and reuse. It is not to estimate a field-wide effect size or to identify a single best system.

A frozen inventory and a supplementary search

The starting inventory is the AwesomeCUA v2026.10.18 data release. It contains 3,048 active resource identities, of which 2,599 are papers or surveys. Of these bibliographic records, 1,123 have catalog editorial review and 1,476 retain a metadata-only state. The remaining 449 active resources describe implementations, models, datasets, benchmarks, tools, infrastructure, products, or publishers. A paper and its implementation are different resources. The 1,572 reviewed-resource total therefore cannot be interpreted as a number of papers. Editorial review establishes the scope and source of a catalog annotation; it does not imply full-paper review or experimental reproduction. [22]

The supplementary query broadened the catalog's existing terminology with mobile-agent, UI-agent, desktop-agent, and screen-grounding phrases. It was executed on 18 October 2026, sorted by first submission, with 100 entries per page. The service reported 1,452 matches, and the retrieved union contained exactly 1,452 unique identifiers. This establishes pagination coverage for that query. It does not establish recall across arXiv, other publishers, or the whole field. In particular, phrase matching returned unrelated work on quantum computation and physical or network-mobile agents. The exact expression, timestamps, response hashes, returned identifiers, and deduplication outcome are supplied in the search ledger.

All 303 titles absent from the active inventory were inspected for this supplementary selection. Twenty-six ambiguous or potentially relevant abstracts received a complete abstract read; 15 concerned interface agents, their evaluation, or an adjacent interface-testing application. Eleven were not selected because the abstract described a different domain or did not establish material software-interface interaction. These are manuscript selection decisions, not new production-catalog exclusions. The other 277 results were not advanced beyond title screening. Additional targeted searches and reference following supplied historical, platform-specific, and method-specific primary sources. Their query and reading records are retained separately, rather than being merged into an artificial single screening denominator.

Selection and extraction

Representative primary reading was selected for coverage of interface families, observation modalities, action abstractions, training procedures, evaluation settings, and documented failure modes. Earlier work was retained when it introduced a still-relevant representation or evaluation contract. Newer work was used to examine developments such as runtime optimization, online learning, proactive assistance, and typed action selection. Selection did not use citation counts, repository stars, or reported leaderboard position as a quality threshold. The prior-survey comparison inspected all 15 survey abstracts and selected eight for relevant full-text sections. Bibliographic identity, research contribution, and artifact availability were extracted as separate fields.

For each source used in a substantive comparison, the reading ledger identifies the primary URL, retrieved or retained version, response fingerprint, sections or pages actually read, supported claims, and qualifications. A full-text section read is not a claim that every page, appendix, or cited experiment was audited. An abstract-only record is labeled accordingly. The original source remains the authority for its reported results; this article contributes a synthesis and a descriptive inventory analysis, without rerunning those experiments. AI assistance was used for retrieval, extraction, drafting, and cross-checking. The parallel readers are AI agents, not independent human reviewers; no human inter-rater agreement or peer-review status is claimed.

Version handling is part of the method. A repository's current default branch may differ from the implementation evaluated in a paper. A revised preprint may change the task set, protocol, or conclusions. A proceedings edition may have a different author list or title from its retained preprint. Citations therefore identify the source edition actually used, and extraction notes preserve distinctions between paper versions, software releases, and dataset distributions. When a full text was unavailable, the claim was narrowed to what the accessible primary source supported. No publication venue, date precision, model version, or missing experimental condition was supplied by inference.

Methods: from interface grounding to learning through interaction

The central methodological question is how an agent's observation, action interface, training data, and learning objective determine what transfers to a new task. Generalization has several meanings here: locating unfamiliar controls, composing familiar operations into a new workflow, recovering from an unexpected state, or moving between operating systems. A method can improve one without establishing the others. We therefore compare systems by the information and control they receive, the experience they reuse, and the feedback that changes their behavior. This separates architectural mechanisms from reported benchmark outcomes, which often use different histories, tools, evaluators, and interaction budgets. [13, 23, 24]

Observation and grounding define the control problem

Structured interface representations make some decisions easier by exposing labels, hierarchy, and actionable objects directly. Mind2Web's baseline first ranks DOM elements, then selects an element and operation from reduced candidate groups. This makes large pages manageable, but the pruning stage creates a recall boundary: downstream reasoning cannot select a target that was removed. SeeAct separates screenshot-based action description from grounding that description to an executable element. Its experiments distinguish textual choices, visual marks, and human oracle grounding. That separation is diagnostically useful: knowing what should happen and identifying where to act are different capabilities, and oracle grounding does not demonstrate autonomous execution. [13, 25]

The label “visual agent” does not uniquely specify the observation contract. WebVoyager uses screenshots annotated with interactive elements obtained through JavaScript, together with auxiliary element text. AppAgent's Android implementation uses XML-derived identifiers and numbered screenshot overlays. Mobile-Agent instead combines optical character recognition (OCR), icon detection, and visual matching to localize actions without XML. These choices exchange structural coverage for perceptual estimation: a missing accessibility node and a misrecognized icon fail differently. Mobile-Agent's experiments are on Android even though the interface motivates broader portability. Cross-platform intent should therefore be distinguished from platforms actually evaluated. [26, 27, 28]

Coordinate-based grounding models predict locations directly, but they still make representation choices. SeeClick trains coordinate-text prediction using GUI grounding and inverse OCR tasks. CogAgent supplements a conventional visual path with a higher-resolution cross-attention module, addressing small text and controls without placing all high-resolution features in one self-attention sequence. UGround separates a screenshot planner from a specialized visual grounder and adapts image resolution for interface localization. These approaches make coordinate prediction trainable, while retaining a practical trade-off between spatial detail and processing cost. Better localization alone does not resolve an incorrect instruction from the planner; Aria-UI explicitly identifies that dependency as a limitation. [29, 30, 23, 31]

Visual deployment also should not be confused with structure-free training. CogAgent, SeeClick, and UGround use structured web information to construct visual supervision. OS-ATLAS extends collection through platform-specific accessibility APIs and exploration, then separates grounding pretraining from action fine-tuning. Thus, structure can serve as a training instrument even when it is absent at inference. The transfer question becomes whether the resulting visual representations cover unfamiliar controls and referring expressions, rather than whether HTML was ever used. Conversely, a shared action schema is an interface convention: standardizing click, tap, and scroll labels does not make their platform-specific effects identical. [30, 29, 23, 32]

ShowUI treats visual efficiency as part of grounding. It groups neighboring patches with similar pixel values and skips selected redundant tokens while preserving the positions of retained tokens. It also interleaves screenshots and actions for navigation, and packs multiple grounding queries against one screenshot during training. Aria-UI adds task and action history to grounding examples, including interleaved visual context. Together, these designs show why neither resolution nor context length is a sufficient description of perception: the agent must retain the small spatial cues and earlier states that matter to the current decision. Compression quality should be evaluated against that decision, not only token savings. [33, 31]

Modular control, native policies, and hybrid execution

Modular systems allocate different responsibilities to different components. UFO's HostAgent selects applications and coordinates work, while an AppAgent operates application controls through Windows automation. Agent S separates a manager's task decomposition from workers' grounded execution and reflection. Its interface augments an accessibility tree with OCR and restricts execution to discrete actions with feedback after each step. These decompositions provide explicit places to diagnose planning, grounding, and execution failures. They also introduce dependencies: UFO cannot operate controls unsupported by its backend, and Agent S's original interface requires more than pixels. Hierarchy organizes those dependencies; it does not eliminate them. [34, 35]

Native policies move more of this coordination into a trained model. Aguvis trains grounding before planning and reasoning, representing actions through PyAutoGUI-style commands plus platform-specific extensions. UI-TARS predicts reasoning and actions from screenshots and interaction history, with shared and device-specific actions and an explicit request for user intervention. This reduces the need to hand-specify every intermediate module, but execution still depends on an external action interpreter and environment. A unified output format also cannot guarantee semantic compatibility: typing into a focused field, selecting an object, and terminating a task require different state conditions even when their commands share one syntax. [36, 24]

The practical boundary is increasingly hybrid. Agent S2 routes actions to visual, OCR-based text-span, or structural specialists; its spreadsheet specialist can update cells programmatically. UFO2 combines Windows UI Automation and visual detections, then prefers registered native application APIs with GUI fallback. UI-TARS-2 extends GUI control with terminal and other tool operations in shared stateful environments. These are substantive changes to the available action space, not interchangeable implementations of a pointer-only agent. They can shorten workflows by expressing structured operations directly, while introducing application bindings, tool permissions, and backend-specific failure modes. Comparisons should state which routes were available and which actually executed. [37, 38, 39]

Jev-Mobile examines a different division of labor: a vision-language model (VLM) sets a local goal and exact input text, while a typed decision service chooses from actions enumerated from the current Android accessibility tree. Candidates are rebuilt after each action, so delegated execution retains feedback rather than replaying an unobserved batch. Typed output constrains the available choices but does not establish that a chosen action is correct. Tree-missing visual targets remain inaccessible, and “local” denotes decision scope rather than on-device inference. Its comparison evaluates complete systems with different executors; it does not isolate a causal advantage of the decision model over a small VLM in the same controller. [40]

Planning, memory, verification, and recovery

A history is not yet a useful state representation. AppAgent turns exploration or demonstrations into documents about application behavior. Agent Workflow Memory extracts reusable subroutines and replaces example-specific values with variables; its offline and online variants differ in when those workflows are acquired. Agent S retrieves high-level narrative experience for planning and more concrete episodic experience for workers. These mechanisms reuse experience without necessarily updating model weights. Their evaluation must specify what memory exists initially, whether it grows across test tasks, and how experiences are admitted. In the online variants, model-based success judgments can propagate mistakes into later retrieval. [27, 41, 35]

Memory can also be learned as a control policy. MementoGUI trains operators for selecting salient events, compressing working memory, writing episodes, and filtering retrieval results. It preserves selected image regions alongside textual summaries and augments a frozen GUI action model. This distinguishes learning what to remember from fine-tuning how to act. However, its reported evaluations use offline GUI benchmarks and model-judged action or memory measures; they should not be read as proof of live task completion. The broader design issue is evidence preservation: compression can discard a transient selection or dialog state that is unavailable in the current screenshot. [42]

Structured planning introduces stronger internal constraints, with correspondingly stronger modeling assumptions. Agent+P builds a UI transition graph using static program analysis and dynamic refinement, translates navigation into a symbolic planning problem, and sends the plan to a UI executor. Its formal guarantees apply to the represented graph and chosen target, not automatically to the real application or the interpretation of user intent. ActionEngine instead learns application states and reusable operations through offline exploration, then compiles a generated program into browser actions and data operations. Both shift work from repeated local deliberation to reusable application knowledge; incomplete graphs, changed controls, and mistaken state identification remain operational concerns. [43, 44]

Verification operates at several levels that should remain separate. Mobile-Agent reflects on unchanged or unexpected screens. Agent S can return failure to its manager, and Agent S2 revises the remaining plan after subgoals. Such checks can guide recovery without independently proving that the user's goal was achieved. A visible draft is not necessarily a saved document, and a model's completion token is not an external state check. Likewise, success judgments used to filter training trajectories need their own validation. The methodological distinction is between detecting a suspicious transition, choosing a corrective action, and establishing the final postcondition. [28, 35, 37, 45]

Recovery becomes more concrete when training includes states reached after mistakes. UI-TARS annotates both a corrected action at the error point and a subsequent correction after the wrong action has already changed the environment. OpenCUA allows imperfect human demonstrations, uses a reflector to identify incorrect or redundant steps, and constructs reasoning around state transitions while excluding unsuitable action targets from training. These recipes address a gap in clean demonstrations: an agent deployed on its own predictions visits states that an expert path may never contain. Reflection text is useful supervision, but its presence does not establish that the generated explanation faithfully causes the action. [24, 46]

What the data teaches, and what collection leaves out

Grounding data and trajectory data supervise different problems. Element descriptions paired with coordinates teach localization; a sequence conditioned on a user goal teaches when and why an action is useful. OS-ATLAS and Aguvis make this distinction explicit through staged training. The referring-expression distribution matters too: functional descriptions, spatial relationships, and visual appearance ask different questions of the same control. UGround and Aria-UI synthesize such variation instead of relying only on literal element text. Scaling annotation count can therefore leave important gaps if most examples repeat easy labels, shallow pages, or one interaction style. [32, 36, 23, 31]

Human demonstration pipelines must reconstruct decision-time information carefully. OpenCUA records screen video and input signals, compresses low-level events into meaningful actions, and aligns actions to prior screen states. For clicks, it searches before cursor movement to reduce leakage from a pointer already indicating the target. This illustrates a general issue in offline training: the recorded observation must contain information available before the decision. Action reduction also chooses the temporal unit of learning. A merged scroll or text insertion is easier to model than thousands of device events, but exposes different opportunities for intermediate feedback. [46]

Synthetic trajectories differ mainly in where their goals come from. AgentTrek filters and standardizes tutorials, then uses a browser agent to replay their instructions and a model to judge the resulting trajectories. Explorer refines a task while navigating, allowing the eventual instruction to reflect what is actually available. OS-Genesis starts from observed action transitions, infers plausible low- and high-level tasks, and subsequently executes those tasks. These approaches reduce dependence on manually recorded demonstrations, but select different experience distributions: published procedures, goals consistent with an exploration path, or functions discovered through traversal. None establishes exhaustive coverage of realistic user intentions. [45, 47, 48]

Filtering determines what the model is allowed to learn from. Explorer halts collection at login, CAPTCHA, and payment boundaries, which limits the resulting workflows. AgentTrek's replay uses structured browser observations even when the resulting data later trains a visual policy. OS-Genesis uses graded trajectory scores to sample some incomplete but informative executions rather than keeping only binary successes. These choices should be reported alongside dataset size. Our synthesis is that diversity has at least three dimensions: interface coverage, goal coverage, and state coverage after errors. A large corpus can be broad in one dimension while remaining narrow in the others. [47, 45, 48]

Generated reasoning adds another supervisory layer. Aguvis constructs inner monologue using the goal, observation, history, and known action. UI-TARS explicitly recognizes the risk of explanations that merely fit a supplied answer and also samples thought–action pairs before selecting those that match the target. OpenCUA adds reflection, memory, and planning information through a multi-stage synthesis pipeline. These are methods for constructing useful training targets, not direct recordings of human reasoning or demonstrations of causal faithfulness. Their value should be assessed through controlled ablations and behavior on unfamiliar states, with the generator and filtering policy treated as part of the data recipe. [36, 24, 46]

Objectives determine which errors receive feedback

Supervised fine-tuning, preference optimization, and reinforcement learning should not be grouped solely by whether a paper uses the word “reasoning.” Supervised learning increases the likelihood of demonstrated outputs. UI-TARS's preference stage additionally contrasts corrected and erroneous actions. GUI-R1 applies group-relative optimization to candidate responses using format, action-type, point-in-box, and input-text rewards. These rewards verify an annotated local prediction; they do not by themselves measure whether a freely executed multi-step task succeeds. Local action accuracy can improve while delayed consequences and recovery remain outside the training signal. [24, 49]

Reward granularity determines which mistakes the learner can distinguish. A point inside the annotated box establishes a localization match, but does not check whether the application accepted the operation or retained the intended change. Conversely, a final success label does not identify which intermediate actions were necessary. In successful-rollout filtering, actions along a retained path receive imitation supervision even when some were redundant. Preference pairs can target a known mistake more precisely, but their coverage is limited to the states and alternatives collected. These are complementary sources of information rather than a universal ordering of objectives. A transfer claim is stronger when evaluation tests both unfamiliar successful paths and the error states that the chosen supervision leaves weakly constrained. [49, 50, 24]

Environment-interacting RL exposes those consequences but must solve reward and credit-assignment problems. WebRL generates curriculum tasks from failed attempts, evaluates outcomes with a learned reward model, and constrains policy updates while retaining replay experience. Its policy observes HTML and history, and its outcome model uses the final page plus actions. UI-TARS-2 trains multi-turn policies with stateful rollouts, using different reward mechanisms for different domains: executable checks, answer judges, or generative outcome scoring. A result described as “verifiable” must therefore specify what is checked and by whom. A learned judge is a useful source of supervision, not an automatic guarantee of goal correctness. [51, 39]

The interaction horizon changes both learning difficulty and the behaviors an agent can express. Test-Time Interaction (TTI) studies a curriculum that progressively permits longer rollouts, using successful trajectories for updates. Short budgets favor tasks solved quickly; longer budgets permit exploration and recovery but make outcome credit harder to assign. Its prompting experiments also show that repeated rechecking can replace a correct answer with an incorrect one. The useful question is consequently not whether longer thinking or more actions always wins, but which additional computation supplies relevant evidence under a fixed task, observation contract, and budget. [50]

Systems choices can change the effective learning objective. AsyncWebRL overlaps rollout, training, and policy refresh through persistent workers and lightweight screenshot references. It separately corrects for stale rollout policies and studies how per-trajectory length normalization underweights tokens in long failed episodes. Replacing that normalization changes the relative influence of those failures. The reported diagnosis is tied to its WebGym setup, memory prompt, and model variants; it should not be generalized into a universal claim about all policy optimization. It nevertheless shows why efficiency analysis must inspect both wall-clock scheduling and the weighting of experience consumed by the learner. [52]

Interaction cost and the execution environment

Fewer model calls can arise through different mechanisms. ActionEngine amortizes exploration into reusable operations and compiles a task program; its design still allows state filtering, repair, and explicit semantic helper calls. Qwen-UI-Agent can emit ordered action batches and combine GUI operations with CLI or service calls. Batching reduces observation and inference boundaries, whereas compilation reuses application knowledge. Both require care when later actions depend on an intermediate result. Reporting only action counts can obscure that difference: one model turn may contain several operations, and a programmatic action may perform work that would require many pointer events. [44, 53]

Branching adds another axis of computation: trying alternatives from the same workspace state. TClone provides local workspace versions using containers, copy-on-write sharing, and asynchronous checkpointing, including GUI state rather than only files. This supports rollback and speculative branches with an explicit commit boundary. Its network design also makes the limit clear: external connections must be re-established, and effects on remote services generally cannot be undone by restoring local state. Thus, cheaper search is not equivalent to harmless exploration. Any use of branching for computer control must distinguish reversible local operations from actions that affect other systems. [54]

Mobile execution brings reset and failure accounting into the training design. Darwin Mobile Agent uses asynchronous cloud Android instances and a setup–execution–teardown lifecycle; minimal cleanup preserves some state between episodes, while infrastructure failures are distinguished from task termination. Its implemented system uses manually selected tasks, an external judge, and a configured history window. Internally learned curricula, verification, and memory remain roadmap goals. Qwen-UI-Agent similarly describes device-health scheduling and separate review of model and environment failures. Treating an unavailable service as an agent error, or silently removing such episodes, would change the learning and evaluation distribution. [55, 53]

Application knowledge and adaptation in recent mobile systems

Recent mobile work asks when an agent should acquire application knowledge rather than repeatedly improvise. Curiosity Driven Knowledge Retrieval compares predicted and observed interface descriptions, triggering retrieval into structured AppCards when its uncertainty signal rises. The cards summarize functions, parameters, interface mappings, and interaction patterns from documentation, code, and earlier trajectories. X-OmniClaw takes a complementary route: it combines Android XML, OCR, visual grounding, and multimodal context, then captures user behavior as reusable skill cards and activity or deeplink entry routes. Its “behavior cloning” describes skill capture at runtime, not demonstrated policy-weight training, and literal trace replay is presented as a future extension. [56, 57]

Adaptation also concerns the user and the stability of repeated workflows. PerPilot identifies missing personal information, retrieves it from accumulated memory, or explores a likely application before requesting clarification if information remains unavailable. This makes instruction interpretation part of control rather than assuming every goal is fully specified. AppCopilot describes a different reuse mechanism: semantic matching selects successful historical logs for direct replay under stable layouts and task structures. Its coordinate calibration moves out-of-region clicks toward detected widgets while remembering ineffective corrections. These mechanisms depend on the validity of stored information and detected regions. Reusing a preference, replaying a path, and correcting a coordinate therefore require different checks; none follows automatically from a high retrieval match. [58, 59]

Across these methods, generalization depends on what is retained and what can be checked. Visual grounding broadens interface access; application knowledge reduces repeated discovery; demonstrations and synthetic rollouts expand experience; outcome learning changes behavior under feedback. Each can also introduce a new dependency: perceptual errors, stale memory, collection bias, or an imperfect verifier. The strongest methodological reports expose those dependencies and distinguish local prediction, offline trajectory scoring, live completion, and infrastructure reliability. This makes progress interpretable without assuming that a larger model, a longer reasoning trace, or a higher score under a different protocol resolves the entire computer-use problem. [13, 31, 42, 53]

Evaluation, interaction data, and safety

A computer-use benchmark measures an agent under a particular observation interface, action interface, task population, initial state, execution budget, and scoring procedure. A screenshot-to-coordinate test measures something different from selecting an element already identified in an accessibility tree. Likewise, predicting the next recorded action differs from completing a task after the agent has changed the environment. Here, interactive evaluation means that executed actions produce subsequent observations; it does not require a live Internet connection or a conversation with a human. A locally hosted website can support interactive evaluation, whereas a Gym-compatible wrapper can expose a single offline prediction problem. BrowserGym makes this distinction concrete by accommodating both executable web tasks and recorded WebLINX action prediction. [60]

These distinctions prevent a defensible universal leaderboard across the resources below. Even within a named benchmark, a changed evaluator, task subset, action budget, or permitted interface changes the experiment. The original World of Bits describes 100 authored MiniWoB environments, while the workflow-guided exploration study selects 40 of 80 then-public tasks for its action space and disables partial rewards. These are different historical populations and protocols, rather than inconsistent measurements of one immutable test set. [9, 10]

Environments and scoring contracts

Executable environments move evaluation beyond matching a demonstration. WebArena checks answers, navigation, and changes to website state; OSWorld checks artifacts, settings, and other task-specific outputs after execution. This allows different successful trajectories to receive credit. However, “functional evaluation” does not imply that every verdict is an exact, deterministic comparison. WebArena includes language-model semantic matching; VisualWebArena adds visual question answering (VQA) and image-similarity checks. Evaluator implementation and judge versions therefore belong in the experimental specification alongside the policy model. [14, 61, 15]

The interface also supplies information and authority. WorkArena can expose element identifiers, coordinates and action-error messages; Windows Agent Arena supports clipboard and window metadata as well as screenshots. AndroidWorld includes limited function-calling options beyond gestures. These are legitimate experimental configurations, but each changes what the agent must infer or execute. A comparison between backbones is therefore interpretable only after aligning the surrounding agent configuration. Conversely, an interface ablation can deliberately vary those capabilities, provided the change is identified as part of the intervention rather than attributed entirely to the underlying model. [62, 63, 16]

State control is part of the measurement, not merely deployment plumbing. BrowserGym documents backend restoration and dependencies that prevent one agent’s modifications from corrupting another run. WorkArena++ uses separate task accounts to isolate state changes. AndroidWorld fixes application versions and the device clock as well as task parameters. In contrast, AssistantBench deliberately uses the open web, selecting comparatively stable answers without eliminating future content changes. These choices trade environmental breadth against control; neither setting alone establishes performance across the other. [60, 64, 16, 65]

Stopping has its own semantics. WebArena includes requests that cannot be fulfilled because necessary evidence, permissions or functionality are absent; VisualWebArena evaluates explanations of unachievable tasks. WorkArena++ also distinguishes declaring infeasibility from justifying it. Thus a successful episode need not end in the requested state change: correctly recognizing an impossible request can itself be the desired outcome. This differs from refusing a feasible but unsafe request. Aggregate results should preserve that distinction, since an agent that stops, answers, requests clarification or changes state may receive superficially similar labels for substantially different reasons. [14, 61, 64]

Interaction data and the limits of offline proxies

Offline corpora provide reproducible observations and labels at a much lower execution cost, but their scores answer narrower questions. Mind2Web supplies the correct action history when scoring each step; its task score requires all recorded steps to match. AITW explicitly compares action matching with human online judgments and notes that alternative successful actions can be rejected by the offline matcher. AndroidControl likewise acknowledges that offline evaluation does not reward alternative routes or corrective behavior. Consequently, an all-steps-correct trace score should not be relabeled as independently observed end-to-end task completion. [13, 11, 67]

Instruction granularity also changes the inference problem. AndroidControl distinguishes high-level goals from prompts that additionally supply the next low-level instruction. GUIOdyssey separately evaluates these settings and tests cross-app histories. Supplying the next instruction removes part of the planning burden, so the resulting score cannot be interpreted as equivalent autonomous competence. Grounding-only sets provide a useful complementary diagnosis: ScreenSpot isolates target localization, while ScreenSpot-Pro stresses small targets in professional high-resolution interfaces. Strong localization is relevant to successful control, but neither benchmark directly tests recovery, stopping, or side effects. [67, 68, 29, 69]

GUIOdyssey pairs human demonstrations with model-generated low-level instructions and semantic annotations. These annotations are training products rather than observations of a demonstrator’s internal reasoning. The methods section discusses the analogous distinction for OpenCUA. [68]

Reference traces and recovery data answer different questions. As discussed under memory and recovery, a useful recovery example must identify the mistaken state, corrective action and observed result. A clean all-correct sequence cannot by itself establish competence on those off-path states.

Generalization, efficiency, and uncertainty

Generalization needs a declared axis. Mind2Web separates unseen tasks, websites and domains; AndroidControl tests unseen apps, tasks and categories; GUIOdyssey adds device transfer. These partitions probe different shifts and should retain their original membership rules. In particular, AndroidControl explicitly allows overlapping held-out subsets. AITW’s limitations further identify English-only prompts, a restricted annotator population, and mouse/keyboard interaction with emulated phones. More recorded episodes therefore do not, by themselves, establish broader linguistic, demographic or physical-interaction coverage. [13, 67, 68, 11]

Train/test separation is necessary but does not certify independence from all earlier training. OpenCUA reports avoiding OSWorld task overlap; OS-Harm requests exclusion from training and supplies a canary marker. These are documented safeguards, not an independent audit of every foundation model’s pretraining corpus. We therefore distinguish known task overlap, benchmark-specific tuning, and unknown pretraining exposure. A related issue is environmental overfitting: AndroidReality perturbs AndroidWorld’s observations, transitions and action execution while preserving task goals and rewards. Its predominantly single-perturbation protocol tests controlled robustness, rather than arbitrary combinations of deployment failures. [46, 70, 71]

Uncertainty also depends on the sampling unit. Steps within a demonstration, variants of an instruction template, and repeated seeds of the same task should not be described as unrelated tasks. WorkArena++ deliberately reuses workflows under different instruction levels, while AndroidWorld’s robustness analysis observes variation across parameter seeds. DIBench explicitly pairs benign and injected runs using matched seeds. These designs support different comparisons: transfer across task families, sensitivity to initial conditions, or a controlled change at a decision point. Reports should identify which population an average describes before attaching an uncertainty interval or claiming a general improvement. [64, 16, 72]

Efficiency cannot be reduced to a step count without defining a step. A single call may issue several operations, and a grouped action may avoid another screenshot or model request. OSWorld-Human provides both single-action and grouped human reference trajectories and proposes success-weighted efficiency with a failure-budget penalty. That metric complements, rather than replaces, elapsed time, model calls, tokens and monetary cost. Comparisons should disclose what is charged for retries, failed attempts, parsing repair and environment recovery; otherwise two agents with the same success score may have substantially different operational costs. [73, 60]

Human reference trajectories also encode choices about permitted tools. OSWorld-Human generally excludes programmatic application-specific solutions unless the task source calls for them, while allowing familiar keyboard shortcuts. Its reference is consequently a documented human interaction procedure, not a proven shortest path through every possible computer interface. Normalization against that reference should retain these assumptions, especially when an agent is allowed code, macros or multiple actions per observation. [73]

Evaluators require their own validation. OSWorld-Pro compares step-, subgoal- and task-level model judgments with human trajectory labels. A recent WebArena-Lite audit retains original and human-corrected outcomes, while explicitly limiting its conclusions because it uses one rollout per task and identifies possible state carryover and guidance leakage. These studies motivate trace inspection and human-checked judges, not treating human correction as infallible. AndroidControl-Curated proposes revised grounding and task annotations, but its exact-point comparison should not be generalized to the original AndroidControl paper, which already describes relaxed action equivalence. Improvements under changed scoring rules must be separated from policy improvements. [66, 74, 75, 67]

Safety beyond nominal task completion

A correctly completed task can still be unsafe. OSWorld’s original discussion acknowledges that its completion checks do not comprehensively detect unnecessary harmful side effects. OS-Harm separates deliberate user misuse, indirect prompt injection and harmful model behavior during otherwise benign tasks. MobileSafetyBench instead makes the utility–safety distinction visible through low-risk completion and high-risk refusal or consent requests. Refusing every request can suppress realized harm while providing little useful assistance; failing to carry out a harmful instruction can also reflect insufficient capability rather than a reliable boundary. [15, 70, 76]

Indirect-injection results are meaningful only relative to attacker control. WASP restricts the adversary to user-editable website content. The VisualWebArena adversarial-robustness study considers restricted text or image modifications and analyzes how compromised observations propagate through agent components. RedTeamCUA extends testing into a hybrid web–desktop sandbox while retaining explicit restrictions on what the attacker may change. These differ from malicious-user requests and from attacks that assume control of the policy itself. Their rates should not be pooled into a single measure of “agent security.” [77, 78, 79]

Exposure and completion are separate denominators. VWA-Adv starts agents at an attack encounter; RedTeamCUA distinguishes this decoupled setting from end-to-end navigation. Such trials isolate susceptibility after exposure, whereas end-to-end trials additionally depend on reaching the relevant state. WASP reports intermediate diversion separately from completed attacker goals, and RedTeamCUA distinguishes attempted from executed harmful behavior. Together, these designs show why evaluation should retain benign utility, exposure, deviation and completed harm as separate outcomes. A navigation failure is not evidence that encountered malicious content would be resisted. [78, 79, 77]

Decision integrity adds another boundary. DIBench holds the instruction and candidate set fixed while comparing benign and injected selection states. Its completion metric means committing to a candidate; correctness means selecting an acceptable candidate; targeted steering is measured through paired choices. Commercial-app tests use fixed screenshots and context, rather than publishing malicious content or executing transactions. The study therefore supports analysis of manipulated choices even when the interaction appears orderly, but does not establish end-to-end transaction safety or the behavior of production phone assistants. [72]

Other threats enter at different stages. The third-party mobile-agent study (A)I Sees What You Don’t assumes an installed malicious Android application and examines perception and automation-channel boundaries, with permission requirements varying by vector. Its controlled feasibility demonstrations do not estimate attack prevalence or directly transfer to first-party and iOS assistants. Poison Once, Control Anywhere instead assumes compromised visual fine-tuning data. Its target-action and contextual-output measurements concern a training-time backdoor, not the same runtime-injection experiment. Distinguishing these assumptions identifies which component, interface or data pipeline a defense must protect. [80, 81]

Safety judges are themselves part of the evaluated system. OS-Harm uses semantic trajectory judgments and explicitly assumes the adversary targets the agent rather than the judge. SafeMobile combines a trajectory-aware guard with automated scoring and human comparisons; its score threshold is a particular judging rule, not a universal scale of risk. Reporting human agreement, disagreements, the evidence available to the judge and changes in benign task completion is therefore essential when interpreting a defense. No result here demonstrates a universally secure computer-use agent. [70, 82]

Reporting recommendations

The following are recommendations from this synthesis, rather than additional experimental findings. First, publish an evaluation contract: task IDs and version; split and sampling unit; observation/action restrictions; environment image, app versions and reset procedure; model snapshot; step and time budgets; seeds; retry policy; and evaluator code or judge configuration. Preserve raw outcomes alongside corrected outcomes and state the correction rules. Report repeated trials where feasible and identify infrastructure failures separately, without silently deleting them from the denominator.

Second, retain enough execution evidence to inspect the conclusion: task instruction, observations, actions, action results, relevant final artifacts and termination reason. Separate grounding, planning, execution, recovery and stopping failures. Report success together with latency and resource consumption, then stratify by meaningful shifts such as unseen applications, task families and devices. For security evaluation, additionally declare attacker permissions, exposure conditions, allowed channels and the definitions of attempted and completed harm.

Finally, document access and licensing by component. Benchmark code, task instructions and screenshots, model weights, and the applications needed for execution can have different terms. Windows Agent Arena explicitly distinguishes its open implementation from the Windows image it cannot redistribute; ScreenSpot-Pro cites professional-software licensing constraints when limiting its scope to grounding. A public repository is consequently not evidence of unrestricted rights to every associated artifact. Report verified terms separately, leave unknown terms unresolved, and distinguish paper-reported collection sizes from the actual version of data made available. [63, 69]

Human agency, personalization, and proactive assistance

Computer use becomes a human–agent interaction problem when a system chooses when to intervene, interprets personal context, or changes information on a user's behalf. A useful distinction is between recognizing a likely intention, obtaining authority to act on it, executing the operation correctly, and explaining the resulting state. These requirements can vary independently. We use recent interface-specific studies to clarify the distinction, rather than treating more autonomous behavior as a universally better outcome.

Act2Intention separates intention understanding, prediction, and execution. Its primary text specifies user-confirmed execution, and its data construction separates observed trajectories from trajectories generated using behavior-derived or generated personas. The limitations explicitly distinguish predicting an intention from choosing an appropriate moment to interrupt. They also state that real-user acceptance of proactive suggestions was not evaluated. Thus, its intention metrics support the proposed modeling tasks; they do not establish that unsolicited assistance is welcome or that synthetic personas represent a deployment population. [83]

ProactiveVA studies assistance inside visual-analytics workflows. It combines interaction-log interpretation with suggestions, previews, and interface operations. Its in-lab evaluation involved ten users and reported feedback on assistance and control; the discussion records requests for finer intervention during the agent's process. These observations motivate studying the timing and controllability of assistance. They should be read as evidence from the reported visual-analytics setting, not as a population-wide estimate of the benefit of proactive agents. [84]

NeXUI makes explanation and user oversight part of an assistive-interface evaluation. Its packaged tasks use recorded linear transition graphs; unsupported actions do not create alternative paths. The reported model experiments used a 25-task validation split and a text-based baseline, without screenshots supplied to the model. This is a controlled setting for studying action–explanation alignment and confirmation boundaries. Its result should not be interpreted as a measurement of live open-web performance or a user study demonstrating accessibility benefits. [85]

A research agenda follows from these distinctions. Personalization experiments should separate recall of an explicit preference from inference about an unspoken preference, and should report what happens when the two conflict. Proactive systems should expose how users reject, correct, or revoke a proposal. Explanations should be checked against observable state changes rather than judged only for fluency. These are evaluation recommendations of this survey. A suitable study would measure task success, correction effort, unwanted actions, interruption burden, and perceived control together, with consent and data minimization appropriate to the setting.

From literature lists to reusable research assets

A survey becomes more useful when a reader can move from an argument to the precise resource needed to investigate it. A paper may document several checkpoints, datasets, evaluators, or implementation components. Each can change independently. Accordingly, an actionable research collection needs identities, explicit relationships, source evidence, and a way to preserve uncertainty. Its design must serve researchers reading a comparison and software agents retrieving records for a later decision.

Documentation precedes aggregation

The FAIR principles connect discovery and reuse with persistent identifiers, rich metadata, standardized access, provenance, and explicit reuse terms. Their machine-oriented perspective is particularly relevant to retrieval systems that cannot safely reconstruct missing context from a title. FAIR4RS extends the discussion to research software and explicitly distinguishes component and version identities. These principles motivate our separation of a research work, its released artifacts, and observations about those artifacts; they do not prescribe our particular schema or establish that this catalog satisfies every FAIR criterion. [86, 87]

Datasheets for Datasets organize documentation around a dataset's purpose, composition, collection, processing, uses, distribution, and maintenance, including cases where information remains unknown. Model Cards similarly connect a model's identity and intended use with evaluation data, relevant factors, metrics, and caveats. For computer-use research, these approaches encourage documenting the conditions behind a resource rather than treating an available download as a complete specification. [88, 89]

The AwesomeCUA snapshot as a worked example

The frozen inventory records 220 reviewed papers with explicit connections to 311 paper–asset pairs. These connections require relation-specific source evidence; incidental README mentions and shared topic labels do not qualify. The ratio describes what this catalog has connected. It is not an estimate of how many papers released code, and an unlinked paper should not be described as unreproducible. The same distinction applies to specification coverage: the snapshot contains structured profiles for 60 of its 83 benchmark resources and 66 of its 85 dataset resources. A missing catalog profile means that this collection has not recorded the protocol in that form. [22]

The release illustrates why the unit of a claim matters. A repository can contain several components without making each component a new independent project. A model family can have several checkpoint releases with different availability or licensing. A paper can have both a preprint and a proceedings edition without becoming two studies. Keeping these identities separate permits a query about a training dataset or evaluation artifact to return a precise object, while relationships recover the broader research context. Identity resolution should therefore require affirmative evidence, rather than merge records solely because names resemble one another.

Evidence hashes are useful but have a limited role. They identify the bytes that were examined when those bytes are retained; they do not prove that an annotation is correct or that a mutable URL will still return the same response. A reliable update procedure must compare observations and reopen affected review decisions. Similarly, a successful HTTP request establishes accessibility of that response, not completeness of an implementation or permission to reuse every file. A paper's claims, the state of its code release, and reproduction by a third party should remain separately expressible.

A shared interface for people, RAG, and agents

Human navigation and machine retrieval should be generated from the same records. Human readers benefit from task-oriented paths, concise purpose statements, comparable specifications, and direct source links. Retrieval systems need stable identifiers, record boundaries, review states, citation URLs, and fingerprints. A useful retrieval-augmented generation (RAG) export carries the descriptive text together with the fields needed to qualify an answer. An agent can then retrieve a reviewed benchmark record, inspect its protocol, and cite the primary source instead of treating a search snippet as the complete evidence.

The frozen release packages 28 main exports and 105 exports across 35 overlapping topics, alongside manifests and guidance. Its reviewed and complete RAG exports have different populations; the complete export includes metadata-only records. Record identity, scope, and review state must survive indexing. Upserts should use canonical identifiers, and deletions should be reconciled against a complete snapshot rather than inferred from a partial search result. A retrieval experiment using this material should report its snapshot and selection policy so that a change in annotation coverage is not mistaken for a change in model quality. [22]

Related collections can extend discovery without collapsing their scope into computer use. For example, Awesome-Jev organizes Jev and System One research, models, evaluation resources, and integrations around typed decisions. It is a useful neighboring entry point for studying bounded decision components. Individual interface applications, such as Jev-Mobile in the methods section, require their own primary evidence. A catalog-to-catalog link is not a claim that every item in the neighboring collection is a GUI agent, or that its annotations independently validate a system's results.

This design also clarifies what an “awesome list” can and cannot contribute. Its scientific value lies in reducing ambiguity about resources and making discovery and verification easier. The number of entries alone does not demonstrate search recall, balanced coverage, or correctness. Progress should be measured through auditable additions, resolved identity conflicts, richer protocol descriptions, and successful reuse of a fixed snapshot. Those outcomes are suitable targets for future usability and retrieval studies; they are not results asserted by the present descriptive analysis.

Open problems and a research agenda

The preceding comparisons suggest a connected research agenda. Better grounding, stronger planning, larger training corpora, and faster inference can each improve part of a system, while changing the assumptions under which the whole system is evaluated. The following directions are proposals derived from the synthesis. They should be tested through controlled studies rather than treated as established consequences of the catalog statistics.

Generalization needs several axes

A useful next step is temporal transfer: freeze training and artifact versions, then test on documented interface changes. Combine this with the task-, site-, app- and device-level distinctions in the evaluation section. State whether failures arose from changed controls, unavailable functionality or mistaken intent. This would make the intended meaning of “generalist” explicit rather than treating a single aggregate score as evidence for every form of transfer.

Measure the system that actually runs

An action policy is embedded in a runtime that parses observations, handles waits and timeouts, invokes tools, maintains state, and decides when to stop. The efficiency survey's accounting across observation, memory, action, and runtime motivates measuring this complete path. Evaluation should distinguish time spent producing a decision from time spent reaching a correct application state. It should also record where failures arise: a model error, an invalid action contract, an unavailable application, or an evaluator that cannot recognize a valid outcome. These categories support diagnosis without automatically excusing any failed deployment run. [21]

There is also a need for calibrated uncertainty at several levels. A well-formed action can target the wrong object; a high-confidence decision can rely on an incomplete candidate set; a fluent completion report can contradict the environment. A research program should therefore examine the relationship between uncertainty in perception, action selection, outcome verification, and escalation. The relevant target is the quality of the resulting decisions under a declared cost and authority model, rather than an isolated confidence value. The trustworthy-GUI review's stage-based analysis provides a useful starting point for that decomposition. [20]

Make evidence maintenance part of research infrastructure

A rapidly changing field needs correction mechanisms as well as new entries. Useful infrastructure would retain the relationship between a paper revision, a dataset release, an evaluator revision, and a result observation. It would expose unresolved differences instead of silently selecting the most convenient value. Stable record identifiers and versioned exports make this tractable for people and retrieval agents, while documentation of intended uses and measurement conditions keeps those exports interpretable. This extends the practical motivation of data and model documentation to the interconnected artifacts of computer-use research. [86, 87, 88, 89]

Limitations

The review's selection is shaped by a public, predominantly English-language catalog, its discovery sources, and the terms used in supplementary searches. Closed deployments, regional products, unpublished systems, and work using different terminology may be missing. Metadata-only records are preserved as discovery evidence, but are not treated as reviewed findings. The catalog's year and topic distributions are affected by its selection and annotation process. They cannot establish global research trends, prevalence, or comparative quality.

Primary reading targeted relevant sections of representative works. It did not independently reproduce benchmark runs, inspect every released file, or adjudicate every claim in every cited paper. Some sources are preprints and may change. When a paper or repository reports a result, this article preserves that provenance rather than presenting it as our measurement. Comparisons emphasize experimental contracts and mechanisms; they do not combine heterogeneous success rates into a universal ranking.

AI assistance introduces additional risks in selection, extraction, and synthesis. Version checks, explicit source receipts, claim-level notes, and separate review states reduce ambiguity but do not eliminate mistakes. Parallel AI readers are not a substitute for independent human scholarly review. The machine-readable ledgers are provided to enable corrections and further verification. Catalog integrity and software checks establish consistency of the supplied artifacts, not the truth of every scientific statement.

Conclusion

Computer-use agents unite interface perception, action selection, learning, and runtime control. Progress is easiest to interpret when the available observations, permitted actions, learning signal, evaluation protocol, and artifact version are explicit. A grounding score, a demonstrated next-action prediction, a successful interactive task, and acceptable assistance to a person answer different questions. This survey connects those questions through representative primary reading, comparisons that retain experimental conditions, and a reproducible inventory of research assets.

The practical direction is to improve both capability and the evidence supporting its use. Research collections can help by connecting papers to precise artifacts, preserving review states and unknowns, and offering consistent interfaces for readers, retrieval systems, and agents. The resulting knowledge base is valuable when it makes a claim easier to inspect and a resource easier to reuse. Its breadth remains a continuing, auditable objective.