Source-reviewed research and assets tagged with Evaluation.
Membership follows reviewed tags; it does not establish a dependency between neighboring records. Dates preserve source precision. Annotations retain their original language.
Analyzes web-agent vulnerabilities at the component level, examining goal placement in prompts, multistep action generation, and observations as sources of increased susceptibility.
Examines energy and carbon-accounting limitations in web-agent evaluation and argues for transparent, dedicated sustainability measurements.
Describes the CUGA enterprise-agent development process, emphasizing iterative evaluation, failure analysis, and refinement on software-task benchmarks.
Introduces a visual IDE environment and benchmark for software-engineering tasks, comparing screenshot-based control with access to editing and shell tools.
Examines active environmental injection against mobile agents, testing whether adversarial interface content redirects decisions during task execution.
Introduces desktop and web tasks with varied initial states to test planning recovery and adaptability beyond canonical starting configurations.
Introduces CowPilot, a web-navigation framework where agents suggest actions and users can pause, reject, override, or resume them. It evaluates autonomous and collaborative use through task success and human effort.
Introduces a screenshot-based Windows GUI agent and the WinSpot benchmark for evaluating localization of actionable interface regions.
Introduces WebWalkerQA and an exploration-and-critique framework for gathering information by traversing multiple levels of a website.
Introduces Android Agent Arena with online application tasks and procedural evaluation based on reaching essential intermediate states.
Evaluates GUI testing through test-intent generation, execution, and defect detection using real, modified, and synthetic mobile application data.
Simulates a software-company workplace to evaluate agents on professional tasks involving websites, programs, files, and coworker communication.
Uses a Windows-agent case study to describe a development workflow for large action models, covering data collection, training, grounding, environment integration, and evaluation.
Describes BrowserGym's shared web interaction environment and AgentLab tooling for consistent experiments across multiple agent benchmarks.
Surveys virtual agents for browsers, applications and operating systems, organizing prior work by task type, observation and action spaces, model strategies and evaluation methods. It examines screen-based interaction and keyboard, mouse and touch control, alongside transferability and long-sequence decision-making challenges.
Presents a preliminary study of Claude 3.5 Computer Use across desktop tasks and software, analyzing planning, action, and critique behavior and providing a framework for deploying GUI-automation models.
Tests whether adversarial visual pop-ups divert computer agents from intended tasks in desktop and web benchmark environments.
Surveys multimodal mobile agents, organizing prompt-based and training-based approaches alongside evaluation benchmarks, supporting technologies, and open challenges in mobile task execution.
Provides a shared Android action environment, reproducible tasks, and instruction data for training and comparing language and multimodal agents.
Introduces CASA to assess web agents' responses to cultural and social norms in shopping and forum tasks, including misleading observations.
Evaluates whether web agents can retain procedural skills and factual information from long video tutorials and use them during tasks.
Evaluates mobile agents on misuse, side effects, and indirect prompt-injection risks in an Android emulator environment.
Introduces a black-box web-agent red-teaming framework that learns adversarial prompts from agent feedback to study security weaknesses.
Compares browsing-only, API-only, and hybrid web agents on WebArena, examining how access to application APIs changes task execution alongside browser interaction.
Introduces a smartphone-agent benchmark with English and Chinese tasks, real-time Android interaction, and completion and resource-use metrics.
Introduces BrowserART and studies whether chat-model refusal behavior transfers to browser agents carrying out potentially harmful instructions.
Develops a taxonomy of mobile-action consequences and annotated interaction data to evaluate whether models recognize impactful or risky UI operations.
Evaluates web-task completion under explicit enterprise policies, measuring policy-compliant completion and safety violations separately.
Studies text and visual validation for web agents and uses the resulting validator to detect failures and guide workflow self-refinement.
Introduces IconDesc, which generates mobile icon alt-text from partial UI context such as element identifiers, bounds, nearby text and hierarchy. Model evaluation and a user study examine the resulting accessibility descriptions.
Studies how the ordering of UI elements in an agent observation affects web and desktop performance. A pixel-based element detector and dimensionality-reduction ordering are evaluated on OmniACT when document hierarchy is unavailable.
Evaluates privacy-directed webpage instruction injection against generalist web agents using PII-related interaction steps from Mind2Web, including attempts to expose specific user information or entire requests.
Adapts real-desktop evaluation to Windows applications and supports parallel execution of benchmark tasks.
Introduces question answering over individual screens, multiple web pages, and navigation traces to test information synthesis across views.
Separates web-agent planning and grounding evaluations on Mind2Web to analyze which components limit task performance.
Benchmarks visual agents across GUI, embodied, and design tasks, with training trajectories collected through solvers, agents, and human demonstrations.
Measures how irrelevant GUI content affects multimodal agents using a simulated dataset and three perception patterns. The study also examines adversarial environment injection and possible ways to improve instruction faithfulness.
Evaluates language agents on office workflows requiring information integration, long-term planning, and switching between applications.
Introduces simulated application APIs and state-based task evaluation for interactive coding agents performing everyday digital workflows.
Introduces realistic open-web tasks and the SeePlanAct agent, analyzing the limitations of browsing and retrieval in time-consuming assistant work.
Benchmarks multimodal agents on data-science and engineering workflows involving enterprise applications, code, and GUI operations in real computer environments.
Develops a threat taxonomy and evaluation framework for attacks against multimodal mobile GUI agents, with case studies and dataset experiments.
Automates mobile GUI tests from natural-language requirements, combining action generation with verification based on interaction traces.
Introduces designer-authored mobile UI critiques and quality ratings, studying how targeted examples improve automated design feedback.
Defines compositional enterprise workflows to evaluate planning and reasoning, with generation of ground-truth interaction traces.
Trident combines exploration, monitoring, and detection agents to identify functional bugs that do not crash a mobile app. It reasons over screenshot sequences and interface transitions to construct test oracles, evaluated on 590 non-crash bugs and additional live apps.
Introduces CRAB, a framework for cross-environment agent tasks with graph-based evaluation and a benchmark spanning desktop and phone interactions.
Applies best-first search to agent actions within interactive web environments, allowing exploration and multistep planning at inference time. WebArena and VisualWebArena experiments examine search behavior and test-time computation.
Evaluates mobile agents using combined UI and API actions, tasks with varying app scope, and checkpoints for intermediate progress.
Defines goal identification from observed UI trajectories and an evaluation method for comparing intent descriptions within the same interface context. Experiments reuse Android and web automation data to compare human and model inference of users’ detailed goals.
Introduces Chinese-language GUI navigation traces and annotated screenshots collected from human interactions with mini-applications.
Introduces WONDERBREAD, a benchmark pairing human WebArena workflow demonstrations with screen recordings, action logs, and written procedures. Six evaluation tasks cover documentation, knowledge transfer, and process improvement beyond end-to-end execution.
Introduces adversarial tasks on VisualWebArena and an evaluation framework that tracks how manipulated information passes between agent components. The analysis examines vulnerabilities introduced by reflection and tree-search evaluators as well as direct action generation.
Introduces WebCanvas and Mind2Web-Live, evaluating online task progress through meaningful intermediate states on changing websites.
Introduces video-based GUI questions and annotations for studying temporal interface understanding across varied interaction settings.
Evaluates GUI automation from instructional videos at separate levels of high-level planning, action narration, and primitive execution.
Introduces GuardAgent, which translates safety requests into executable guardrail code using plans and retrieved demonstrations. Mind2Web-SC supplies a web-agent safety-policy evaluation alongside a separate healthcare access-control benchmark.
ScenTest builds scenario-specific event knowledge graphs from crowdsourced test reports containing GUI and action information. It uses the current app state and testing context to generate scenario tests across applications.
Introduces a mobile-agent benchmark with tasks across open-source applications, organized by difficulty to reduce manual evaluation effort.
Studies how Android task performance changes with demonstration scale and releases AndroidControl with high- and low-level instructions.
Showing 481–540 of 720 resources.
Get full records and evidence through the topic export manifest. Source records can be passed to the standard RAG and citation exporters after cloning the catalog.
