Source-reviewed research and assets tagged with Efficiency.
Membership follows reviewed tags; it does not establish a dependency between neighboring records. Dates preserve source precision. Annotations retain their original language.
Uses an LLM judge to evaluate and filter synthetic GUI interaction trajectories for training a compact vision-language agent. OSWorld experiments assess the policy designed for local execution.
GEM detects instructions outside a GUI agent’s capabilities by modeling distances in its input-embedding space with a Gaussian mixture. Experiments across mobile, computer and browser data also examine selective escalation to a cloud model when an input is detected as unfamiliar.
LLM-Explorer uses language models to maintain compact knowledge for mobile app exploration, then generates actions without invoking the model at every step. The study compares exploration coverage and cost across twenty apps.
EcoAgent splits mobile automation between cloud planning and on-device observation and verification. Compact screen descriptions and feedback reduce communication costs, with evaluation on AndroidWorld.
Introduces V-Droid, which verifies candidate actions in a discrete action space rather than directly generating each mobile operation.
Combines rapid target prediction with deliberate interface analysis, adaptively selecting the grounding process according to task complexity.
Records interaction history and converts repeated action sequences into reusable high-level shortcuts for mobile GUI tasks.
Selects critical steps from expert interaction trajectories for targeted agent fine-tuning, studying efficiency and generalization across tasks.
Examines energy and carbon-accounting limitations in web-agent evaluation and argues for transparent, dedicated sustainability measurements.
Learns action-value functions from offline device-control experience and uses them to rank candidate actions for policy improvement.
Couples large- and small-model web-agent exploration with trajectory synthesis and distillation to improve both model types iteratively.
AppVLM first learns app actions from AndroidControl and then refines its policy through data collected in AndroidWorld. The study compares offline action prediction and online task completion for a lightweight vision-language smartphone agent.
Combines information-sensitive image cropping with a learning loop between element description and grounding for complex interface understanding.
Combines visual token selection, interleaved visual-action history, and curated instruction data in a compact GUI action model.
LiMAC combines a compact Action Transformer with a fine-tuned vision-language model for Android app control. It uses textual goals and histories of screenshots and UI trees to predict actions, with evaluation on two mobile-control datasets.
Introduces a smartphone-agent benchmark with English and Chinese tasks, real-time Android interaction, and completion and resource-use metrics.
Introduces asynchronous distributed reinforcement learning with centralized updates and decentralized interaction collection for device-control agents.
Adapts a compact Florence-2 model to map user instructions to screen coordinates using multitask training and augmented data.
Studies agent-system and interface co-design that incorporates user interruptions and intermediate feedback to reduce planning latency.
Introduces AXIS, an API-first application agent that can use UI actions and expand its available APIs through automated application exploration. The work evaluates the approach on Microsoft Word tasks.
Introduces Steward, a natural-language browser automation system that repeatedly plans and executes website actions. The paper examines completion detection, state representation, responsiveness, and caching in real-site tasks.
Agent-E combines hierarchical web-task control with DOM distillation and observations of interface changes. WebVoyager experiments inform a broader discussion of primitive skills, observation reduction, and self-improvement in agent architecture.
Introduces UReader, which combines unified instruction tuning, auxiliary text-reading tasks and shape-adaptive image crops for OCR-free visual language understanding. Its evaluation includes webpage screenshots alongside documents, tables, charts and natural images.
Synapse filters task-irrelevant page state and places complete abstracted state-action trajectories into the prompt. An embedding-based exemplar memory retrieves relevant demonstrations for new tasks, evaluated on MiniWoB++ and Mind2Web.
Separates web navigation prompting into a SUMMARIZER that extracts action-relevant observations and an ACTOR that chooses the next action from the condensed context.
The Versatile UI Transformer jointly models interface screenshots, structure, and language for five UI tasks. Its encoder supports element detection when structural metadata is absent, while an autoregressive component handles UI question answering and command grounding.
Human reference trajectories and scoring code for measuring the action efficiency of computer-use agents on OSWorld tasks.
Local-page benchmark comparing six browser representations on token use, factual reading and target-element selection, with deterministic answer scoring.
GUI grounding research using modality-aware visual reasoning and test-time scaling.
Lightweight GUI-agent vision-language-action model from Show Lab and Microsoft, released with UI-guided token selection, training code, the ShowUI-web and ShowUI-desktop datasets and vLLM inference.
ShowUI vision-language-action checkpoint fine-tuned from Qwen2-VL-2B-Instruct, published as a single PyTorch weight file under the MIT license and tagged for GUI agents and computer use.
The released UI-AGILE model for GUI grounding with reward shaping, crop-based resampling and decomposed selection during inference.
Qwen2.5-VL-3B-based GUI grounding checkpoint that predicts click locations from screenshots and text instructions across desktop, mobile and web screens.
Offline device-control training code that learns action values and uses them to select promising actions, with linked trajectories and checkpoints.
Distributed online reinforcement-learning framework for Android agents, with environment setup and trajectory collection tools.
ScreenSpot-Pro evaluation harness with explicit coordinate transforms, prompt and crop-based inference comparisons, and per-sample grounding accuracy and cost records.
GUI-grounding research code for spatial voting across sampled predictions and test-time policy optimization using region-consistency rewards.
Computer-use training framework with VeriGen task synthesis, adaptive rollout sampling, and visual-context segmentation.
Showing 121–158 of 158 resources.
Get full records and evidence through the topic export manifest. Source records can be passed to the standard RAG and citation exporters after cloning the catalog.
