Research papers

From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces

Studies instruction-following agents that observe screenshots and act through a generic mouse-and-keyboard action space, evaluating pixel-based pretraining for GUI tasks on MiniWoB++.

AI-assisted editorial review against a primary source. This is a catalog annotation, not an independent reproduction or endorsement.

Original source ↗Read as Markdown Record JSON

What is this resource?

Resource type: Research papers
Environment: web
Topics: grounding, action-prediction, web-navigation
Record publication date: 2023-05-31
Evidence retrieved: 2026-10-16T06:31:20Z
Resource ID: arxiv:2306.00245

Recorded paper version

arXiv v2

Version updated: 2023-12-06T23:46:36Z.

This identifies the retained arXiv metadata snapshot; newly observed changes require separate review.

Abstract fingerprint

d06f7480695f45546b62b5b8c3d8c174ca52936e158d46e626689d0918711417

Authors

Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, Kristina Toutanova

Cite this paper

Download BibTeX · Download CSL JSON

Citations preserve the selected source edition, its authors and recorded date precision. Unrecorded venues and identifiers are omitted.

Publication record

Title: From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
Authors: Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, Kristina Toutanova
Publication date (recorded precision): 2023-12-15
Venue: Advances in Neural Information Processing Systems
Volume: 36
Pages: 34354-34370
DOI: 10.52202/075280-1490

Publication source ↗ · Publisher or author PDF ↗

Publication BibTeX · Publication CSL JSON

This publication is a separately recorded edition of the same work. Its citation uses its own title, authors and date; the retained arXiv citation above is unchanged.

Date source: Explicit publisher metadata: 2023-12-15; precision=day.

Source notes: The NeurIPS landing metadata adds the initial N to Kristina Toutanova; the formal paper title page does not. Publication authors use the PDF. Publisher-hosted proceedings record; distinct from the retained arXiv preprint snapshot.

Explore related topics

GUI grounding Action prediction Web navigation

Scope and limitations

Reviewed against the retained original arXiv abstract. Experimental results and artifact availability have not been independently reproduced.

Primary evidence and verification

Source retrieval is separate from experiment reproduction. Use the original publication or publisher page when citing claims about the resource.

arxiv api
Retrieved 2026-10-16T06:31:20Z. Supports: name, authors, published_at, updated_at, identifiers.arxiv, details.arxiv_snapshot.
Content fingerprint
cbe8bfcbce4170a94a6cf42de01fbe0a448e7b1d29fa75a0de877105a98ec08e
editorial primary review
Retrieved 2026-10-17T09:53:48+00:00. Supports: details.publication.date_basis, details.publication.date_parts, details.publication.doi, details.publication.identity_basis, details.publication.notes, details.publication.pages, details.publication.pdf_url, details.publication.title, details.publication.type, details.publication.url, details.publication.venue, details.publication.volume.
Content fingerprint
938e0a31512bb9a7793b568858f81eb3ceb650a71aae2d91b5292990e890201b
editorial primary review
Retrieved 2026-10-17T09:59:47+00:00. Supports: details.publication.authors, details.publication.identity_basis, details.publication.notes.
Content fingerprint
3827298f44d2f24f1333b4d46ecb5eb88c8b028d192b11424580655b38e024f9
editorial primary review
Retrieved 2026-10-16T06:31:20Z. Supports: name, kind, summary, environments, topics, scope, notes.
Content fingerprint
cbe8bfcbce4170a94a6cf42de01fbe0a448e7b1d29fa75a0de877105a98ec08e

Suggest a correction

Open a source-backed correction. Include this record ID and the original source. Read our editorial policy and coverage boundaries.