# Mapping Natural Language Instructions to Mobile UI Action Sequences

Introduces PIXELHELP for predicting mobile UI action sequences from instructions, together with action-phrase annotations and synthetic UI action descriptions for training. Separate Transformers extract instruction steps and ground object descriptions using interface content and screen position.

- Catalog URL: https://awesomecua.github.io/resources/arxiv-2005-03776/
- Original source: https://arxiv.org/abs/2005.03776
- Kind: paper
- Review: curated
- Scope: core
- Environments: unspecified
- Topics: mobile-navigation, grounding, data-generation, action-prediction
- Evidence checked: 2026-10-16T06:31:20Z
- ID: arxiv:2005.03776

## Limitations

The annotation and scope were reviewed against the primary paper abstract. Experimental results and released artifacts have not been independently reproduced.

- Record JSON: https://awesomecua.github.io/resources/arxiv-2005-03776/record.json

## Authors

Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, Jason Baldridge

## Citation exports

- BibTeX: https://awesomecua.github.io/resources/arxiv-2005-03776/citation.bib
- CSL JSON: https://awesomecua.github.io/resources/arxiv-2005-03776/citation.csl.json

## Publication record

- Title: Mapping Natural Language Instructions to Mobile UI Action Sequences
- Authors: Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, Jason Baldridge
- Publication date (recorded precision): 2020-07
- Publication source: https://aclanthology.org/2020.acl-main.729/
- Publication BibTeX: https://awesomecua.github.io/resources/arxiv-2005-03776/publication-citation.bib
- Publication CSL JSON: https://awesomecua.github.io/resources/arxiv-2005-03776/publication-citation.csl.json
- venue: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- publisher: Association for Computational Linguistics
- pages: 8198-8210
- doi: 10.18653/v1/2020.acl-main.729
- date basis: Explicit publisher metadata: 2020/7; precision=month.
- identity basis: Same work: exact normalized landing title, corresponding author list, and matching specific contribution. Publisher and arXiv versions remain distinct.
- notes: Publisher-hosted proceedings record; distinct from the retained arXiv preprint snapshot.
- Publication evidence: https://aclanthology.org/2020.acl-main.729/

## Recorded arXiv snapshot

- Version: 2
- Version source: https://arxiv.org/abs/2005.03776v2
- First submission: 2020-05-07T21:41:40Z
- Version updated: 2020-06-05T02:11:56Z
- Abstract SHA-256: 3af7b45455ffbd616e8cdf5612b19275ecb7bf07b1065af9f5b724ae19e96fb4
- Abstract hash normalization: sha256-utf8-whitespace-collapsed-v1

## License declarations

- code: not recorded
- data: not recorded
- document: not recorded
- weights: not recorded

## Primary evidence

- https://export.arxiv.org/api/query?id_list=1802.08802%2C1808.09132%2C2005.03776%2C2101.09465%2C2103.16057%2C2104.08560%2C2105.13231%2C2107.03374%2C2107.13731%2C2108.03353%2C2202.02312%2C2202.08137%2C2204.01691%2C2205.11029%2C2207.01206%2C2209.08199%2C2209.08655%2C2209.14927%2C2209.15099%2C2210.02663%2C2210.03629%2C2210.03945%2C2211.07615%2C2301.10165%2C2301.13280%2C2302.04761%2C2303.07280%2C2303.08774%2C2303.17491%2C2303.17580%2C2304.04370%2C2304.07061%2C2305.04091%2C2305.05662%2C2305.08144%2C2305.10601%2C2305.11854%2C2305.12487%2C2305.14257%2C2305.14325%2C2305.16504%2C2305.16653%2C2305.19308%2C2306.00245%2C2306.06070%2C2306.07863%2C2307.10088%2C2307.12856%2C2307.13854%2C2308.01990&max_results=50 (retrieved 2026-10-16T06:31:20Z; SHA-256 cbe8bfcbce4170a94a6cf42de01fbe0a448e7b1d29fa75a0de877105a98ec08e)
- https://aclanthology.org/2020.acl-main.729/ (retrieved 2026-10-17T09:53:48+00:00; SHA-256 c196294e19378d08d71a7cfd41c2e368f5a6b49ee7c3601257537b9ae5c42add)
- https://export.arxiv.org/api/query?id_list=1802.08802%2C1808.09132%2C2005.03776%2C2101.09465%2C2103.16057%2C2104.08560%2C2105.13231%2C2107.03374%2C2107.13731%2C2108.03353%2C2202.02312%2C2202.08137%2C2204.01691%2C2205.11029%2C2207.01206%2C2209.08199%2C2209.08655%2C2209.14927%2C2209.15099%2C2210.02663%2C2210.03629%2C2210.03945%2C2211.07615%2C2301.10165%2C2301.13280%2C2302.04761%2C2303.07280%2C2303.08774%2C2303.17491%2C2303.17580%2C2304.04370%2C2304.07061%2C2305.04091%2C2305.05662%2C2305.08144%2C2305.10601%2C2305.11854%2C2305.12487%2C2305.14257%2C2305.14325%2C2305.16504%2C2305.16653%2C2305.19308%2C2306.00245%2C2306.06070%2C2306.07863%2C2307.10088%2C2307.12856%2C2307.13854%2C2308.01990&max_results=50 (retrieved 2026-10-16T06:31:20Z; SHA-256 cbe8bfcbce4170a94a6cf42de01fbe0a448e7b1d29fa75a0de877105a98ec08e)