# Grounded GUI Understanding for Vision-Based Spatial Intelligent Agent: Exemplified by Extended Reality Apps

Orienter detects interactable GUI elements in XR applications by interpreting scene context and revisiting candidate elements through a validation and reflection loop. It addresses open-vocabulary element types and context-dependent interactability.

- Catalog URL: https://awesomecua.github.io/resources/arxiv-2409-10811/
- Original source: https://arxiv.org/abs/2409.10811
- Kind: paper
- Review: curated
- Scope: core
- Environments: unspecified
- Topics: screen-understanding, grounding
- Evidence checked: 2026-10-18T04:21:52Z
- ID: arxiv:2409.10811

## Limitations

Reviewed against the complete official arXiv abstract. Experimental results and linked artifact availability have not been independently reproduced. The described environment is XR/VR applications; the operating system is not specified in the abstract.

- Record JSON: https://awesomecua.github.io/resources/arxiv-2409-10811/record.json

## Authors

Shuqing Li, Binchang Li, Yepang Liu, Cuiyun Gao, Jianping Zhang, Shing-Chi Cheung, Michael R. Lyu

## Citation exports

- BibTeX: https://awesomecua.github.io/resources/arxiv-2409-10811/citation.bib
- CSL JSON: https://awesomecua.github.io/resources/arxiv-2409-10811/citation.csl.json

## Recorded arXiv snapshot

- Version: 4
- Version source: https://arxiv.org/abs/2409.10811v4
- First submission: 2024-09-17T00:58:00Z
- Version updated: 2025-10-01T17:58:44Z
- Abstract SHA-256: 2a32462e4cb09763750f8f22d7f4be32a6bdfe3db6c9b3de9c386dc7cd6b6f30
- Abstract hash normalization: sha256-utf8-whitespace-collapsed-v1

## License declarations

- code: not recorded
- data: not recorded
- document: not recorded
- weights: not recorded

## Primary evidence

- https://export.arxiv.org/api/query?id_list=2210.03347%2C2403.04984%2C2405.07090%2C2406.06282%2C2409.00088%2C2409.10811%2C2411.10943%2C2503.09241%2C2503.20279%2C2503.23350%2C2505.05440%2C2505.10593%2C2505.16422%2C2505.21279%2C2505.24878%2C2506.02153%2C2506.02839%2C2506.05079%2C2506.07976%2C2508.02046%2C2511.07885%2C2511.11248%2C2512.21720%2C2603.20351%2C2604.19299&max_results=25 (retrieved 2026-10-18T04:21:52Z; SHA-256 c194d15b7e422aae48b4e05216a1f1d84471b6666718b9d3dd4c9d250c356e77)
- https://export.arxiv.org/api/query?id_list=2210.03347%2C2403.04984%2C2405.07090%2C2406.06282%2C2409.00088%2C2409.10811%2C2411.10943%2C2503.09241%2C2503.20279%2C2503.23350%2C2505.05440%2C2505.10593%2C2505.16422%2C2505.21279%2C2505.24878%2C2506.02153%2C2506.02839%2C2506.05079%2C2506.07976%2C2508.02046%2C2511.07885%2C2511.11248%2C2512.21720%2C2603.20351%2C2604.19299&max_results=25 (retrieved 2026-10-18T04:21:52Z; SHA-256 c194d15b7e422aae48b4e05216a1f1d84471b6666718b9d3dd4c9d250c356e77)