面向基于视觉的空间智能体的接地式GUI理解：以扩展现实应用为例
Grounded GUI Understanding for Vision-Based Spatial Intelligent Agent: Exemplified by Extended Reality Apps
AI总结：
针对XR应用可交互GUI元素检测的现有方法适配难题，提出零样本上下文相关检测框架Orienter，经语义理解、候选检测、可交互性分类及反馈循环实现检测，效果优于现有方法。
摘要：
近年来，空间计算又称扩展现实（XR）已成为一项变革性技术，为用户在多样化虚拟环境中提供沉浸式交互体验。用户可通过立体三维（3D）图形用户界面（GUI）上的可交互GUI元素（IGE）与XR应用交互。准确识别这些IGE至关重要，是自动化测试、高效GUI搜索等诸多软件工程任务的基础。当前针对2D移动应用的IGE检测方法通常基于大规模人工标注的GUI数据集训练监督式目标检测模型，预定义按钮、下拉框等可点击GUI元素类别。这类方法难以应用于XR应用的IGE检测，面临多重挑战：开放词汇与异构IGE类别带来的复杂性、上下文相关可交互性的错综性，以及实现精准IGE检测所需的精确空间感知与视觉-语义对齐需求。因此，有必要开展针对XR应用的IGE研究。本文提出首个面向虚拟现实应用的零样本上下文相关可交互GUI元素检测框架，命名为Orienter。通过模仿人类行为，Orienter先观察并理解XR应用场景的语义上下文，再执行检测，检测过程在反馈驱动的验证与反思循环中迭代。具体而言，Orienter包含三个组件：（1）语义上下文理解，（2）反思驱动的IGE候选检测，（3）上下文相关可交互性分类。大量实验表明，Orienter比现有最先进的GUI元素检测方法更有效。
英文摘要：
In recent years, spatial computing a.k.a. Extended Reality (XR) has emerged as a transformative technology, offering users immersive and interactive experiences across diversified virtual environments. Users can interact with XR apps through interactable GUI elements (IGEs) on the stereoscopic three-dimensional (3D) graphical user interface (GUI). The accurate recognition of these IGEs is instrumental, serving as the foundation of many software engineering tasks, including automated testing and effective GUI search. The most recent IGE detection approaches for 2D mobile apps typically train a supervised object detection model based on a large-scale manually-labeled GUI dataset, usually with a pre-defined set of clickable GUI element categories like buttons and spinners. Such approaches can hardly be applied to IGE detection in XR apps, due to a multitude of challenges including complexities posed by open-vocabulary and heterogeneous IGE categories, intricacies of context-sensitive interactability, and the necessities of precise spatial perception and visual-semantic alignment for accurate IGE detection results. Thus, it is necessary to embark on the IGE research tailored to XR apps. In this paper, we propose the first zero-shot cOntext-sensitive inteRactable GUI ElemeNT dEtection framework for virtual Reality apps, named Orienter. By imitating human behaviors, Orienter observes and understands the semantic contexts of XR app scenes first, before performing the detection. The detection process is iterated within a feedback-directed validation and reflection loop. Specifically, Orienter contains three components, including (1) Semantic context comprehension, (2) Reflection-directed IGE candidate detection, and (3) Context-sensitive interactability classification. Extensive experiments demonstrate that Orienter is more effective than the state-of-the-art GUI element detection approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
