arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.12836cs.HC
收藏

SituationAdapt:基于LLM推理的情境感知混合现实上下文UI优化

SituationAdapt: Contextual UI Optimization in Mixed Reality with Situation Awareness via LLM Reasoning

Zhipeng Li, Christoph Gebhardt, Yves Inglin, Nicolas Steck, Paul Streli, Christian Holz

AI总结:

SituationAdapt利用视觉-语言模型推理,结合感知与优化模块,在移动混合现实中动态调整UI布局,避免遮挡环境线索并尊重社会规范,实验证明其优于现有自适应方法。

摘要:

混合现实越来越多地用于受控家庭和办公空间之外的移动场景。这种移动性带来了对能够适应不同情境的用户界面布局的需求。然而,现有的自适应系统仅针对静态环境设计。在本文中,我们介绍了SituationAdapt,一个通过考虑共享环境中的环境和社会线索来调整混合现实用户界面以适应真实世界环境的系统。我们的系统由感知、推理和优化模块组成,用于UI自适应。我们的感知模块识别用户周围的物体和个人,而我们的推理模块利用视觉-语言模型来评估交互式UI元素的放置位置。这确保了调整后的布局不会遮挡相关的环境线索或干扰社会规范。然后,我们的优化模块生成考虑这些因素以及时间约束的混合现实界面。为了评估,我们首先验证了我们的推理模块在评估UI上下文方面与人类专家用户相比的能力。在一项在线用户研究中,我们随后确立了SituationAdapt为混合现实生成情境感知布局的能力,它优于之前的自适应布局方法。最后,我们通过一系列应用和场景来展示SituationAdapt的多功能性。

英文摘要:

Mixed Reality is increasingly used in mobile settings beyond controlled home and office spaces. This mobility introduces the need for user interface layouts that adapt to varying contexts. However, existing adaptive systems are designed only for static environments. In this paper, we introduce SituationAdapt, a system that adjusts Mixed Reality UIs to real-world surroundings by considering environmental and social cues in shared settings. Our system consists of perception, reasoning, and optimization modules for UI adaptation. Our perception module identifies objects and individuals around the user, while our reasoning module leverages a Vision-and-Language Model to assess the placement of interactive UI elements. This ensures that adapted layouts do not obstruct relevant environmental cues or interfere with social norms. Our optimization module then generates Mixed Reality interfaces that account for these considerations as well as temporal constraints. For evaluation, we first validate our reasoning module's capability of assessing UI contexts in comparison to human expert users. In an online user study, we then establish SituationAdapt's capability of producing context-aware layouts for Mixed Reality, where it outperformed previous adaptive layout methods. We conclude with a series of applications and scenarios to demonstrate SituationAdapt's versatility.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑