arXiv 2504.173312025-04cs.HCcs.AI

探索用于沉浸式虚拟现实的上下文感知和大语言模型驱动的移动

Exploring Context-aware and LLM-driven Locomotion for Immersive Virtual Reality

Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Human-Centered Technologies for Learning（以人为本的学习技术）

Suleyman Ozdel, Kadir Burak Buldu, Enkelejda Kasneci, Efe Bozkir

备注：: 28 pages. To appear in the Proceedings of the ACM on Human-Computer Interaction (PACM HCI), Vol. 10, No. 5; presented at the 28th ACM International Conference on Mobile Human-Computer Interaction (MobileHCI 2026)
DOI：: 10.1145/3821669

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

研究提出由大语言模型驱动的移动技术，让用户用自然语言在虚拟环境中导航。通过评估三种移动方法，结合眼动追踪数据分析和标准化问卷，发现该方法有潜力成为可行的免手持替代方案，能促进虚拟空间免手持移动。

摘要：

移动在塑造虚拟现实环境中的用户体验方面起着至关重要的作用。特别是，免手持移动通过支持无障碍访问并使用户摆脱对手持控制器的依赖，提供了一种有价值的替代方案。为此，传统的基于语音的方法通常依赖于僵化的命令集，限制了交互的自然性和灵活性。在本研究中，我们提出了一种由大语言模型（LLMs）驱动的新型移动技术，该技术允许用户在具有上下文感知的情况下使用自然语言在虚拟环境中导航。我们评估了三种移动方法：基于控制器的瞬移、基于语音的转向和我们的语言模型驱动方法。我们的评估结合了眼动追踪数据分析，包括使用SHAP进行探索性可解释机器学习分析，以及标准化问卷（SUS、IPQ、CSQ-VR、NASA-TLX），以通过基于客观注视的测量和关于可用性、临场感、网络晕动症和认知负荷的主观自我报告来检查用户体验。我们的研究结果表明，在可用性、临场感或网络晕动症方面，大语言模型驱动的移动与瞬移等既定方法之间没有统计学上的显著差异，这表明其作为一种可行的、基于自然语言的免手持替代方案的潜力。此外，眼动追踪分析揭示了一些模式，表明在大语言模型驱动的条件下，用户的注意力和参与度有增加的趋势。作为这些发现的补充，探索性SHAP分析表明，注视、扫视和瞳孔相关特征在不同技术之间有所不同，表明视觉注意力和认知处理的不同模式。总体而言，我们指出我们的方法可以促进虚拟空间中的免手持移动，特别是在支持无障碍访问方面。

英文摘要：

Locomotion plays a crucial role in shaping the user experience within virtual reality environments. In particular, hands-free locomotion offers a valuable alternative by supporting accessibility and freeing users from reliance on handheld controllers. To this end, traditional speech-based methods often depend on rigid command sets, limiting the naturalness and flexibility of interaction. In this study, we propose a novel locomotion technique powered by large language models (LLMs), which allows users to navigate virtual environments using natural language with contextual awareness. We evaluate three locomotion methods: controller-based teleportation, voice-based steering, and our language model-driven approach. Our evaluation combines eye-tracking data analysis, including exploratory explainable machine learning analysis with SHAP, and standardized questionnaires (SUS, IPQ, CSQ-VR, NASA-TLX) to examine user experience through both objective gaze-based measures and subjective self-reports of usability, presence, cybersickness, and cognitive load. Our findings show no statistically significant differences in usability, presence, or cybersickness between LLM-driven locomotion and established methods such as teleportation, suggesting its potential as a viable, natural language-based, hands-free alternative. In addition, eye-tracking analysis revealed patterns suggesting tendency toward increased user attention and engagement in the LLM-driven condition. Complementary to these findings, exploratory SHAP analysis revealed that fixation, saccade, and pupil-related features vary across techniques, indicating distinct patterns of visual attention and cognitive processing. Overall, we state that our method can facilitate hands-free locomotion in virtual spaces, especially in supporting accessibility.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文