arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.21091cs.HCcs.AI
收藏

大语言模型辅助的语音与指向有益于虚拟现实中的多3D对象选择

Large Language Model-assisted Speech and Pointing Benefits Multiple 3D Object Selection in Virtual Reality

University of Cambridge(剑桥大学);Coburg University of Applied Sciences and Arts(科堡应用科学与艺术大学)

Junlong Chen, Jens Grubert, Per Ola Kristensson

AI总结:

本文提出大语言模型辅助的多模态语音与射线投射技术 AssistVR,用于虚拟现实多3D对象选择,并通过24人用户研究证明其优于基于小地图的基线。

摘要:

在虚拟现实中,选择被遮挡的对象是一个具有挑战性的问题,当涉及多个对象时更是如此。随着新型人工智能技术的出现,我们探索利用大语言模型,通过多模态语音和射线投射交互技术来辅助虚拟现实中多对象选择任务的可能性。我们在一项对比用户研究(n=24)中验证了这些发现,参与者在具有不同场景复杂度级别的虚拟现实场景中选择目标对象。研究将性能指标和用户体验指标与一种基于小地图的被遮挡对象选择技术进行比较,后者作为基线。结果表明,所提出的技术 AssistVR 在存在多个目标对象时优于基线技术。与对语音界面的普遍看法相反,即使目标对象难以用语言指称,AssistVR 仍能够优于基线。本工作证明了由大语言模型驱动的智能多模态交互系统的可行性和交互潜力。基于这些结果,我们讨论了研究结果对未来沉浸式环境中智能多模态交互系统设计的启示。

英文摘要:

Selection of occluded objects is a challenging problem in virtual reality, even more so if multiple objects are involved. With the advent of new artificial intelligence technologies, we explore the possibility of leveraging large language models to assist multi-object selection tasks in virtual reality via a multimodal speech and raycast interaction technique. We validate the findings in a comparative user study (n=24), where participants selected target objects in a virtual reality scene with different levels of scene perplexity. The performance metrics and user experience metrics are compared against a mini-map based occluded object selection technique that serves as the baseline. Results indicate that the introduced technique, AssistVR, outperforms the baseline technique when there are multiple target objects. Contrary to the common belief for speech interfaces, AssistVR was able to outperform the baseline even when the target objects were difficult to reference verbally. This work demonstrates the viability and interaction potential of an intelligent multimodal interactive system powered by large laguage models. Based on the results, we discuss the implications for design of future intelligent multimodal interactive systems in immersive environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑