arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.11537cs.ROcs.AIcs.ET
收藏

VR-GPT:面向智能虚拟现实应用的视觉语言模型

VR-GPT: Visual Language Model for Intelligent Virtual Reality Applications

Skolkovo Institute of Science and Technology (Skoltech)(斯科尔科沃科学技术研究院(斯科尔泰克))

Mikhail Konenkov, Artem Lykov, Daria Trinitatova, Dzmitry Tsetserukou

AI总结:

本文提出 VR-GPT,将定制 VLM 与 Unity 及语音交互结合,用于智能 VR 应用,实验表明其可提升任务效率、舒适度与参与度。

摘要:

沉浸式虚拟现实应用的出现已经改变了多个领域,但它们与视觉语言模型等先进人工智能技术的融合仍未得到充分探索。本研究提出了一种开创性方法,在 VR 环境中利用 VLM 来增强用户交互并提高任务效率。借助 Unity 引擎和定制开发的 VLM,我们的系统通过自然语言处理实现实时、直观的用户交互,而无需依赖可视化文本指令。语音转文本和文本转语音技术的引入,使用户与 VLM 之间能够进行无缝通信,使系统能够有效引导用户完成复杂任务。初步实验结果表明,与传统 VR 交互方法相比,使用 VLM 不仅缩短了任务完成时间,还提高了用户舒适度和任务参与度。

英文摘要:

The advent of immersive Virtual Reality applications has transformed various domains, yet their integration with advanced artificial intelligence technologies like Visual Language Models remains underexplored. This study introduces a pioneering approach utilizing VLMs within VR environments to enhance user interaction and task efficiency. Leveraging the Unity engine and a custom-developed VLM, our system facilitates real-time, intuitive user interactions through natural language processing, without relying on visual text instructions. The incorporation of speech-to-text and text-to-speech technologies allows for seamless communication between the user and the VLM, enabling the system to guide users through complex tasks effectively. Preliminary experimental results indicate that utilizing VLMs not only reduces task completion times but also improves user comfort and task engagement compared to traditional VR interaction methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑