arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.15083cs.HCcs.AIcs.CL
收藏

免提VR

Hands-Free VR

Institute of Problem Solving, XYZ University(XYZ大学问题解决研究所);Main University(梅因大学)

Jorge Askur Vazquez Fernandez, Jae Joong Lee, Santiago Andrés Serrano Vacca, Alejandra Magana, Radim Pesam, Bedrich Benes, Voicu Popescu

摘要:

本文介绍了免提VR,一个基于语音的自然语言界面的VR。用户使用他们的语音给出命令,使用语音到文本深度学习模型将语音音频数据转换为文本,该语音到文本深度学习模型针对单词语音相似性和口语英语口音的鲁棒性进行了微调,并且使用对自然语言多样性鲁棒的大语言模型将文本映射到可执行VR命令。在一项受试者内对照研究(N = 22)中对免提VR进行了评估,该研究要求参与者找到特定的物体并将其放置在各种配置中。在控制条件下,参与者使用传统的VR用户界面来抓取,携带和使用手持控制器定位对象。在实验条件下,参与者使用免提VR。结果证实:(1)Hands-Free VR对英语口语口音具有鲁棒性,因为我们的20名参与者英语不是他们的第一语言,并且单词语音相似性,正确转录语音命令的时间为96.71%;(2)Hands-Free VR对自然语言多样性具有鲁棒性,在97.83%的时间内正确将转录的命令映射为可执行命令;(3)在任务完成时间、总视角平移、总视角方向旋转和总左右手平移方面,免提VR比传统VR界面具有显著的效率优势;(4)在易用性、直观性、人体工程学、可靠性和可取性方面,免提VR获得了较高的用户偏好评级。

英文摘要:

The paper introduces Hands-Free VR, a voice-based natural-language interface for VR. The user gives a command using their voice, the speech audio data is converted to text using a speech-to-text deep learning model that is fine-tuned for robustness to word phonetic similarity and to spoken English accents, and the text is mapped to an executable VR command using a large language model that is robust to natural language diversity. Hands-Free VR was evaluated in a controlled within-subjects study (N = 22) that asked participants to find specific objects and to place them in various configurations. In the control condition participants used a conventional VR user interface to grab, carry, and position the objects using the handheld controllers. In the experimental condition participants used Hands-Free VR. The results confirm that: (1) Hands-Free VR is robust to spoken English accents, as for 20 of our participants English was not their first language, and to word phonetic similarity, correctly transcribing the voice command 96.71% of the time; (2) Hands-Free VR is robust to natural language diversity, correctly mapping the transcribed command to an executable command in 97.83% of the time; (3) Hands-Free VR had a significant efficiency advantage over the conventional VR interface in terms of task completion time, total viewpoint translation, total view direction rotation, and total left and right hand translations; (4) Hands-Free VR received high user preference ratings in terms of ease of use, intuitiveness, ergonomics, reliability, and desirability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑