The paper introduces Hands-Free VR, a voice-based natural-language interface for VR. The user gives a command using their voice, the speech audio data is converted to text using a speech-to-text deep learning model that is fine-tuned for robustness to word phonetic similarity and to spoken English accents, and the text is mapped to an executable VR command using a large language model that is robust to natural language diversity. Hands-Free VR was evaluated in a controlled within-subjects study (N = 22) that asked participants to find specific objects and to place them in various configurations. In the control condition participants used a conventional VR user interface to grab, carry, and position the objects using the handheld controllers. In the experimental condition participants used Hands-Free VR. The results confirm that: (1) Hands-Free VR is robust to spoken English accents, as for 20 of our participants English was not their first language, and to word phonetic similarity, correctly transcribing the voice command 96.71% of the time; (2) Hands-Free VR is robust to natural language diversity, correctly mapping the transcribed command to an executable command in 97.83% of the time; (3) Hands-Free VR had a significant efficiency advantage over the conventional VR interface in terms of task completion time, total viewpoint translation, total view direction rotation, and total left and right hand translations; (4) Hands-Free VR received high user preference ratings in terms of ease of use, intuitiveness, ergonomics, reliability, and desirability.
Tactical Decision for Multi-UGV Confrontation with a Vision-Language Model-Based Commander
Yangtze Delta Region Academy(扬子江地区学院);Beijing Institute of Technology(北京理工大学);School of Automation(自动化学院);Beihang University(北航大学);Advanced Research Institute of Multidisciplinary Sciences(多学科科学高级研究机构)
HiSync: Spatio-Temporally Aligning Hand Motion from Wearable IMU and On-Robot Camera for Command Source Identification in Long-Range HRI
Department of Computer Science;Technology, BNRist Tsinghua University Beijing China;Technology, BNRist, College of AI Tsinghua University Beijing China;Technology Tsinghua University Beijing China;Beijing University of Posts;Academy of Arts \& Design Tsinghua University Beijing China;Qinghai University Xining China;Tsinghua University;Technology, BNRist, College of AI;Technology Tsinghua University;Academy of Arts \& Design Tsinghua University;Qinghai University
Speech recognition assisted by large language models to command software orally -- Application to an augmented and virtual reality web app for immersive molecular graphics