通过虚拟现实和大语言模型辅助的人形机器人实现沉浸式社交互动
Immersive Social Interaction with VR and LLM-Assisted Humanoids
AI总结：
研究通过集成语音控制移动、VR操作和双向社交互动的框架实现人形机器人全身控制，利用苹果视觉专业版等，经大语言模型辅助语音控制等技术，在宇树H1人形机器人上评估，新手操作成功率可观，证明该方式在多方面应用的潜力。
摘要：
人形机器人可将人类存在扩展到远程、受限或危险环境，但现有遥操作界面往往要求高体力的运动跟踪或高认知水平的低级控制。本文提出了一个沉浸式遥操作框架，集成语音控制的移动、基于虚拟现实的操作和双向社交互动以进行全身人形机器人控制。利用苹果视觉专业版，操作员接收以自我为中心的视觉反馈，发出自然语言移动命令，并通过手腕和手指跟踪遥操作机器人的手臂和灵巧的手。一个大语言模型辅助的语音控制模块将语音指令转换为高级移动命令，而操作模块通过逆运动学和比例-微分控制将人类手部动作重新定位到机器人上。该系统还记录多模态数据，包括以自我为中心的RGB观察、语音/文本命令、关节状态、手部动作和目光信号，以支持未来的模仿学习和自主性。我们在配备灵巧手的宇树H1人形机器人上对该框架进行操作和社交互动任务评估。结果表明，新手用户在短暂熟悉后能够成功操作该系统，在物体操作任务中成功率达80%，在社交传方块任务中成功率达70%。这些结果证明了沉浸式、语言辅助遥操作作为人形机器人交互、远程协助和多模态数据收集的可访问接口的潜力。
英文摘要：
Humanoid robots can extend human presence to remote, constrained, or hazardous environments, but existing teleoperation interfaces often require physically demanding motion tracking or cognitively demanding low-level control. This paper presents an immersive teleoperation framework that integrates voice-controlled locomotion, VR-based manipulation, and bidirectional social interaction for whole-body humanoid control. Using Apple Vision Pro, the operator receives egocentric visual feedback, issues natural-language locomotion commands, and teleoperates the robot's arms and dexterous hands through wrist and finger tracking. An LLM-assisted voice-control module converts spoken instructions into high-level locomotion commands, while the manipulation module retargets human hand motions to the robot through inverse kinematics and PD control. The system also records multimodal data, including egocentric RGB observations, voice/text commands, joint states, hand motions, and eye-gaze signals, supporting future imitation learning and autonomy. We evaluate the framework on a Unitree H1 humanoid equipped with dexterous hands in manipulation and social interaction tasks. Results show that novice users can successfully operate the system after brief familiarization, achieving 80\% success in object manipulation and 70\% success in a social cube-passing task. These results demonstrate the potential of immersive, language-assisted teleoperation as an accessible interface for humanoid interaction, remote assistance, and multimodal data collection.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
