人机协作中基于心智理论的务实具身口语指令跟随
Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mind
AI总结：
针对嘈杂语音下的人机协作指令跟随难题，提出基于心智理论的神经符号模型SIFToM，利用视觉语言模型进行务实推断，在模拟和真实环境中显著提升性能并接近人类水平。
摘要：
口语指令在智能体协作中无处不在。然而，在真实世界的人机协作中，由于说话者和环境的各种因素（如背景噪音或发音错误），遵循人类口语指令可能具有挑战性。当面对嘈杂的听觉输入时，人类可以利用具身环境中的协作上下文来解读嘈杂的口语指令，并采取务实的辅助行动。在本文中，我们提出了一种认知启发的神经符号模型——通过心智理论进行口语指令跟随（SIFToM），该模型利用视觉语言模型和基于模型的思维推断，使机器人能够在多种语音条件下务实地遵循人类指令。我们在模拟环境（VirtualHome）和真实世界的人机协作场景中，通过人类评估对SIFToM进行了测试。结果表明，SIFToM能够显著提升轻量级基础VLM（Gemini 2.5 Flash）的性能，超越最先进的VLM（Gemini 2.5 Pro），并在具有挑战性的口语指令跟随任务上接近人类水平的准确率。
英文摘要：
Spoken language instructions are ubiquitous in agent collaboration. However, in real-world human-robot collaboration, following human spoken instructions can be challenging due to various speaker and environmental factors, such as background noise or mispronunciation. When faced with noisy auditory inputs, humans can leverage the collaborative context in the embodied environment to interpret noisy spoken instructions and take pragmatic assistive actions. In this paper, we present a cognitively inspired neurosymbolic model, Spoken Instruction Following through Theory of Mind (SIFToM), which leverages a Vision-Language Model with model-based mental inference to enable robots to pragmatically follow human instructions under diverse speech conditions. We test SIFToM in both simulated environments (VirtualHome) and real-world human-robot collaborative settings with human evaluations. Results show that SIFToM can significantly improve the performance of a lightweight base VLM (Gemini 2.5 Flash), outperforming state-of-the-art VLMs (Gemini 2.5 Pro) and approaching human-level accuracy on challenging spoken instruction following tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
