RoboOmni：全模态上下文中的主动机器人操作
RoboOmni: Proactive Robot Manipulation in Omni-modal Context
AI总结：
提出跨模态上下文指令设定及RoboOmni框架，融合视听信号实现主动意图识别与机器人操作，并构建OmniAction数据集，在多指标上超越基线。
摘要：
多模态大型语言模型（MLLMs）的最新进展推动了用于机器人操作的视觉-语言-动作（VLA）模型的快速发展。尽管在许多场景中很有效，但当前的方法主要依赖显式指令，而在现实世界的交互中，人类很少直接发出指令。有效的协作要求机器人主动推断用户意图。在这项工作中，我们引入了跨模态上下文指令，这是一种新设定，其中意图源自口头对话、环境声音和视觉线索，而不是显式命令。为了解决这一新设定，我们提出了RoboOmni，这是一个基于端到端全模态LLMs的感知者-思考者-对话者-执行者框架，统一了意图识别、交互确认和动作执行。RoboOmni在时空上融合听觉和视觉信号以实现稳健的意图识别，同时支持直接语音交互。为了解决机器人操作中主动意图识别训练数据缺失的问题，我们构建了OmniAction，包含140k个回合、5k+说话者、2.4k个事件声音、640个背景和六种上下文指令类型。在仿真和真实世界设置中的实验表明，RoboOmni在成功率、推理速度、意图识别和主动协助方面超越了基于文本和ASR的基线。
英文摘要：
Recent advances in Multimodal Large Language Models (MLLMs) have driven rapid progress in Vision-Language-Action (VLA) models for robotic manipulation. Although effective in many scenarios, current approaches largely rely on explicit instructions, whereas in real-world interactions, humans rarely issue instructions directly. Effective collaboration requires robots to infer user intentions proactively. In this work, we introduce cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands. To address this new setting, we present RoboOmni, a Perceiver-Thinker-Talker-Executor framework based on end-to-end omni-modal LLMs that unifies intention recognition, interaction confirmation, and action execution. RoboOmni fuses auditory and visual signals spatiotemporally for robust intention recognition, while supporting direct speech interaction. To address the absence of training data for proactive intention recognition in robotic manipulation, we build OmniAction, comprising 140k episodes, 5k+ speakers, 2.4k event sounds, 640 backgrounds, and six contextual instruction types. Experiments in simulation and real-world settings show that RoboOmni surpasses text- and ASR-based baselines in success rate, inference speed, intention recognition, and proactive assistance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
