Surfer：基于世界模型的机器人操作渐进式推理
Surfer: Progressive Reasoning with World Models for Robotic Manipulation
AI总结：
本文提出基于世界模型的机器人操作框架 Surfer，通过解耦动作与场景并显式建模多模态信息以提升泛化能力，并构建基于 MuJoCo 的模拟器与 SeaWave 基准，在四级渐进推理任务上达到 54.74% 成功率，超越最佳基线。
摘要：
考虑如何使模型准确理解并遵循自然语言指令，并执行与世界知识相一致的动作，是机器人操作中的一个关键挑战。这主要包括人类模糊指令推理和物理知识的遵循。因此，具身智能体必须具备从训练数据中建模世界知识的能力。然而，现有的大多数视觉和语言机器人操作方法主要在不够真实的模拟器和语言设置下运行，缺乏对世界知识的显式建模。为了弥补这一差距，我们引入了一种新颖且简单的机器人操作框架，称为 Surfer。它基于世界模型，将机器人操作视为视觉场景的状态转移，并将其解耦为两部分：动作和场景。然后，通过对多模态信息中的动作和场景预测进行显式建模，增强了模型在新指令和新场景上的泛化能力。除了该框架，我们还构建了一个基于 MuJoCo 物理引擎、支持全物理执行的机器人操作模拟器。它可以自动生成演示训练数据和测试数据，有效降低人工成本。为了在语言理解和物理执行方面对机器人操作模型进行全面且系统的评估，我们还创建了一个具有渐进式推理任务的机器人操作基准，称为 SeaWave。它包含 4 个级别的渐进式推理任务，可为多模态环境中的具身 AI 智能体提供标准化测试平台。平均而言，Surfer 在定义的四个级别的操作任务上达到了 54.74% 的成功率，超过了 47.64% 的最佳基线性能。
英文摘要：
Considering how to make the model accurately understand and follow natural language instructions and perform actions consistent with world knowledge is a key challenge in robot manipulation. This mainly includes human fuzzy instruction reasoning and the following of physical knowledge. Therefore, the embodied intelligence agent must have the ability to model world knowledge from training data. However, most existing vision and language robot manipulation methods mainly operate in less realistic simulator and language settings and lack explicit modeling of world knowledge. To bridge this gap, we introduce a novel and simple robot manipulation framework, called Surfer. It is based on the world model, treats robot manipulation as a state transfer of the visual scene, and decouples it into two parts: action and scene. Then, the generalization ability of the model on new instructions and new scenes is enhanced by explicit modeling of the action and scene prediction in multi-modal information. In addition to the framework, we also built a robot manipulation simulator that supports full physics execution based on the MuJoCo physics engine. It can automatically generate demonstration training data and test data, effectively reducing labor costs. To conduct a comprehensive and systematic evaluation of the robot manipulation model in terms of language understanding and physical execution, we also created a robotic manipulation benchmark with progressive reasoning tasks, called SeaWave. It contains 4 levels of progressive reasoning tasks and can provide a standardized testing platform for embedded AI agents in multi-modal environments. On average, Surfer achieved a success rate of 54.74% on the defined four levels of manipulation tasks, exceeding the best baseline performance of 47.64%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
