通过预测交互学习操作
Learning Manipulation by Predicting Interaction
浏览论文内容
中文总结 AI 辅助
针对机器人操作表征学习忽略交互动态的问题，提出通用预训练流程MPI，通过预测过渡帧和检测交互对象增强视觉表征，在真实与仿真任务中较SOTA提升10%至64%。
中文摘要 AI 辅助
近年来，用于机器人操作（robotic manipulation）的表征学习方法大量涌现。由于领域内机器人数据稀缺，主流方法倾向于利用大规模人类视频数据集，为视觉运动策略（visuomotor policy）学习提取可泛化的特征。尽管取得了一定进展，以往的研究忽略了捕捉操作过程中行为模式和物理交互的交互动态（interactive dynamics），导致对物体与环境之间关系的理解不够充分。为此，我们提出了一种通用的预训练流程，通过预测交互学习操作（MPI），以增强视觉表征。给定代表初始和最终状态的一对关键帧以及语言指令，我们的算法分别预测过渡帧并检测交互对象。这两个学习目标实现了对“如何交互”和“在哪交互”的深入理解。我们对几个具有挑战性的机器人任务进行了全面评估。实验结果表明，无论是在真实世界机器人平台还是仿真环境中，与之前的 state-of-the-art 相比，MPI 都展现出显著的性能提升，幅度从 10% 到 64% 不等。代码和模型检查点已在 https://github.com/OpenDriveLab/MPI 公开分享。
英文摘要
Representation learning approaches for robotic manipulation have boomed in recent years. Due to the scarcity of in-domain robot data, prevailing methodologies tend to leverage large-scale human video datasets to extract generalizable features for visuomotor policy learning. Despite the progress achieved, prior endeavors disregard the interactive dynamics that capture behavior patterns and physical interaction during the manipulation process, resulting in an inadequate understanding of the relationship between objects and the environment. To this end, we propose a general pre-training pipeline that learns Manipulation by Predicting the Interaction (MPI) and enhances the visual representation.Given a pair of keyframes representing the initial and final states, along with language instructions, our algorithm predicts the transition frame and detects the interaction object, respectively. These two learning objectives achieve superior comprehension towards "how-to-interact" and "where-to-interact". We conduct a comprehensive evaluation of several challenging robotic tasks.The experimental results demonstrate that MPI exhibits remarkable improvement by 10% to 64% compared with previous state-of-the-art in real-world robot platforms as well as simulation environments. Code and checkpoints are publicly shared at https://github.com/OpenDriveLab/MPI.
发表机构
- Shanghai AI Lab（上海人工智能实验室）
- Shanghai Jiao Tong University（上海交通大学）
- Renmin University of China（中国人民大学）
- Peking University（北京大学）
- Northwestern Polytechnical University（西北工业大学）
- TeleAI, China Telecom Corp Ltd（中国电信集团有限公司 TeleAI）
机构由 AI 辅助整理，请以论文原文为准。
