面向机器人操作的以对象为中心的指令增强方法
Object-Centric Instruction Augmentation for Robotic Manipulation
AI总结：
针对机器人操作中位置信息利用不足的问题，提出OCI框架，通过MLLM为指令融入物体位置线索，并复用预训练MLLM特征，提升模仿策略性能。
摘要：
人类通过识别观察中物体的身份和位置来理解场景。对于执行“拾取放置”这类任务的机器人而言，理解物体是什么以及位于何处至关重要。现有研究已大量探讨利用大语言模型丰富文本描述来解决前者，但后者的研究仍不充分。本研究提出Object-Centric Instruction Augmentation（OCI，以对象为中心的指令增强）框架，为高语义、信息密集的语言指令补充位置线索。我们利用多模态大语言模型（MLLM）将物体位置知识融入自然语言指令，从而帮助策略网络掌握多样化操作动作。此外，我们提出一种特征复用机制，将现成预训练MLLM的视觉-语言特征整合到策略网络中。通过一系列模拟和真实世界机器人任务，我们证明，使用增强后指令训练的机器人机械臂模仿策略，性能优于仅依赖传统语言指令的策略。
英文摘要：
Humans interpret scenes by recognizing both the identities and positions of objects in their observations. For a robot to perform tasks such as \enquote{pick and place}, understanding both what the objects are and where they are located is crucial. While the former has been extensively discussed in the literature that uses the large language model to enrich the text descriptions, the latter remains underexplored. In this work, we introduce the \textit{Object-Centric Instruction Augmentation (OCI)} framework to augment highly semantic and information-dense language instruction with position cues. We utilize a Multi-modal Large Language Model (MLLM) to weave knowledge of object locations into natural language instruction, thus aiding the policy network in mastering actions for versatile manipulation. Additionally, we present a feature reuse mechanism to integrate the vision-language features from off-the-shelf pre-trained MLLM into policy networks. Through a series of simulated and real-world robotic tasks, we demonstrate that robotic manipulator imitation policies trained with our enhanced instructions outperform those relying solely on traditional language instructions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
