arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.09676cs.ROcs.AI
收藏

通过预训练和多任务微调掌握多模态提示的机器人操作

Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning

University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);Amazon AGI(亚马逊AGI);University of California, Santa Cruz(加利福尼亚大学圣克鲁兹分校)

Jiachen Li, Qiaozi Gao, Michael Johnston, Xiaofeng Gao, Xuehai He, Suhaila Shakiah, Hangjie Shi, Reza Ghanadan, William Yang Wang

AI总结:

提出一种两阶段训练框架,通过逆动力学预训练和多任务微调,使机器人理解多模态提示并执行操作,在VIMA-BENCH上取得10%成功率提升,并展现上下文学习能力。

摘要:

基于提示的学习已被证明是一种引人注目的范式,为大型语言模型(LLMs)的巨大成功做出了贡献。受其在语言任务中成功的启发,现有研究已将LLMs用于具身指令跟随和任务规划。在这项工作中,我们解决了训练机器人理解多模态提示的问题,这些提示将视觉信号与文本描述交织在一起。这类任务对机器人理解视觉和语言信号之间相互联系和互补性的能力构成了重大挑战。在这项工作中,我们引入了一个有效的框架,该框架从多任务专家轨迹中学习执行机器人操作策略,以应对多模态提示。我们的方法包括一个两阶段训练流程,执行逆动力学预训练和多任务微调。为了促进多模态理解,我们设计了多模态提示编码器,通过将预训练的语言模型与视觉输入的残差连接增强,并建模动作维度之间的依赖关系。在实验上,我们在VIMA-BENCH上评估了我们的方法的有效性,并建立了新的最先进水平(成功率提高10%)。此外,我们证明了我们的模型表现出显著的上下文学习能力。项目页面:\url{https://midas-icml.github.io/}。

英文摘要:

Prompt-based learning has been demonstrated as a compelling paradigm contributing to large language models' tremendous success (LLMs). Inspired by their success in language tasks, existing research has leveraged LLMs in embodied instruction following and task planning. In this work, we tackle the problem of training a robot to understand multimodal prompts, interleaving vision signals with text descriptions. This type of task poses a major challenge to robots' capability to understand the interconnection and complementarity between vision and language signals. In this work, we introduce an effective framework that learns a policy to perform robot manipulation with multimodal prompts from multi-task expert trajectories. Our methods consist of a two-stage training pipeline that performs inverse dynamics pretraining and multi-task finetuning. To facilitate multimodal understanding, we design our multimodal prompt encoder by augmenting a pretrained LM with a residual connection to the visual input and model the dependencies among action dimensions. Empirically, we evaluate the efficacy of our method on the VIMA-BENCH and establish a new state-of-the-art (10% improvement in success rate). Moreover, we demonstrate that our model exhibits remarkable in-context learning ability. Project page: \url{https://midas-icml.github.io/}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑