arXiv 2311.013782024-02-05cs.ROcs.AIcs.LG

视觉语言基础模型作为有效的机器人模仿者

Vision-Language Foundation Models as Effective Robot Imitators

ByteDance Research（字节跳动研究院）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；National University of Singapore（新加坡国立大学）

Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, Hang Li, Tao Kong

备注：: Fix typos. Project page: https://roboflamingo.github.io

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出RoboFlamingo框架，基于开源视觉语言模型OpenFlamingo微调，结合显式策略头处理序列历史，通过模仿学习实现语言条件机器人操作，在基准上大幅超越现有方法。

摘要：

近期视觉语言基础模型的进展表明，它们能够理解多模态数据并解决复杂的视觉语言任务，包括机器人操作。我们寻求一种直接的方法，利用现有的视觉语言模型（VLMs），在机器人数据上进行简单微调。为此，我们提出一个简单而新颖的视觉语言操作框架，称为RoboFlamingo，它基于开源VLM OpenFlamingo构建。与先前工作不同，RoboFlamingo利用预训练的VLM进行单步视觉语言理解，通过显式策略头建模序列历史信息，并仅通过模仿学习在语言条件操作数据集上进行轻微微调。这种分解使RoboFlamingo具备开环控制的灵活性，并能在低性能平台上部署。通过在测试基准上大幅超越现有技术水平，我们展示了RoboFlamingo可以成为将VLM适配到机器人控制的有效且有竞争力的替代方案。我们广泛的实验结果还揭示了不同预训练VLM在操作任务上行为的几个有趣结论。我们相信RoboFlamingo有潜力成为机器人操作的一种经济高效且易于使用的解决方案，使每个人都能微调自己的机器人策略。

英文摘要：

Recent progress in vision language foundation models has shown their ability to understand multimodal data and resolve complicated vision language tasks, including robotics manipulation. We seek a straightforward way of making use of existing vision-language models (VLMs) with simple fine-tuning on robotics data. To this end, we derive a simple and novel vision-language manipulation framework, dubbed RoboFlamingo, built upon the open-source VLMs, OpenFlamingo. Unlike prior works, RoboFlamingo utilizes pre-trained VLMs for single-step vision-language comprehension, models sequential history information with an explicit policy head, and is slightly fine-tuned by imitation learning only on language-conditioned manipulation datasets. Such a decomposition provides RoboFlamingo the flexibility for open-loop control and deployment on low-performance platforms. By exceeding the state-of-the-art performance with a large margin on the tested benchmark, we show RoboFlamingo can be an effective and competitive alternative to adapt VLMs to robot control. Our extensive experimental results also reveal several interesting conclusions regarding the behavior of different pre-trained VLMs on manipulation tasks. We believe RoboFlamingo has the potential to be a cost-effective and easy-to-use solution for robotics manipulation, empowering everyone with the ability to fine-tune their own robotics policy.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。