arXiv 2305.111762023-05-24cs.ROcs.AI

Instruct2Act：使用大语言模型将多模态指令映射为机器人动作

Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model

Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；University of Chinese Academy of Sciences（中国科学院大学）；The Chinese University of Hong Kong（香港中文大学）

Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, Hongsheng Li

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Instruct2Act框架，利用大语言模型生成Python程序，整合SAM和CLIP等基础模型实现感知、规划与动作循环，将多模态指令转换为机器人操作策略，在桌面任务中零样本表现优于现有学习方法。

摘要：

基础模型在文本到图像生成、全景分割和自然语言处理等各种应用中取得了显著进展。本文提出了Instruct2Act，这是一个利用大语言模型（LLM）将多模态指令映射为机器人操作任务中连续动作的框架。具体而言，Instruct2Act采用LLM模型生成Python程序，这些程序构成了机器人任务的感知、规划和动作循环。在感知部分，使用预定义API访问多个基础模型，其中Segment Anything Model（SAM）准确定位候选对象，CLIP对其进行分类。通过这种方式，该框架利用基础模型的专业知识和机器人能力，将复杂的高层指令转换为精确的策略代码。我们的方法具有可调节性和灵活性，能够适应各种指令模态和输入类型，并满足特定任务需求。我们通过在桌面操作领域内不同场景的机器人任务上评估该方法，验证了其实用性和效率。此外，我们的零样本方法在多项任务中优于许多基于学习的先进策略。我们提出方法的代码可在https://github.com/OpenGVLab/Instruct2Act获取，可作为具有多种模态输入的高层机器人指令任务的强大基准。

英文摘要：

Foundation models have made significant strides in various applications, including text-to-image generation, panoptic segmentation, and natural language processing. This paper presents Instruct2Act, a framework that utilizes Large Language Models to map multi-modal instructions to sequential actions for robotic manipulation tasks. Specifically, Instruct2Act employs the LLM model to generate Python programs that constitute a comprehensive perception, planning, and action loop for robotic tasks. In the perception section, pre-defined APIs are used to access multiple foundation models where the Segment Anything Model (SAM) accurately locates candidate objects, and CLIP classifies them. In this way, the framework leverages the expertise of foundation models and robotic abilities to convert complex high-level instructions into precise policy codes. Our approach is adjustable and flexible in accommodating various instruction modalities and input types and catering to specific task demands. We validated the practicality and efficiency of our approach by assessing it on robotic tasks in different scenarios within tabletop manipulation domains. Furthermore, our zero-shot method outperformed many state-of-the-art learning-based policies in several tasks. The code for our proposed approach is available at https://github.com/OpenGVLab/Instruct2Act, serving as a robust benchmark for high-level robotic instruction tasks with assorted modality inputs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。