Robotic Programmer：面向机器人操作的视频指令策略代码生成

中文摘要 AI 辅助

在不同机器人、任务和环境之间实现零样本泛化，仍是机器人操作领域的一项重大挑战。策略代码生成方法利用可执行代码连接高层任务描述与低层动作序列，并借助大语言模型和原子技能库的泛化能力。在本研究中，我们提出 Robotic Programmer（RoboPro），这是一种机器人基础模型，能够感知视觉信息并遵循自由形式指令，以零样本方式通过策略代码执行机器人操作。为了解决收集机器人任务运行时代码数据效率低、成本高的问题，我们设计了 Video2Code，利用现成的视觉语言模型和代码领域大语言模型，从大量野外视频中合成可执行代码。大量实验表明，RoboPro 在模拟器和真实世界环境中的机器人操作任务上均取得了最先进的零样本性能。具体而言，RoboPro 在 RLBench 上的零样本成功率比最先进模型 GPT-4o 高出 11.6%，甚至可与强大的监督训练基线相媲美。此外，RoboPro 对 API 格式和技能集的变化具有鲁棒性。

英文摘要

Zero-shot generalization across various robots, tasks and environments remains a significant challenge in robotic manipulation. Policy code generation methods use executable code to connect high-level task descriptions and low-level action sequences, leveraging the generalization capabilities of large language models and atomic skill libraries. In this work, we propose Robotic Programmer (RoboPro), a robotic foundation model, enabling the capability of perceiving visual information and following free-form instructions to perform robotic manipulation with policy code in a zero-shot manner. To address low efficiency and high cost in collecting runtime code data for robotic tasks, we devise Video2Code to synthesize executable code from extensive videos in-the-wild with off-the-shelf vision-language model and code-domain large language model. Extensive experiments show that RoboPro achieves the state-of-the-art zero-shot performance on robotic manipulation in both simulators and real-world environments. Specifically, the zero-shot success rate of RoboPro on RLBench surpasses the state-of-the-art model GPT-4o by 11.6%, which is even comparable to a strong supervised training baseline. Furthermore, RoboPro is robust to variations on API formats and skill sets.

发表机构

• Institute of Computing Technology, Chinese Academy of Sciences (CAS)（中国科学院计算技术研究所）
• University of Chinese Academy of Sciences（中国科学院大学）

机构由 AI 辅助整理，请以论文原文为准。