arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.05973cs.ROcs.AIcs.CLcs.CVcs.LG
收藏

VoxPoser:用于语言模型机器人操作的可组合3D价值图

VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models

Stanford University(斯坦福大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, Li Fei-Fei

AI总结:

本文提出VoxPoser,利用LLM与VLM组合3D价值图,并结合基于模型的规划零样本合成鲁棒机器人轨迹,支持开放语言指令下的多样日常操作任务。

摘要:

大型语言模型(LLM)被证明拥有丰富的可行动知识,这些知识可以以推理和规划的形式提取出来,用于机器人操作。尽管取得了进展,但大多数方法仍依赖预定义的运动原语来执行与环境的物理交互,这仍然是一个主要瓶颈。在本研究中,我们的目标是在给定开放指令集和开放物体集的情况下,为种类繁多的操作任务合成机器人轨迹,即6-DoF末端执行器航点的密集序列。我们首先观察到,LLM擅长在给定自由形式语言指令的情况下推断可供性和约束,并由此实现这一目标。更重要的是,通过利用其代码编写能力,它们可以与视觉语言模型(VLM)交互,组合3D价值图,将知识落地到智能体的观测空间中。随后,组合出的价值图被用于基于模型的规划框架,以零样本方式合成对动态扰动具有鲁棒性的闭环机器人轨迹。我们进一步展示了所提出框架如何通过为涉及富接触交互的场景高效学习动力学模型,从在线经验中获益。我们在仿真和真实机器人环境中对所提方法进行了大规模研究,展示了其执行以自由形式自然语言指定的多种日常操作任务的能力。视频和代码位于 https://voxposer.github.io。

英文摘要:

Large language models (LLMs) are shown to possess a wealth of actionable knowledge that can be extracted for robot manipulation in the form of reasoning and planning. Despite the progress, most still rely on pre-defined motion primitives to carry out the physical interactions with the environment, which remains a major bottleneck. In this work, we aim to synthesize robot trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a large variety of manipulation tasks given an open-set of instructions and an open-set of objects. We achieve this by first observing that LLMs excel at inferring affordances and constraints given a free-form language instruction. More importantly, by leveraging their code-writing capabilities, they can interact with a vision-language model (VLM) to compose 3D value maps to ground the knowledge into the observation space of the agent. The composed value maps are then used in a model-based planning framework to zero-shot synthesize closed-loop robot trajectories with robustness to dynamic perturbations. We further demonstrate how the proposed framework can benefit from online experiences by efficiently learning a dynamics model for scenes that involve contact-rich interactions. We present a large-scale study of the proposed method in both simulated and real-robot environments, showcasing the ability to perform a large variety of everyday manipulation tasks specified in free-form natural language. Videos and code at https://voxposer.github.io

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑