迈向可泛化的视觉-语言机器人操作：一个基准与LLM引导的3D策略
Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policy
AI总结：
本文提出GemBench基准评估视觉-语言机器人操作泛化能力，并引入利用3D信息的3D-LOTUS方法及结合LLM与VLM的3D-LOTUS++框架，在新颖任务上取得SOTA性能。
摘要：
将语言条件化的机器人策略泛化到新任务仍是一项重大挑战，且受限于缺乏合适的模拟基准。本文通过引入GemBench填补了这一空白，这是一个用于评估视觉-语言机器人操作策略泛化能力的新颖基准。GemBench包含七种基本动作原语和四个泛化级别，涵盖新颖放置、刚性与铰接物体以及复杂的长时程任务。我们在GemBench上评估了现有最先进的方法，并提出了一种新方法。我们的方法3D-LOTUS利用丰富的3D信息进行以语言为条件的动作预测。尽管3D-LOTUS在已见任务的效率和性能上均表现出色，但在处理新颖任务时仍显吃力。为解决此问题，我们提出了3D-LOTUS++框架，该框架将3D-LOTUS的运动规划能力与LLM的任务规划能力及VLM的物体定位精度相集成。3D-LOTUS++在GemBench的新颖任务上取得了最先进的性能，为机器人操作泛化树立了新标准。基准、代码和训练好的模型可在https://www.di.ens.fr/willow/research/gembench/获取。
英文摘要：
Generalizing language-conditioned robotic policies to new tasks remains a significant challenge, hampered by the lack of suitable simulation benchmarks. In this paper, we address this gap by introducing GemBench, a novel benchmark to assess generalization capabilities of vision-language robotic manipulation policies. GemBench incorporates seven general action primitives and four levels of generalization, spanning novel placements, rigid and articulated objects, and complex long-horizon tasks. We evaluate state-of-the-art approaches on GemBench and also introduce a new method. Our approach 3D-LOTUS leverages rich 3D information for action prediction conditioned on language. While 3D-LOTUS excels in both efficiency and performance on seen tasks, it struggles with novel tasks. To address this, we present 3D-LOTUS++, a framework that integrates 3D-LOTUS's motion planning capabilities with the task planning capabilities of LLMs and the object grounding accuracy of VLMs. 3D-LOTUS++ achieves state-of-the-art performance on novel tasks of GemBench, setting a new standard for generalization in robotic manipulation. The benchmark, codes and trained models are available at https://www.di.ens.fr/willow/research/gembench/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
