基于自由形式语言的机器人推理与抓取
Free-form language-based robotic reasoning and grasping
AI总结：
针对自由形式语言引导的杂乱场景抓取，本文提出 FreeGrasp，借助关键点标注增强 GPT-4o 的零样本空间推理，并构建 FreeGraspData 验证其推理与执行性能。
摘要：
根据人类指令从杂乱料箱中执行机器人抓取是一项具有挑战性的任务，因为它既需要理解自由形式语言的细微差别，也需要理解物体之间的空间关系。在网络规模数据上训练的视觉-语言模型（VLM），例如 GPT-4o，已经在文本和图像上展现出卓越的推理能力。但是，它们真的能在零样本设置中用于这项任务吗？它们的局限性又是什么？在本文中，我们通过基于自由形式语言的机器人抓取任务探讨这些研究问题，并提出一种新方法 FreeGrasp，该方法利用预训练 VLM 的世界知识来推理人类指令和物体空间布局。我们的方法将所有物体检测为关键点，并使用这些关键点在图像上标注标记，旨在促进 GPT-4o 的零样本空间推理。这使我们的方法能够判断被请求的物体是否可以直接抓取，或者是否必须先抓取并移走其他物体。由于现有数据集都不是专门为这项任务设计的，我们通过扩展 MetaGraspNetV2 数据集并加入人工标注指令和真值抓取序列，引入了一个合成数据集 FreeGraspData。我们在 FreeGraspData 上进行了大量分析，并使用配备夹爪的机械臂进行了真实世界验证，结果表明该方法在抓取推理和执行方面达到了最先进性能。项目网站：https://tev-fbk.github.io/FreeGrasp/。
英文摘要：
Performing robotic grasping from a cluttered bin based on human instructions is a challenging task, as it requires understanding both the nuances of free-form language and the spatial relationships between objects. Vision-Language Models (VLMs) trained on web-scale data, such as GPT-4o, have demonstrated remarkable reasoning capabilities across both text and images. But can they truly be used for this task in a zero-shot setting? And what are their limitations? In this paper, we explore these research questions via the free-form language-based robotic grasping task, and propose a novel method, FreeGrasp, leveraging the pre-trained VLMs' world knowledge to reason about human instructions and object spatial arrangements. Our method detects all objects as keypoints and uses these keypoints to annotate marks on images, aiming to facilitate GPT-4o's zero-shot spatial reasoning. This allows our method to determine whether a requested object is directly graspable or if other objects must be grasped and removed first. Since no existing dataset is specifically designed for this task, we introduce a synthetic dataset FreeGraspData by extending the MetaGraspNetV2 dataset with human-annotated instructions and ground-truth grasping sequences. We conduct extensive analyses with both FreeGraspData and real-world validation with a gripper-equipped robotic arm, demonstrating state-of-the-art performance in grasp reasoning and execution. Project website: https://tev-fbk.github.io/FreeGrasp/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
