SeededGrasp：复杂场景中多实体语言引导抓取

中文摘要 AI 辅助

复杂场景中的实际机器人抓取需要 3D 空间推理和与特定任务要求对齐。视觉语言模型（VLM）提供了用语言指定这些要求的自然方式，但现有方法存在局限。本文提出 SeededGrasp，一种数据高效的框架，使 VLM 能预测种子点，为后续轻量级抓取生成模型提供条件。其架构解耦高级语义推理与低级几何执行，支持多实体且无需昂贵的端到端训练。还发布了首个多实体桌面抓取数据集。实验结果表明该方法优于现有基线，在模拟中成功率达 72%，在实际抓取实验中达 78%。

英文摘要

Practical robotic grasping in complex scenes requires both 3D spatial reasoning and alignment with task-specific requirements. Vision-language models (VLMs) offer a natural way to specify these requirements using language, but existing approaches either use a VLM to predict the grasp directly with limited spatial awareness, or train the VLM together with the grasping model, which requires significantly more data and compute. These limitations impede performance and have prevented scaling to multiple embodiments in complex scenes. We address this by proposing SeededGrasp, a novel data-efficient framework that enables a VLM to predict a seed point to be used as conditioning for a subsequent lightweight grasp-generation model. Our architecture decouples high-level semantic reasoning from low-level geometric execution, enabling multi-embodiment support while bypassing the need for expensive end-to-end training. To enable training such models, we release the first multi-embodiment tabletop grasping dataset comprising over 2.5M grasps in cluttered scenes. Experimental results demonstrate that our approach outperforms existing baselines, achieving 72% success in simulation and 78% in real-world grasping experiments. See our project site for data and code: https://uoft-isl.github.io/seeded-grasp/

发表机构

• University of Toronto（多伦多大学）
• Vector Institute（向量研究所）
• University of British Columbia（英属哥伦比亚大学）
• Google Deepmind（谷歌DeepMind）

机构由 AI 辅助整理，请以论文原文为准。