语言引导的机器人抓取：基于CLIP的杂乱场景指代抓取合成
Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Clutter
AI总结：
针对杂乱场景中指代抓取合成任务，提出基于CLIP的端到端模型CROG，直接学习图像-文本对到抓取姿态的映射，并在OCID基准及真实机器人实验中显著提升定位与抓取性能。
摘要：
在以人为本的环境中运行的机器人需要整合视觉定位和抓取能力，以便根据用户指令有效地操作物体。本工作聚焦于指代抓取合成任务，即在杂乱场景中预测自然语言所指物体的抓取姿态。现有方法通常采用多阶段流水线，先分割所指物体，再提出合适的抓取，并在私有数据集或模拟器上进行评估，这些数据集或模拟器未能捕捉自然室内场景的复杂性。为解决这些局限，我们基于OCID数据集中的杂乱室内场景开发了一个具有挑战性的基准，为其生成指代表达并将其与4自由度抓取姿态关联。此外，我们提出了一种新颖的端到端模型（CROG），利用CLIP的视觉定位能力直接从图像-文本对学习抓取合成。我们的结果表明，将CLIP与预训练模型进行朴素集成在我们的挑战性基准上迁移效果不佳，而CROG在定位和抓取方面均取得了显著改进。在模拟和硬件上的大量机器人实验证明了我们方法在包含杂乱物的挑战性交互式物体抓取场景中的有效性。
英文摘要：
Robots operating in human-centric environments require the integration of visual grounding and grasping capabilities to effectively manipulate objects based on user instructions. This work focuses on the task of referring grasp synthesis, which predicts a grasp pose for an object referred through natural language in cluttered scenes. Existing approaches often employ multi-stage pipelines that first segment the referred object and then propose a suitable grasp, and are evaluated in private datasets or simulators that do not capture the complexity of natural indoor scenes. To address these limitations, we develop a challenging benchmark based on cluttered indoor scenes from OCID dataset, for which we generate referring expressions and connect them with 4-DoF grasp poses. Further, we propose a novel end-to-end model (CROG) that leverages the visual grounding capabilities of CLIP to learn grasp synthesis directly from image-text pairs. Our results show that vanilla integration of CLIP with pretrained models transfers poorly in our challenging benchmark, while CROG achieves significant improvements both in terms of grounding and grasping. Extensive robot experiments in both simulation and hardware demonstrate the effectiveness of our approach in challenging interactive object grasping scenarios that include clutter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
