SemGrasp：通过语言对齐离散化实现语义抓取生成
SemGrasp: Semantic Grasp Generation via Language Aligned Discretization
AI总结：
本文提出 SemGrasp，通过语言对齐的离散抓取表示与微调 MLLM，实现符合语言指令的自然人类抓取生成，并构建 CapGrasp 数据集。
摘要：
生成自然的人类抓取不仅需要考虑物体几何形状，还必须考虑语义信息。仅依赖物体形状进行抓取生成，会限制以往方法在下游任务中的应用。本文提出一种新颖的基于语义的抓取生成方法，称为 SemGrasp，它通过将语义信息纳入抓取表示来生成静态人类抓取姿态。我们引入了一种离散表示，将抓取空间与语义空间对齐，从而能够按照语言指令生成抓取姿态。随后，对一个多模态大语言模型（MLLM）进行微调，在统一语义空间中整合物体、抓取和语言。为促进 SemGrasp 的训练，我们构建了一个大规模抓取—文本对齐数据集 CapGrasp，包含约 26 万条详细描述和 5 万种多样化抓取。实验结果表明，SemGrasp 能高效生成符合语言意图的自然人类抓取。我们的代码、模型和数据集公开发布于：https://kailinli.github.io/SemGrasp。
英文摘要：
Generating natural human grasps necessitates consideration of not just object geometry but also semantic information. Solely depending on object shape for grasp generation confines the applications of prior methods in downstream tasks. This paper presents a novel semantic-based grasp generation method, termed SemGrasp, which generates a static human grasp pose by incorporating semantic information into the grasp representation. We introduce a discrete representation that aligns the grasp space with semantic space, enabling the generation of grasp postures in accordance with language instructions. A Multimodal Large Language Model (MLLM) is subsequently fine-tuned, integrating object, grasp, and language within a unified semantic space. To facilitate the training of SemGrasp, we have compiled a large-scale, grasp-text-aligned dataset named CapGrasp, featuring about 260k detailed captions and 50k diverse grasps. Experimental findings demonstrate that SemGrasp efficiently generates natural human grasps in alignment with linguistic intentions. Our code, models, and dataset are available publicly at: https://kailinli.github.io/SemGrasp.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
