Kestrel：用于部件感知接地描述的三维多模态大语言模型
Kestrel: 3D Multimodal LLM for Part-Aware Grounded Description
AI总结：
本文提出Kestrel，一种部件感知三维多模态大语言模型，结合多级点特征传播与查询细化，解决部件级接地描述任务，并构建3DCoMPaT-GrIn数据集，实验证明其有效连接语言理解与三维分割，助力机器人应用。
摘要：
在本文中，我们提出了部件感知点接地描述（PaPGD）这一具有挑战性的任务，旨在推进三维多模态学习，实现细粒度的部件感知分割接地以及三维物体的详细解释。现有的三维数据集主要侧重于仅视觉的部件分割或视觉-语言场景分割，缺乏真实世界环境中机器人导航和交互所需的细粒度多模态分割。为解决这一空白，我们提出了3DCoMPaT接地指令（3DCoMPaT-GrIn）数据集，这是一个综合资源，将丰富的点云描述与相应的部件级分割掩码配对。该数据集包含大量样本，专为PaPGD和细粒度单部件接地任务设计。为应对在部件级别接地物体并生成接地描述的固有挑战，我们提出了Kestrel，一种部件感知的三维多模态大语言模型，它集成了先进的语言模型以实现细致的语言理解，并采用多级点特征传播和查询细化机制来增强部件级别的空间推理。大量实验表明，Kestrel有效弥合了部件感知语言理解与三维分割接地之间的差距，为更鲁棒和可解释的三维物体理解铺平了道路，满足了真实世界机器人应用的需求。项目页面：https://feielysia.github.io/Kestrel.github.io/
英文摘要：
In this paper, we introduce Part-Aware Point Grounded Description (PaPGD), a challenging task aimed at advancing 3D multimodal learning for fine-grained, part-aware segmentation grounding and detailed explanation of 3D objects. Existing 3D datasets largely focus on either vision-only part segmentation or vision-language scene segmentation, lacking the fine-grained multimodal segmentation needed for robotic navigation and interaction in real-world environments. To address this gap, we present the 3DCoMPaT Grounded Instructions (3DCoMPaT-GrIn) Dataset, a comprehensive resource that pairs rich point cloud descriptions with corresponding part-level segmentation masks. This dataset encompasses extensive samples designed for both PaPGD and fine-grained single-part grounding tasks. To tackle the inherent challenges of grounding objects and generating grounded descriptions at the part level, we propose Kestrel, a part-aware 3D multimodal large language model that integrates an advanced language model for nuanced language comprehension with multi-level point feature propagation and query refinement mechanism to enhance spatial reasoning at the part level. The extensive experiments demonstrate that Kestrel effectively bridges the gap between part-aware language understanding and 3D segmentation grounding, paving the way for more robust and interpretable 3D object comprehension that meets the demands of real-world robotic applications. Project page at https://feielysia.github.io/Kestrel.github.io/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
