RefMask3D：用于3D指代分割的语言引导Transformer
RefMask3D: Language-Guided Transformer for 3D Referring Segmentation
AI总结：
提出RefMask3D模型，通过几何增强组词注意力、语言基元构建和物体聚类模块解决3D指代分割中的视觉-语言融合对齐问题，在多个任务上达到最先进性能。
摘要：
3D指代分割是一项新兴且充满挑战的视觉-语言任务，旨在点云场景中分割出由自然语言表达所描述的物体。该任务背后的关键挑战在于视觉-语言特征融合与对齐。在本研究中，我们提出RefMask3D来探索全面的多模态特征交互与理解。首先，我们提出Geometry-Enhanced Group-Word Attention（几何增强组词注意力），通过跨模态组词注意力将语言与几何连贯的子点云融合，有效应对了点云稀疏且不规则特性带来的挑战。接着，我们引入Linguistic Primitives Construction（语言基元构建）以生成表示不同语义属性的语义基元，这极大增强了解码阶段的视觉-语言理解。此外，我们引入Object Cluster Module（物体聚类模块），分析语言基元之间的相互关系以巩固其见解并准确定位共同特征，有助于捕获整体信息并提升目标识别的精确度。所提出的RefMask3D在3D指代分割、3D视觉定位以及2D指代图像分割上均取得了新的最先进性能。特别是在具有挑战性的ScanRefer数据集上，RefMask3D以3.16% mIoU的巨大优势超越了以往的最先进方法。代码可在https://github.com/heshuting555/RefMask3D获取。
英文摘要：
3D referring segmentation is an emerging and challenging vision-language task that aims to segment the object described by a natural language expression in a point cloud scene. The key challenge behind this task is vision-language feature fusion and alignment. In this work, we propose RefMask3D to explore the comprehensive multi-modal feature interaction and understanding. First, we propose a Geometry-Enhanced Group-Word Attention to integrate language with geometrically coherent sub-clouds through cross-modal group-word attention, which effectively addresses the challenges posed by the sparse and irregular nature of point clouds. Then, we introduce a Linguistic Primitives Construction to produce semantic primitives representing distinct semantic attributes, which greatly enhance the vision-language understanding at the decoding stage. Furthermore, we introduce an Object Cluster Module that analyzes the interrelationships among linguistic primitives to consolidate their insights and pinpoint common characteristics, helping to capture holistic information and enhance the precision of target identification. The proposed RefMask3D achieves new state-of-the-art performance on 3D referring segmentation, 3D visual grounding, and also 2D referring image segmentation. Especially, RefMask3D outperforms previous state-of-the-art method by a large margin of 3.16% mIoU} on the challenging ScanRefer dataset. Code is available at https://github.com/heshuting555/RefMask3D.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
