UniM-OV3D：基于细粒度特征表示的单模态开放词汇3D场景理解
UniM-OV3D: Uni-Modality Open-Vocabulary 3D Scene Understanding with Fine-Grained Feature Representation
AI总结：
本文提出UniM-OV3D网络，通过分层点云特征提取和分层3D描述对，融合点云、图像、语言和深度模态，在室内外基准上实现了最先进的3D开放词汇语义与实例分割性能。
摘要：
3D开放词汇场景理解旨在识别超越基础标签空间的任意新类别。然而，现有工作不仅未能充分利用3D领域中所有可用的模态信息，而且在表示每种模态的特征时缺乏足够的粒度。在本文中，我们提出了一个统一的多模态3D开放词汇场景理解网络，即UniM-OV3D，它将点云与图像、语言和深度对齐。为了更好地融合点云的全局和局部特征，我们设计了一个分层点云特征提取模块，用于学习全面的细粒度特征表示。此外，为了促进从描述中学习由粗到细的点-语义表示，我们提出利用分层3D描述对，借助3D场景不同视角间的几何约束。大量实验结果证明了我们的方法在开放词汇语义和实例分割中的有效性和优越性，在ScanNet、ScanNet200、S3IDS和nuScenes等室内外基准上均实现了最先进的性能。代码可在https://github.com/hithqd/UniM-OV3D获取。
英文摘要：
3D open-vocabulary scene understanding aims to recognize arbitrary novel categories beyond the base label space. However, existing works not only fail to fully utilize all the available modal information in the 3D domain but also lack sufficient granularity in representing the features of each modality. In this paper, we propose a unified multimodal 3D open-vocabulary scene understanding network, namely UniM-OV3D, which aligns point clouds with image, language and depth. To better integrate global and local features of the point clouds, we design a hierarchical point cloud feature extraction module that learns comprehensive fine-grained feature representations. Further, to facilitate the learning of coarse-to-fine point-semantic representations from captions, we propose the utilization of hierarchical 3D caption pairs, capitalizing on geometric constraints across various viewpoints of 3D scenes. Extensive experimental results demonstrate the effectiveness and superiority of our method in open-vocabulary semantic and instance segmentation, which achieves state-of-the-art performance on both indoor and outdoor benchmarks such as ScanNet, ScanNet200, S3IDS and nuScenes. Code is available at https://github.com/hithqd/UniM-OV3D.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
