释放多视角融合潜力：VLM中的噪声校正用于开放词汇3D场景理解
Unleashing the Multi-View Fusion Potential: Noise Correction in VLM for Open-Vocabulary 3D Scene Understanding
AI总结：
提出MVOV3D方法，通过CLIP编码器提取精确区域级特征并引入3D几何先验，在无需训练的情况下减少VLM固有噪声以优化2D多视角融合，在开放词汇3D语义分割任务上大幅超越现有3D网络。
摘要：
近期的开放词汇3D场景理解方法主要通过点-文本对的对比学习训练3D网络，或通过点-像素对齐将2D特征蒸馏至3D模型。尽管这些方法在词汇量有限的基准上表现可观，但由于3D数据量有限，难以训练强大的开放词汇3D模型，从而难以处理多样化的目标类别。我们观察到，2D多视角融合方法在理解3D场景中的多样概念方面更具优势。然而，视觉-语言模型中的固有噪声导致多视角融合性能次优。为此，我们提出MVOV3D，旨在释放2D多视角融合在开放词汇3D场景理解中的潜力。我们专注于在无需训练的情况下减少固有噪声，从而在保持泛化能力的同时增强开放世界能力。具体而言，MVOV3D利用CLIP编码器编码的精确区域级图像特征和文本特征来改进多视角2D特征，并引入3D几何先验以优化多视角融合。在多个数据集上的大量实验证明了本方法的有效性。值得注意的是，MVOV3D在ScanNet200上取得14.7% mIoU、在Matterport160上取得16.2% mIoU的新纪录，用于挑战性开放词汇语义分割，大幅超越当前领先的已训练3D网络。
英文摘要：
Recent open-vocabulary 3D scene understanding approaches mainly focus on training 3D networks through contrastive learning with point-text pairs or by distilling 2D features into 3D models via point-pixel alignment. While these methods show considerable performance in benchmarks with limited vocabularies, they struggle to handle diverse object categories as the limited amount of 3D data upbound training strong open-vocabulary 3d models. We observe that 2D multi-view fusion methods take precedence in understanding diverse concepts in 3D scenes. However, inherent noises in vision-language models lead multi-view fusion to sub-optimal performance. To this end, we introduce MVOV3D, a novel approach aimed at unleashing the potential of 2D multi-view fusion for open-vocabulary 3D scene understanding. We focus on reducing the inherent noises without training, thereby preserving the generalizability while enhancing open-world capabilities. Specifically, MVOV3D improves multi-view 2D features by leveraging precise region-level image features and text features encoded by CLIP encoders and incorporates 3D geometric priors to optimize multi-view fusion. Extensive experiments on various datasets demonstrate the effectiveness of our method. Notably, our MVOV3D achieves a new record with 14.7% mIoU on ScanNet200 and 16.2% mIoU on Matterport160 for challenge open-vocabulary semantic segmentation, outperforming current leading trained 3D networks by a significant margin.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
