arXiv 2503.167072025-03-28cs.CV

跨模态与不确定性感知聚合用于开放词汇3D场景理解

Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding

University of Trento（特伦托大学）；Fondazione Bruno Kessler（布鲁诺·凯塞勒基金会）

Jinlong Li, Cristiano Saltori, Fabio Poiesi, Nicu Sebe

备注：: Accepted by CVPR 2025

AI总结：

提出CUA-O3D，首个集成CLIP、DINOv2和Stable Diffusion等多基础模型的开放词汇3D场景理解方法，通过确定性不确定性估计自适应融合异构2D特征，在ScanNetV2和Matterport3D上实现先进分割与跨域对齐。

摘要：

大规模3D-文本语料库的缺乏促使近期工作从视觉-语言模型（VLMs）中蒸馏开放词汇知识。然而，这些方法通常依赖单一VLM将3D模型的特征空间对齐到共同的语言空间，这限制了3D模型利用各种基础模型所蕴含的多样化空间与语义能力的潜力。在本文中，我们提出了用于开放词汇3D场景理解的跨模态与不确定性感知聚合方法，称为CUA-O3D，这是首个将多个基础模型（如CLIP、DINOv2和Stable Diffusion）集成到3D场景理解中的模型。我们进一步引入了一种确定性不确定性估计，以自适应地蒸馏并协调来自这些模型的异构2D特征嵌入。我们的方法解决了两个关键挑战：（1）将来自VLMs的语义先验与空间感知视觉基础模型的几何知识相结合；（2）使用新颖的确定性不确定性估计来捕获跨不同语义和几何敏感性的模型特定不确定性，帮助在训练过程中调和异构表示。在ScanNetV2和Matterport3D上的大量实验表明，我们的方法不仅推进了开放词汇分割，还实现了稳健的跨域对齐和具有竞争力的空间感知能力。代码将在https://github.com/TyroneLi/CUA_O3D提供。

英文摘要：

The lack of a large-scale 3D-text corpus has led recent works to distill open-vocabulary knowledge from vision-language models (VLMs). However, these methods typically rely on a single VLM to align the feature spaces of 3D models within a common language space, which limits the potential of 3D models to leverage the diverse spatial and semantic capabilities encapsulated in various foundation models. In this paper, we propose Cross-modal and Uncertainty-aware Agglomeration for Open-vocabulary 3D Scene Understanding dubbed CUA-O3D, the first model to integrate multiple foundation models-such as CLIP, DINOv2, and Stable Diffusion-into 3D scene understanding. We further introduce a deterministic uncertainty estimation to adaptively distill and harmonize the heterogeneous 2D feature embeddings from these models. Our method addresses two key challenges: (1) incorporating semantic priors from VLMs alongside the geometric knowledge of spatially-aware vision foundation models, and (2) using a novel deterministic uncertainty estimation to capture model-specific uncertainties across diverse semantic and geometric sensitivities, helping to reconcile heterogeneous representations during training. Extensive experiments on ScanNetV2 and Matterport3D demonstrate that our method not only advances open-vocabulary segmentation but also achieves robust cross-domain alignment and competitive spatial perception capabilities. The code will be available at: https://github.com/TyroneLi/CUA_O3D.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。