面向开放词汇3D场景理解的密集多模态对齐
Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understanding
AI总结：
针对现有开放词汇3D场景理解方法忽略多模态协同的问题，提出DMA框架，通过密集多模态对齐结合CLIP特征与可学习掩码特征，在多类场景任务中取得优异的开放词汇分割性能。
摘要：
近期的视觉-语言预训练模型在零样本识别任务中展现出了卓越的泛化能力。已有的开放词汇3D场景理解方法大多侧重于利用图像或文本单一模态的监督来训练3D模型，忽略了所有模态的协同优势。本文提出了Dense Multimodal Alignment（DMA，密集多模态对齐）框架，将不同模态密集地共同嵌入到一个公共空间中，以最大化它们的协同效益。不同于提取粗粒度的视图级或区域级文本提示，我们利用大型视觉-语言模型提取完整的类别信息和可扩展的场景描述来构建文本模态，并以图像模态为桥梁建立密集的点-像素-文本关联。此外，为了在不损害开放词汇能力的前提下增强2D模型对下游3D任务的泛化能力，我们采用双路径整合方法，结合冻结的CLIP视觉特征和可学习的掩码特征。大量实验表明，我们的DMA方法在多种室内和室外任务上取得了极具竞争力的开放词汇分割性能。
英文摘要：
Recent vision-language pre-training models have exhibited remarkable generalization ability in zero-shot recognition tasks. Previous open-vocabulary 3D scene understanding methods mostly focus on training 3D models using either image or text supervision while neglecting the collective strength of all modalities. In this work, we propose a Dense Multimodal Alignment (DMA) framework to densely co-embed different modalities into a common space for maximizing their synergistic benefits. Instead of extracting coarse view- or region-level text prompts, we leverage large vision-language models to extract complete category information and scalable scene descriptions to build the text modality, and take image modality as the bridge to build dense point-pixel-text associations. Besides, in order to enhance the generalization ability of the 2D model for downstream 3D tasks without compromising the open-vocabulary capability, we employ a dual-path integration approach to combine frozen CLIP visual features and learnable mask features. Extensive experiments show that our DMA method produces highly competitive open-vocabulary segmentation performance on various indoor and outdoor tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
