基于语言和几何的稀疏体素表示用于整体场景理解
Language and Geometry Grounded Sparse Voxel Representations for Holistic Scene Understanding
AI总结：
本文提出基于语言和几何的稀疏体素表示方法，通过统一框架建模3D场景的外观、语义和几何，提升整体场景理解和重建性能。
摘要：
现有的3D开放词汇场景理解方法大多强调从2D基础模型中蒸馏语言特征到3D特征场，但 largely 忽略了场景外观、语义和几何之间的协同作用。结果，场景理解往往偏离场景的底层几何结构，并与重建过程脱节。在本文中，我们提出了一种新的方法，利用语言和几何引导的稀疏体素表示，以统一的框架全面建模外观、语义和几何。具体来说，我们使用3D稀疏体素作为基本元素，并采用外观场、密度场、特征场和置信度场来整体表示3D场景。为了促进外观、密度和特征场之间的协同作用，我们构建了一个特征调节模块，并从2D基础模型中蒸馏语言特征到我们的3D场景模型中。此外，我们整合了几何蒸馏到特征场蒸馏中，通过深度相关正则化和模式一致性正则化将几何知识从几何基础模型转移到我们的3D场景表示中。这些组件共同作用，以协同建模3D场景的外观、语义和几何。广泛的实验表明，我们的方法在整体场景理解和重建方面相比最先进的方法取得了更优越的整体性能。
英文摘要：
Existing 3D open-vocabulary scene understanding methods mostly emphasize distilling language features from 2D foundation models into 3D feature fields, but largely overlook the synergy among scene appearance, semantics, and geometry. As a result, scene understanding often deviates from the underlying geometric structure of scenes and becomes decoupled from the reconstruction process. In this work, we propose a novel approach that leverages language and geometry grounded sparse voxel representations to comprehensively model appearance, semantics, and geometry within a unified framework. Specifically, we use 3D sparse voxels as primitives and employ an appearance field, a density field, a feature field, and a confidence field to holistically represent a 3D scene. To promote synergy among the appearance, density, and feature fields, we construct a feature modulation module and distill language features from a 2D foundation model into our 3D scene model. In addition, we integrate geometric distillation into feature field distillation to transfer geometric knowledge from a geometry foundation model to our 3D scene representations via depth correlation regularization and pattern consistency regularization. These components work together to synergistically model the appearance, semantics, and geometry of the 3D scene within a unified framework. Extensive experiments demonstrate that our approach achieves superior overall performance compared with state-of-the-art methods in holistic scene understanding and reconstruction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
