Semantic Gaussians：基于3D Gaussian Splatting的开放词汇场景理解
Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting
AI总结：
本文提出Semantic Gaussians，通过将2D预训练模型知识投影至3D Gaussians实现开放词汇场景理解，支持快速推理并在分割、定位等任务中表现优越。
摘要：
开放词汇3D场景理解在计算机视觉中是一项重大挑战，在具身智能体和增强现实系统中有广泛应用。现有方法采用神经渲染方法作为3D表示，并联合优化颜色和语义特征，以同时实现渲染和场景理解。本文提出Semantic Gaussians，一种基于3D Gaussian Splatting的新型开放词汇场景理解方法。核心思想是将2D预训练模型的知识蒸馏到3D Gaussians中。与现有方法不同，我们设计了一种通用的投影方法，基于空间关系将来自预训练图像编码器的各种2D语义特征映射到3D Gaussians的新语义组件中，且无需额外训练。我们进一步构建了一个3D语义网络，可直接从原始3D Gaussians预测语义组件，实现快速推理。在ScanNet分割和LERF对象定位上的定量结果证明了本方法的优越性能。此外，我们探索了Semantic Gaussians的多种应用，包括对象部件分割、实例分割、场景编辑和时空分割，相比2D和3D基线取得了更好的定性结果，凸显了其在支持多样下游任务上的通用性和有效性。
英文摘要：
Open-vocabulary 3D scene understanding presents a significant challenge in computer vision, with wide-ranging applications in embodied agents and augmented reality systems. Existing methods adopt neurel rendering methods as 3D representations and jointly optimize color and semantic features to achieve rendering and scene understanding simultaneously. In this paper, we introduce Semantic Gaussians, a novel open-vocabulary scene understanding approach based on 3D Gaussian Splatting. Our key idea is to distill knowledge from 2D pre-trained models to 3D Gaussians. Unlike existing methods, we design a versatile projection approach that maps various 2D semantic features from pre-trained image encoders into a novel semantic component of 3D Gaussians, which is based on spatial relationship and need no additional training. We further build a 3D semantic network that directly predicts the semantic component from raw 3D Gaussians for fast inference. The quantitative results on ScanNet segmentation and LERF object localization demonstates the superior performance of our method. Additionally, we explore several applications of Semantic Gaussians including object part segmentation, instance segmentation, scene editing, and spatiotemporal segmentation with better qualitative results over 2D and 3D baselines, highlighting its versatility and effectiveness on supporting diverse downstream tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
