Feature 3DGS：增强 3D Gaussian Splatting 以实现蒸馏特征场
Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields
AI总结：
本文提出 Feature 3DGS，通过架构与训练改进将 2D 基础模型特征蒸馏到 3D Gaussian Splatting，实现快速高质量的 3D 语义特征场及点、框提示操控。
摘要：
近年来，3D 场景表示受到了广泛关注。使用 Neural Radiance fields（神经辐射场，NeRF）的方法在新视角合成等传统任务中用途广泛。近期出现了一些工作，旨在通过从 2D 基础模型蒸馏 3D 特征场，将 NeRF 的功能扩展到视角合成之外，以支持编辑和分割等语义感知任务。然而，这些方法存在两个主要局限：（a）它们受到 NeRF 管线渲染速度的限制；（b）隐式表示的特征场会产生连续性伪影，从而降低特征质量。最近，3D Gaussian Splatting（3DGS）在实时辐射场渲染方面展现出最先进的性能。在本工作中，我们更进一步：除辐射场渲染外，我们还通过 2D 基础模型蒸馏，使 3D Gaussian splatting 能够支持任意维度的语义特征。这种转换并不直接：在 3DGS 框架中朴素地引入特征场会遇到重大挑战，尤其是 RGB 图像与特征图之间在空间分辨率和通道一致性上的差异。我们提出了架构和训练方面的改进，以有效规避这一问题。我们提出的方法具有通用性，实验通过从 SAM 和 CLIP-LSeg 等最先进的 2D 基础模型学习特征场，展示了新视角语义分割、语言引导编辑以及 segment anything（分割一切）能力。在所有实验中，我们的蒸馏方法都能提供相当或更好的结果，同时训练和渲染速度显著更快。此外，据我们所知，我们是首个通过利用 SAM 模型实现点提示和边界框提示以进行辐射场操控的方法。项目网站：https://feature-3dgs.github.io/
英文摘要：
3D scene representations have gained immense popularity in recent years. Methods that use Neural Radiance fields are versatile for traditional tasks such as novel view synthesis. In recent times, some work has emerged that aims to extend the functionality of NeRF beyond view synthesis, for semantically aware tasks such as editing and segmentation using 3D feature field distillation from 2D foundation models. However, these methods have two major limitations: (a) they are limited by the rendering speed of NeRF pipelines, and (b) implicitly represented feature fields suffer from continuity artifacts reducing feature quality. Recently, 3D Gaussian Splatting has shown state-of-the-art performance on real-time radiance field rendering. In this work, we go one step further: in addition to radiance field rendering, we enable 3D Gaussian splatting on arbitrary-dimension semantic features via 2D foundation model distillation. This translation is not straightforward: naively incorporating feature fields in the 3DGS framework encounters significant challenges, notably the disparities in spatial resolution and channel consistency between RGB images and feature maps. We propose architectural and training changes to efficiently avert this problem. Our proposed method is general, and our experiments showcase novel view semantic segmentation, language-guided editing and segment anything through learning feature fields from state-of-the-art 2D foundation models such as SAM and CLIP-LSeg. Across experiments, our distillation method is able to provide comparable or better results, while being significantly faster to both train and render. Additionally, to the best of our knowledge, we are the first method to enable point and bounding-box prompting for radiance field manipulation, by leveraging the SAM model. Project website at: https://feature-3dgs.github.io/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
