UniForward：仅通过稀疏视图图像的前馈 Gaussian Splatting 实现统一的 3D 场景与语义场重建
UniForward: Unified 3D Scene and Semantic Field Reconstruction via Feed-Forward Gaussian Splatting from Only Sparse-View Images
AI总结：
本文提出 UniForward，仅从未标定、无位姿的稀疏视图图像端到端预测带语义特征的 3D Gaussians，实现实时、可泛化的 3D 场景与语义场统一重建，并在新视图合成和分割任务上达到 SOTA。
摘要：
我们提出了一种前馈 Gaussian Splatting 模型，用于统一 3D 场景与语义场重建。将 3D 场景与语义场相结合有助于感知和理解周围环境。然而，关键挑战包括：将语义嵌入 3D 表示、实现可泛化的实时重建，以及仅使用图像作为输入而不依赖相机参数或真值深度来确保实际适用性。为此，我们提出 UniForward，这是一种前馈模型，能够仅从未标定且无相机位姿的稀疏视图图像预测带有各向异性语义特征的 3D Gaussians。为实现 3D 场景与语义场的统一表示，我们将语义特征嵌入 3D Gaussians，并通过双分支解耦解码器进行预测。在训练过程中，我们提出一种损失引导的视图采样器，按照由易到难的方式采样视图，从而消除以往方法对真值深度或掩码的需求，并稳定训练过程。整个模型可以利用光度损失和蒸馏损失进行端到端训练，其中蒸馏损失利用来自预训练 2D 语义模型的语义特征。在推理阶段，我们的 UniForward 能够仅根据稀疏视图图像实时重建 3D 场景及相应语义场。重建的 3D 场景实现高质量渲染，重建的 3D 语义场支持从任意视图渲染视图一致的语义特征，并可进一步以开放词汇方式解码为密集分割掩码。在新视图合成和新视图分割上的实验表明，我们的方法在统一 3D 场景与语义场重建方面取得了最先进性能。
英文摘要：
We propose a feed-forward Gaussian Splatting model that unifies 3D scene and semantic field reconstruction. Combining 3D scenes with semantic fields facilitates the perception and understanding of the surrounding environment. However, key challenges include embedding semantics into 3D representations, achieving generalizable real-time reconstruction, and ensuring practical applicability by using only images as input without camera parameters or ground truth depth. To this end, we propose UniForward, a feed-forward model to predict 3D Gaussians with anisotropic semantic features from only uncalibrated and unposed sparse-view images. To enable the unified representation of the 3D scene and semantic field, we embed semantic features into 3D Gaussians and predict them through a dual-branch decoupled decoder. During training, we propose a loss-guided view sampler to sample views from easy to hard, eliminating the need for ground truth depth or masks required by previous methods and stabilizing the training process. The whole model can be trained end-to-end using a photometric loss and a distillation loss that leverages semantic features from a pre-trained 2D semantic model. At the inference stage, our UniForward can reconstruct 3D scenes and the corresponding semantic fields in real time from only sparse-view images. The reconstructed 3D scenes achieve high-quality rendering, and the reconstructed 3D semantic field enables the rendering of view-consistent semantic features from arbitrary views, which can be further decoded into dense segmentation masks in an open-vocabulary manner. Experiments on novel view synthesis and novel view segmentation demonstrate that our method achieves state-of-the-art performances for unifying 3D scene and semantic field reconstruction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
