Splats 的一课：用于 2D 监督生成 3D Gaussian Splats 的教师引导扩散模型
A Lesson in Splats: Teacher-Guided Diffusion for 3D Gaussian Splats Generation with 2D Supervision
AI总结：
本文提出教师引导的 3D 扩散框架，通过解耦 3D 含噪样本与 2D 监督信号，仅用稀疏视图监督实现高保真 3DGS 生成。
摘要：
我们提出了一种仅使用 2D 监督来训练以 3D 图像为条件的扩散模型的新颖框架。由于可能重建结果的歧义性，从 2D 图像恢复 3D 结构本质上是一个不适定问题，因此生成模型成为自然的选择。然而，大多数现有 3D 生成模型依赖完整的 3D 监督，由于大规模 3D 数据集稀缺，这并不现实。为解决该问题，我们提议利用稀疏视图监督作为一种可扩展的替代方案。尽管近期重建模型使用稀疏视图监督和可微渲染将 2D 图像提升至 3D，但它们主要是确定性的，无法捕捉多样化的合理解集，并会在不确定区域产生模糊预测。使用 2D 监督训练 3D 扩散模型的一个关键挑战在于，标准训练范式要求去噪过程和监督处于同一模态。我们通过将正在去噪的含噪样本与监督信号解耦来解决这一问题，使前者保持在 3D 中，而后者以 2D 形式提供。我们的方法利用确定性图像到 3D 模型（充当“教师”）的次优预测来生成含噪 3D 输入，从而无需完整 3D 真值即可实现有效的 3D 扩散训练。我们在对象级和场景级数据集上，使用两个不同的 3D Gaussian Splat（3DGS）教师验证了该框架。结果表明，我们的方法持续改进了这些确定性教师，证明了其在可扩展、高保真 3D 生成建模中的有效性。项目页面见 https://lesson-in-splats.github.io/
英文摘要：
We present a novel framework for training 3D image-conditioned diffusion models using only 2D supervision. Recovering 3D structure from 2D images is inherently ill-posed due to the ambiguity of possible reconstructions, making generative models a natural choice. However, most existing 3D generative models rely on full 3D supervision, which is impractical due to the scarcity of large-scale 3D datasets. To address this, we propose leveraging sparse-view supervision as a scalable alternative. While recent reconstruction models use sparse-view supervision with differentiable rendering to lift 2D images to 3D, they are predominantly deterministic, failing to capture the diverse set of plausible solutions and producing blurry predictions in uncertain regions. A key challenge in training 3D diffusion models with 2D supervision is that the standard training paradigm requires both the denoising process and supervision to be in the same modality. We address this by decoupling the noisy samples being denoised from the supervision signal, allowing the former to remain in 3D while the latter is provided in 2D. Our approach leverages suboptimal predictions from a deterministic image-to-3D model-acting as a "teacher"-to generate noisy 3D inputs, enabling effective 3D diffusion training without requiring full 3D ground truth. We validate our framework on both object-level and scene-level datasets, using two different 3D Gaussian Splat (3DGS) teachers. Our results show that our approach consistently improves upon these deterministic teachers, demonstrating its effectiveness in scalable and high-fidelity 3D generative modeling. See our project page at https://lesson-in-splats.github.io/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
