利用高斯溅射与混合扩散先验增强单图像到3D生成
Enhancing Single Image to 3D Generation using Gaussian Splatting and Hybrid Diffusion Priors
AI总结：
本文提出融合2D与3D扩散先验，通过基于频率的两阶段蒸馏损失结合高斯溅射，在保持几何一致性的同时提升纹理细节，从而增强单图像3D生成质量并超越现有最优方法。
摘要：
从单张图像生成3D对象涉及从野外拍摄的无位姿RGB图像中估计未见视角的完整3D几何与纹理。准确重建对象的完整3D结构和纹理在现实场景中具有众多应用，包括机器人操作、抓取、3D场景理解以及AR/VR。近期3D对象生成的进展引入了通过优化高斯溅射的高效表示来重建对象3D形状和纹理的技术，并由预训练的2D或3D扩散模型引导。然而，这些模型的训练数据集之间存在显著差异，导致其输出有明显不同。2D模型生成高度详细的视觉效果，但缺乏跨视角的几何和纹理一致性。相比之下，3D模型确保不同视角间的一致性，但往往产生过于平滑的纹理。我们提出弥合2D与3D扩散模型之间的差距，以解决这一局限，方法是将基于频率的两阶段蒸馏损失与高斯溅射相结合。具体而言，我们利用来自3D扩散模型的低频频谱中的几何先验来保持一致几何，并使用2D扩散模型在生成3D结构的高频频谱中细化保真度和纹理，从而获得更详细和细粒度的结果。我们的方法增强了几何一致性和视觉质量，超越了当前的最先进技术。此外，我们展示了该方法在高效对象姿态估计和跟踪中的易适应性。
英文摘要：
3D object generation from a single image involves estimating the full 3D geometry and texture of unseen views from an unposed RGB image captured in the wild. Accurately reconstructing an object's complete 3D structure and texture has numerous applications in real-world scenarios, including robotic manipulation, grasping, 3D scene understanding, and AR/VR. Recent advancements in 3D object generation have introduced techniques that reconstruct an object's 3D shape and texture by optimizing the efficient representation of Gaussian Splatting, guided by pre-trained 2D or 3D diffusion models. However, a notable disparity exists between the training datasets of these models, leading to distinct differences in their outputs. While 2D models generate highly detailed visuals, they lack cross-view consistency in geometry and texture. In contrast, 3D models ensure consistency across different views but often result in overly smooth textures. We propose bridging the gap between 2D and 3D diffusion models to address this limitation by integrating a two-stage frequency-based distillation loss with Gaussian Splatting. Specifically, we leverage geometric priors in the low-frequency spectrum from a 3D diffusion model to maintain consistent geometry and use a 2D diffusion model to refine the fidelity and texture in the high-frequency spectrum of the generated 3D structure, resulting in more detailed and fine-grained outcomes. Our approach enhances geometric consistency and visual quality, outperforming the current SOTA. Additionally, we demonstrate the easy adaptability of our method for efficient object pose estimation and tracking.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
