arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.07765cs.CV
收藏

基于像素空间扩散模型的新视角合成

Novel View Synthesis with Pixel-Space Diffusion Models

Technion(以色列理工学院);Apple(苹果公司)

Noam Elata, Bahjat Kawar, Yaron Ostrovsky-Berman, Miriam Farber, Ron Sokolovsky

AI总结:

本研究采用现代扩散模型架构在像素空间进行端到端新视角合成,显著超越SOTA,并引入利用单视图数据集的新训练方案以提升对域外场景的泛化能力。

摘要:

从单张输入图像合成新视角是一项具有挑战性的任务。传统上,该任务通过估计场景深度、图像变形和图像修复来完成,机器学习模型使部分流程成为可能。最近,生成模型越来越多地被用于新视角合成(NVS),通常涵盖整个端到端系统。在本研究中,我们采用现代扩散模型架构在像素空间进行端到端NVS,显著优于之前的最先进(SOTA)技术。我们探索了将几何信息编码到网络中的不同方法。实验表明,虽然这些方法可以提升性能,但与利用改进的生成模型相比,其影响较小。此外,我们引入了一种新的NVS训练方案,利用单视图数据集,利用其相对于多视图数据集的丰富性。这提高了对域外内容场景的泛化能力。

英文摘要:

Synthesizing a novel view from a single input image is a challenging task. Traditionally, this task was approached by estimating scene depth, warping, and inpainting, with machine learning models enabling parts of the pipeline. More recently, generative models are being increasingly employed in novel view synthesis (NVS), often encompassing the entire end-to-end system. In this work, we adapt a modern diffusion model architecture for end-to-end NVS in the pixel space, substantially outperforming previous state-of-the-art (SOTA) techniques. We explore different ways to encode geometric information into the network. Our experiments show that while these methods may enhance performance, their impact is minor compared to utilizing improved generative models. Moreover, we introduce a novel NVS training scheme that utilizes single-view datasets, capitalizing on their relative abundance compared to their multi-view counterparts. This leads to improved generalization capabilities to scenes with out-of-domain content.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑