arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.01962cs.CVcs.AIcs.GRcs.RO
收藏

NeoMap: 无需训练的单图像和视频新视角合成

NeoMap: Training-free Novel-View Synthesis from Single Images and Videos

Shenzhen Research Institute, The Hong Kong Polytechnic University(香港理工大学深圳研究院);vLAR Group, The Hong Kong Polytechnic University(香港理工大学vLAR实验室)

Jinxi Li, Tianyi Zhang, Yafei Yang, Zihui Zhang, Peng Huang, Koon Wing Macgyver Lin, Bo Yang

AI总结:

提出NeoMap,一种无需训练的框架,通过收敛流形交替投影迭代优化初始噪声,从预训练视频模型中定位高保真、视角一致的新视角解,在多个基准上达到最优。

摘要:

我们研究了从单图像或单目视频进行新视角视频合成的挑战性问题。现有方法假设预训练视频模型缺乏原生新视角合成能力,并通过相机条件、任务特定微调或逐步硬去噪指导来强制视角对齐,但常常出现伪影并损害全局场景一致性。在本文中,我们介绍NeoMap,一种新颖的无需训练框架,旨在从通用预训练视频模型中定位高保真、视角一致的新视角解。我们方法的核心洞察是,有前景的新视角解本质上编码在预训练模型学习的自然视频数据流形中,核心挑战仅仅是定位这个最优解。我们通过核心机制解决这一问题:收敛流形交替投影迭代,优化初始噪声。大量实验表明,NeoMap在3个标准新视角合成基准(包括具有挑战性的Tanks-and-Temples、LLFF和DAVIS数据集)上显著优于所有现有方法,实现了最先进的生成保真度和顶级视角一致性。

英文摘要:

We study the challenging problem of novel view video synthesis from single images or monocular videos. Existing methods, which operate under the assumption that pre-trained video models lack native novel view synthesis capability and enforce view alignment via camera conditioning, task-specific fine-tuning, or stepwise hard denoising guidance, often suffer from artifacts and compromised global scene consistency. In this paper, we introduce NeoMap, a novel training-free framework designed to locate high-fidelity, view-consistent novel view solutions from general pre-trained video models. The key to our approach is the core insight that promising novel view solutions are inherently encoded within the natural video data manifold learned by pre-trained models, and the core challenge is simply to locate this optimal solution. We solve this via our core mechanism: convergent manifold alternating projection iterations that optimize the initial noise. Extensive experiments demonstrate that NeoMap significantly outperforms all existing methods across 3 standard novel view synthesis benchmarks, including the challenging Tanks-and-Temples, LLFF and DAVIS datasets, achieving state-of-the-art generation fidelity and top-tier view consistency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑