arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.19613cs.CV

OrbitNVS: 利用视频扩散先验进行新视角合成

OrbitNVS: Harnessing Video Diffusion Priors for Novel View Synthesis

South China University of Technology(华南理工大学);King's College London(伦敦大学国王学院);Agentic Labs, XGTech(Agentic实验室,XGTech);Pazhou Laboratory(Pazhou实验室)

Jinglin Liang, Zijian Zhou, Rui Huang, Shuangping Huang, Yichen Gong

AI总结:

OrbitNVS将新视角合成重新定义为轨道视频生成任务,通过定制模型设计和训练策略,利用预训练视频生成模型的丰富视觉先验,提升几何和外观一致性,实验表明在GSO和OmniObject3D基准上表现优异。

摘要:

新视角合成(NVS)旨在根据有限已知视角生成3D物体的未见视角。现有方法在单视角输入下难以生成合理视角,且在保持几何和外观一致性方面存在挑战。为此,我们提出OrbitNVS,将NVS重新定义为轨道视频生成任务。通过定制化模型设计和训练策略,将预训练视频生成模型适应到NVS任务中,利用其丰富的视觉先验实现高质量视角合成。具体而言,我们将在视频模型中加入相机适配器以实现精确的相机控制。为增强3D物体的几何和外观特性,我们设计了法线图生成分支,并利用法线图特征通过注意力机制指导目标视角的合成,从而提高几何一致性。此外,我们应用像素空间监督以缓解潜在空间中的空间压缩导致的模糊外观。大量实验表明,OrbitNVS在GSO和OmniObject3D基准上显著优于先前方法,特别是在具有挑战性的单视角设置中(例如,PSNR提升+2.9 dB和+2.4 dB)。

英文摘要:

Novel View Synthesis (NVS) aims to generate unseen views of a 3D object given a limited number of known views. Existing methods often struggle to synthesize plausible views for unobserved regions, particularly under single-view input, and still face challenges in maintaining geometry- and appearance-consistency. To address these issues, we propose OrbitNVS, which reformulates NVS as an orbit video generation task. Through tailored model design and training strategies, we adapt a pre-trained video generation model to the NVS task, leveraging its rich visual priors to achieve high-quality view synthesis. Specifically, we incorporate camera adapters into the video model to enable accurate camera control. To enhance two key properties of 3D objects, geometry and appearance, we design a normal map generation branch and use normal map features to guide the synthesis of the target views via attention mechanism, thereby improving geometric consistency. Moreover, we apply a pixel-space supervision to alleviate blurry appearance caused by spatial compression in the latent space. Extensive experiments show that OrbitNVS significantly outperforms previous methods on the GSO and OmniObject3D benchmarks, especially in the challenging single-view setting (\eg, +2.9 dB and +2.4 dB PSNR).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑