OmniRoam:通过长视界全景视频生成实现世界漫游
OmniRoam: World Wandering via Long-Horizon Panoramic Video Generation
AI总结:
本文提出OmniRoam框架,利用全景视频生成实现长视界场景漫游,通过预览和精炼阶段提升视频质量和一致性,实验表明其在视觉质量、可控性和长期一致性方面优于现有方法。
摘要:
近年来,使用视频生成模型建模场景引起了越来越多的研究兴趣。然而,大多数现有方法依赖于视角视频模型,只能合成有限的场景观察,导致完整性和全局一致性的问题。我们提出OmniRoam,一种可控的全景视频生成框架,利用每帧场景覆盖丰富和全景表示中固有的长期空间和时间一致性,实现长视界场景漫游。我们的框架首先通过轨迹控制视频生成模型从给定的输入图像或视频快速生成场景概览。然后,在精炼阶段,该视频被时间扩展和空间上采样,以生成长距离、高分辨率视频,从而实现高保真的世界漫游。为了训练我们的模型,我们引入了两个全景视频数据集,结合了合成和真实世界拍摄的视频。实验表明,我们的框架在视觉质量和可控性以及长期场景一致性方面,无论是定性还是定量,都优于现有方法。我们进一步展示了该框架的几个扩展,包括实时视频生成和3D重建。代码可在https://github.com/yuhengliu02/OmniRoam上获得。
英文摘要:
Modeling scenes using video generation models has garnered growing research interest in recent years. However, most existing approaches rely on perspective video models that synthesize only limited observations of a scene, leading to issues of completeness and global consistency. We propose OmniRoam, a controllable panoramic video generation framework that exploits the rich per-frame scene coverage and inherent long-term spatial and temporal consistency of panoramic representation, enabling long-horizon scene wandering. Our framework begins with a preview stage, where a trajectory-controlled video generation model creates a quick overview of the scene from a given input image or video. Then, in the refine stage, this video is temporally extended and spatially upsampled to produce long-range, high-resolution videos, thus enabling high-fidelity world wandering. To train our model, we introduce two panoramic video datasets that incorporate both synthetic and real-world captured videos. Experiments show that our framework consistently outperforms state-of-the-art methods in terms of visual quality, controllability, and long-term scene consistency, both qualitatively and quantitatively. We further showcase several extensions of this framework, including real-time video generation and 3D reconstruction. Code is available at https://github.com/yuhengliu02/OmniRoam.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。