arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.03552cs.CV
收藏

Imagine360:基于透视锚点的沉浸式360视频生成

Imagine360: Immersive 360 Video Generation from Perspective Anchor

The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Nanyang Technological University(南洋理工大学)

Jing Tan, Shuai Yang, Tong Wu, Jingwen He, Yuwei Guo, Ziwei Liu, Dahua Lin

AI总结:

本文提出Imagine360框架,利用双分支去噪、对跖掩码和高程感知设计,从透视锚点视频生成高质量、运动连贯的360°全景视频。

摘要:

360°视频提供超沉浸式体验,使观看者能够从完整360度探索动态场景。为了以360°视频格式实现更用户友好且个性化的内容创作,我们寻求将标准透视视频提升为360°等距柱状投影视频。为此,我们提出Imagine360,这是首个从透视视频到360°视频的生成框架,可基于视频锚点创建具有丰富多样运动模式的高质量360°视频。Imagine360通过若干关键设计,从有限的360°视频数据中学习细粒度球面视觉与运动模式。1)首先,我们采用双分支设计,包括透视视频去噪分支和全景视频去噪分支,为360°视频生成提供局部和全局约束;其中运动模块和空间LoRA层在扩展的网络360°视频上进行微调。2)此外,设计了对跖掩码以捕获长程运动依赖,增强跨半球对跖像素之间的反向相机运动。3)为处理多样的透视视频输入,我们提出高程感知设计,以适应因帧间高程变化而导致的不同视频掩码。大量实验表明,在现有最先进的360°视频生成方法中,Imagine360实现了更优的图形质量和运动一致性。我们相信Imagine360有潜力推动个性化、沉浸式360°视频创作的发展。

英文摘要:

$360^\circ$ videos offer a hyper-immersive experience that allows the viewers to explore a dynamic scene from full 360 degrees. To achieve more user-friendly and personalized content creation in $360^\circ$ video format, we seek to lift standard perspective videos into $360^\circ$ equirectangular videos. To this end, we introduce Imagine360, the first perspective-to-$360^\circ$ video generation framework that creates high-quality $360^\circ$ videos with rich and diverse motion patterns from video anchors. Imagine360 learns fine-grained spherical visual and motion patterns from limited $360^\circ$ video data with several key designs. 1) Firstly we adopt the dual-branch design, including a perspective and a panorama video denoising branch to provide local and global constraints for $360^\circ$ video generation, with motion module and spatial LoRA layers fine-tuned on extended web $360^\circ$ videos. 2) Additionally, an antipodal mask is devised to capture long-range motion dependencies, enhancing the reversed camera motion between antipodal pixels across hemispheres. 3) To handle diverse perspective video inputs, we propose elevation-aware designs that adapt to varying video masking due to changing elevations across frames. Extensive experiments show Imagine360 achieves superior graphics quality and motion coherence among state-of-the-art $360^\circ$ video generation methods. We believe Imagine360 holds promise for advancing personalized, immersive $360^\circ$ video creation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑