arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.09151cs.CVcs.AI
收藏

Reangle-A-Video:将4D视频生成视为视频到视频的转换

Reangle-A-Video: 4D Video Generation as Video-to-Video Translation

KAIST(韩国科学技术院);Adobe Research(Adobe研究)

Hyeonho Jeong, Suhyeon Lee, Jong Chul Ye

AI总结:

Reangle-A-Video提出将多视角视频生成重构为视频到视频转换,通过两阶段方法利用扩散先验,在实验中超越现有方法,实现高效的多视角视频生成。

摘要:

我们提出了Reangle-A-Video,一个用于从单个输入视频生成同步多视角视频的统一框架。与主流方法在大规模4D数据集上训练多视角视频扩散模型不同,我们的方法将多视角视频生成任务重新定义为视频到视频的转换,利用公开可用的图像和视频扩散先验。本质上,Reangle-A-Video分两个阶段运行。(1) 多视角运动学习:一个图像到视频的扩散变换器以自监督方式同步微调,从一组扭曲视频中提炼视角不变的运动。(2) 多视角一致的图像到图像转换:在推理时使用DUSt3R的跨视角一致性引导下,输入视频的第一帧被扭曲并修复到各种相机视角,生成多视角一致的起始图像。在静态视角传输和动态相机控制上的大量实验表明,Reangle-A-Video超越了现有方法,为多视角视频生成建立了新的解决方案。我们将公开发布我们的代码和数据。项目页面:https://hyeonho99.github.io/reangle-a-video/

英文摘要:

We introduce Reangle-A-Video, a unified framework for generating synchronized multi-view videos from a single input video. Unlike mainstream approaches that train multi-view video diffusion models on large-scale 4D datasets, our method reframes the multi-view video generation task as video-to-videos translation, leveraging publicly available image and video diffusion priors. In essence, Reangle-A-Video operates in two stages. (1) Multi-View Motion Learning: An image-to-video diffusion transformer is synchronously fine-tuned in a self-supervised manner to distill view-invariant motion from a set of warped videos. (2) Multi-View Consistent Image-to-Images Translation: The first frame of the input video is warped and inpainted into various camera perspectives under an inference-time cross-view consistency guidance using DUSt3R, generating multi-view consistent starting images. Extensive experiments on static view transport and dynamic camera control show that Reangle-A-Video surpasses existing methods, establishing a new solution for multi-view video generation. We will publicly release our code and data. Project page: https://hyeonho99.github.io/reangle-a-video/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑