arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.11325cs.CVeess.IV
收藏

MoVideo:基于扩散模型的运动感知视频生成

MoVideo: Motion-Aware Video Generation with Diffusion Models

ETH Zurich(苏黎世联邦理工学院);Meta Inc.(元公司);Nanjing University(南京大学);University of Würzburg(维尔茨堡大学);CAIDAS(卡伊达斯研究中心);IFI(信息与通信工程研究所)

Jingyun Liang, Yuchen Fan, Kai Zhang, Radu Timofte, Luc Van Gool, Rakesh Ranjan

AI总结:

提出运动感知视频生成框架MoVideo,通过视频深度和光流两方面建模运动,利用时空扩散模型生成深度与光流指导潜在空间视频生成,并在解码时用光流对齐细化帧,在文本到视频和图像到视频任务上达到SOTA。

摘要:

尽管近年来在使用扩散模型进行视频生成方面取得了巨大进展,但大多数方法只是图像生成框架的简单扩展,未能明确考虑视频与图像之间的关键差异之一,即运动。在本文中,我们提出了一种新颖的运动感知视频生成(MoVideo)框架,该框架从两个方面考虑运动:视频深度和光流。前者通过逐帧的对象距离和空间布局来调节运动,而后者通过跨帧对应关系来描述运动,这有助于保留精细细节并提高时间一致性。更具体地说,给定一个关键帧(该关键帧是已有的或从文本提示生成的),我们首先设计了一个具有时空模块的扩散模型,以生成视频深度和相应的光流。然后,在深度、基于光流的扭曲潜在视频和计算出的遮挡掩模的指导下,通过另一个时空扩散模型在潜在空间中生成视频。最后,我们再次使用光流来对齐和细化不同的帧,以便更好地将视频从潜在空间解码到像素空间。在实验中,MoVideo在文本到视频和图像到视频生成方面均取得了最先进的结果,显示出良好的提示一致性、帧一致性和视觉质量。

英文摘要:

While recent years have witnessed great progress on using diffusion models for video generation, most of them are simple extensions of image generation frameworks, which fail to explicitly consider one of the key differences between videos and images, i.e., motion. In this paper, we propose a novel motion-aware video generation (MoVideo) framework that takes motion into consideration from two aspects: video depth and optical flow. The former regulates motion by per-frame object distances and spatial layouts, while the later describes motion by cross-frame correspondences that help in preserving fine details and improving temporal consistency. More specifically, given a key frame that exists or generated from text prompts, we first design a diffusion model with spatio-temporal modules to generate the video depth and the corresponding optical flows. Then, the video is generated in the latent space by another spatio-temporal diffusion model under the guidance of depth, optical flow-based warped latent video and the calculated occlusion mask. Lastly, we use optical flows again to align and refine different frames for better video decoding from the latent space to the pixel space. In experiments, MoVideo achieves state-of-the-art results in both text-to-video and image-to-video generation, showing promising prompt consistency, frame consistency and visual quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑