arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.01203cs.CV
收藏

基于扩散模型的视频插帧

Video Interpolation with Diffusion Models

Google Research(谷歌研究院);Google DeepMind(谷歌DeepMind)

Siddhant Jain, Daniel Watson, Eric Tabellion, Aleksander Hołyński, Ben Poole, Janne Kontkanen

AI总结:

本文提出基于级联扩散模型的视频插帧方法 VIDIM,通过低分辨率生成再超分的方式处理复杂非线性运动,并利用无分类器引导和高分辨率条件实现高保真快速插帧。

摘要:

我们提出 VIDIM,一种用于视频插帧的生成模型,能够在给定起始帧和结束帧的情况下生成短视频。为了实现高保真度并生成输入数据中未见的运动,VIDIM 使用级联扩散模型,先在低分辨率下生成目标视频,再以低分辨率生成视频为条件生成高分辨率视频。我们将 VIDIM 与以往最先进的视频插帧方法进行比较,并展示这些方法在底层运动复杂、非线性或模糊的大多数场景中会失败,而 VIDIM 能够轻松处理此类情况。我们还进一步证明,对起始帧和结束帧使用无分类器引导,并在不增加额外参数的情况下以原始高分辨率帧为条件训练超分辨率模型,可以获得高保真结果。VIDIM 采样速度快,因为它对所有待生成帧进行联合去噪,每个扩散模型只需不到十亿参数即可产生令人信服的结果,并且在更大参数量下仍具有良好的可扩展性和质量提升。

英文摘要:

We present VIDIM, a generative model for video interpolation, which creates short videos given a start and end frame. In order to achieve high fidelity and generate motions unseen in the input data, VIDIM uses cascaded diffusion models to first generate the target video at low resolution, and then generate the high-resolution video conditioned on the low-resolution generated video. We compare VIDIM to previous state-of-the-art methods on video interpolation, and demonstrate how such works fail in most settings where the underlying motion is complex, nonlinear, or ambiguous while VIDIM can easily handle such cases. We additionally demonstrate how classifier-free guidance on the start and end frame and conditioning the super-resolution model on the original high-resolution frames without additional parameters unlocks high-fidelity results. VIDIM is fast to sample from as it jointly denoises all the frames to be generated, requires less than a billion parameters per diffusion model to produce compelling results, and still enjoys scalability and improved quality at larger parameter counts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑