arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.00908cs.CV
收藏

ZeroSmooth:用于高帧率视频生成的免训练扩散模型适配

ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video Generation

School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Tencent AI Lab(腾讯AI实验室)

Shaoshu Yang, Yong Zhang, Xiaodong Cun, Ying Shan, Ran He

AI总结:

本文提出一种即插即用的免训练视频插值方法ZeroSmooth,通过研究特征空间非线性并引入自级联架构与隐藏状态校正模块,将现有视频扩散模型转化为高帧率生成模型,保持时间一致性且性能媲美需大量资源训练的模型。

摘要:

近年来,视频生成取得了显著进展,尤其是自视频扩散模型问世以来。许多视频生成模型能够生成合理的合成视频,例如Stable Video Diffusion(SVD)。然而,由于GPU内存有限以及对大量帧建模的困难,大多数视频模型只能生成低帧率视频。训练视频始终以指定间隔均匀采样以进行时间压缩。以往的方法提升帧率的方式是,要么在像素空间训练视频插值模型作为后处理阶段,要么为特定的基础视频模型在潜在空间训练插值模型。在本文中,我们提出了一种针对生成式视频扩散模型的免训练视频插值方法,该方法能够以即插即用的方式泛化到不同模型。我们研究了视频扩散模型特征空间中的非线性,并将视频模型转化为自级联视频扩散模型,同时结合了设计的隐藏状态校正模块。提出自级联架构和校正模块旨在保持关键帧与插值帧之间的时间一致性。在多个流行视频模型上进行了广泛评估,以证明所提方法的有效性,特别是我们的免训练方法甚至可与由庞大计算资源和大规模数据集支持的已训练插值模型相媲美。

英文摘要:

Video generation has made remarkable progress in recent years, especially since the advent of the video diffusion models. Many video generation models can produce plausible synthetic videos, e.g., Stable Video Diffusion (SVD). However, most video models can only generate low frame rate videos due to the limited GPU memory as well as the difficulty of modeling a large set of frames. The training videos are always uniformly sampled at a specified interval for temporal compression. Previous methods promote the frame rate by either training a video interpolation model in pixel space as a postprocessing stage or training an interpolation model in latent space for a specific base video model. In this paper, we propose a training-free video interpolation method for generative video diffusion models, which is generalizable to different models in a plug-and-play manner. We investigate the non-linearity in the feature space of video diffusion models and transform a video model into a self-cascaded video diffusion model with incorporating the designed hidden state correction modules. The self-cascaded architecture and the correction module are proposed to retain the temporal consistency between key frames and the interpolated frames. Extensive evaluations are preformed on multiple popular video models to demonstrate the effectiveness of the propose method, especially that our training-free method is even comparable to trained interpolation models supported by huge compute resources and large-scale datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑