ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video Generation
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Tencent AI Lab(腾讯AI实验室)
Shaoshu Yang, Yong Zhang, Xiaodong Cun, Ying Shan, Ran He
近年来,视频生成取得了显著进展,尤其是自视频扩散模型问世以来。许多视频生成模型能够生成合理的合成视频,例如Stable Video Diffusion(SVD)。然而,由于GPU内存有限以及对大量帧建模的困难,大多数视频模型只能生成低帧率视频。训练视频始终以指定间隔均匀采样以进行时间压缩。以往的方法提升帧率的方式是,要么在像素空间训练视频插值模型作为后处理阶段,要么为特定的基础视频模型在潜在空间训练插值模型。在本文中,我们提出了一种针对生成式视频扩散模型的免训练视频插值方法,该方法能够以即插即用的方式泛化到不同模型。我们研究了视频扩散模型特征空间中的非线性,并将视频模型转化为自级联视频扩散模型,同时结合了设计的隐藏状态校正模块。提出自级联架构和校正模块旨在保持关键帧与插值帧之间的时间一致性。在多个流行视频模型上进行了广泛评估,以证明所提方法的有效性,特别是我们的免训练方法甚至可与由庞大计算资源和大规模数据集支持的已训练插值模型相媲美。
英文摘要:
Video generation has made remarkable progress in recent years, especially since the advent of the video diffusion models. Many video generation models can produce plausible synthetic videos, e.g., Stable Video Diffusion (SVD). However, most video models can only generate low frame rate videos due to the limited GPU memory as well as the difficulty of modeling a large set of frames. The training videos are always uniformly sampled at a specified interval for temporal compression. Previous methods promote the frame rate by either training a video interpolation model in pixel space as a postprocessing stage or training an interpolation model in latent space for a specific base video model. In this paper, we propose a training-free video interpolation method for generative video diffusion models, which is generalizable to different models in a plug-and-play manner. We investigate the non-linearity in the feature space of video diffusion models and transform a video model into a self-cascaded video diffusion model with incorporating the designed hidden state correction modules. The self-cascaded architecture and the correction module are proposed to retain the temporal consistency between key frames and the interpolated frames. Extensive evaluations are preformed on multiple popular video models to demonstrate the effectiveness of the propose method, especially that our training-free method is even comparable to trained interpolation models supported by huge compute resources and large-scale datasets.
RL-RIG: A Generative Spatial Reasoner via Intrinsic Reflection
Zhiyuan College, Shanghai Jiao Tong University(上海交通大学紫元学院);Huazhong University of Science and Technology(华中科技大学);National University of Singapore(新加坡国立大学);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);Shanghai AI Laboratory(上海人工智能实验室)
PeQuENet: Perceptual Quality Enhancement of Compressed Video with Adaptation- and Attention-based Network
Xidian University(西安电子科技大学);Universitat Autònoma de Barcelona(巴塞罗那自治大学);BBC Research & Development(英国广播公司研发部);Northwestern Polytechnical University(西北工业大学)
Robust Source-Free Domain Adaptation for Medical Image Segmentation based on Curriculum Learning
Shanghai Digital Medicine Innovation Center, Ruijin Hospital(上海数字医学创新中心,瑞金医院);Tsinghua University(清华大学);Medical AI ReSearch (MARS) Group, Guangxi Key Laboratory for Genomic and Personalized Medicine, Guangxi Medical University(医学AI研究组(MARS), 广西基因组与个性化医学重点实验室,广西医科大学);Jarvis Research Center, Tencent Youtu Lab(Jarvis研究中心,腾讯优图实验室);University of California, Berkeley(加州大学伯克利分校);Westlake University(西湖大学)
Dynamic and Compressive Adaptation of Transformers From Images to Videos
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Platform and Content Group (PCG), Tencent(腾讯平台与内容事业群);Shanghai AI Lab(上海人工智能实验室)