arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.16693cs.CV
收藏

I2V-Adapter:一种用于扩散模型的通用图像到视频适配器

I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models

University of Science and Technology of China(中国科学技术大学);Kuaishou Technology(快手科技);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)

Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Pengfei Wan, Di Zhang, Yufan Liu, Weiming Hu, Zhengjun Zha, Haibin Huang, Chongyang Ma

AI总结:

本文提出I2V-Adapter,通过跨帧注意力机制保留输入图像特征,在不改变预训练T2V模型的前提下实现高质量图像到视频生成,并引入帧相似性先验平衡运动幅度与稳定性。

摘要:

文本引导的图像到视频(I2V)生成旨在生成连贯的视频,该视频保留输入图像的身份特征,并在语义上与输入提示对齐。现有方法通常通过两种方式增强预训练的文本到视频(T2V)模型:要么在将图像与加噪视频帧在通道维度上拼接后输入模型,要么在交叉注意力模块中注入由预训练图像编码器产生的图像嵌入。然而,前一种方法通常需要改变预训练T2V模型的基础权重,从而限制了模型在开源社区中的兼容性,并破坏了模型的先验知识。同时,后一种方法通常无法保留输入图像的身份特征。我们提出了I2V-Adapter以克服这些限制。I2V-Adapter通过跨帧注意力机制,巧妙地将未加噪的输入图像传播到后续的加噪帧中,在不改变预训练T2V模型的情况下维持输入图像的身份特征。值得注意的是,I2V-Adapter仅引入少量可训练参数,显著降低了训练成本,并确保了与现有社区驱动的个性化模型和控制工具的兼容性。此外,我们提出了一种新颖的帧相似性先验,通过两个可调节的控制系数来平衡生成视频的运动幅度与稳定性。我们的实验结果表明,I2V-Adapter能够生成高质量视频。这种性能加上其敏捷性和适应性,代表了I2V领域的重大进步,特别是在个性化和可控应用方面。

英文摘要:

Text-guided image-to-video (I2V) generation aims to generate a coherent video that preserves the identity of the input image and semantically aligns with the input prompt. Existing methods typically augment pretrained text-to-video (T2V) models by either concatenating the image with noised video frames channel-wise before being fed into the model or injecting the image embedding produced by pretrained image encoders in cross-attention modules. However, the former approach often necessitates altering the fundamental weights of pretrained T2V models, thus restricting the model's compatibility within the open-source communities and disrupting the model's prior knowledge. Meanwhile, the latter typically fails to preserve the identity of the input image. We present I2V-Adapter to overcome such limitations. I2V-Adapter adeptly propagates the unnoised input image to subsequent noised frames through a cross-frame attention mechanism, maintaining the identity of the input image without any changes to the pretrained T2V model. Notably, I2V-Adapter only introduces a few trainable parameters, significantly alleviating the training cost and also ensures compatibility with existing community-driven personalized models and control tools. Moreover, we propose a novel Frame Similarity Prior to balance the motion amplitude and the stability of generated videos through two adjustable control coefficients. Our experimental results demonstrate that I2V-Adapter is capable of producing high-quality videos. This performance, coupled with its agility and adaptability, represents a substantial advancement in the field of I2V, particularly for personalized and controllable applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑