Text-guided image-to-video (I2V) generation aims to generate a coherent video that preserves the identity of the input image and semantically aligns with the input prompt. Existing methods typically augment pretrained text-to-video (T2V) models by either concatenating the image with noised video frames channel-wise before being fed into the model or injecting the image embedding produced by pretrained image encoders in cross-attention modules. However, the former approach often necessitates altering the fundamental weights of pretrained T2V models, thus restricting the model's compatibility within the open-source communities and disrupting the model's prior knowledge. Meanwhile, the latter typically fails to preserve the identity of the input image. We present I2V-Adapter to overcome such limitations. I2V-Adapter adeptly propagates the unnoised input image to subsequent noised frames through a cross-frame attention mechanism, maintaining the identity of the input image without any changes to the pretrained T2V model. Notably, I2V-Adapter only introduces a few trainable parameters, significantly alleviating the training cost and also ensures compatibility with existing community-driven personalized models and control tools. Moreover, we propose a novel Frame Similarity Prior to balance the motion amplitude and the stability of generated videos through two adjustable control coefficients. Our experimental results demonstrate that I2V-Adapter is capable of producing high-quality videos. This performance, coupled with its agility and adaptability, represents a substantial advancement in the field of I2V, particularly for personalized and controllable applications.
IP-Adapter Is All You Need: Towards Fine-Tuning-Free Diffusion-Based Talking Face Generation
Information Engineering University(信息工程大学);Huai’an University(淮安大学);Chongqing University of Post and Telecommunications(重庆邮电大学);Nankai University(南开大学)
Efficient Temporal Consistency in Diffusion-Based Video Editing with Adaptor Modules: A Theoretical Framework
Emory University(埃默里大学);University of Minnesota—Twin Cities(明尼苏达大学双城分校);Peking University(北京大学);University of Electronic Science and Technology of China(电子科技大学);University of Warwick(华威大学);Washington University(华盛顿大学);Tsinghua University(清华大学);Cornell University(康奈尔大学);Santa Clara University(圣克拉拉大学);Stanford University(斯坦福大学);Hong Kong University of Science and Technology(香港科技大学)
LILAC: Layer-Wise Independent LoRAs and Cascaded Conditioning for Multi-Concept Customization of Diffusion Models
Adobe Research, Romania(罗马尼亚Adobe研究院);Department of Computer Science, University of Bucharest, Romania(罗马尼亚布加勒斯特大学计算机科学系);International Computer High School of Bucharest, Romania(罗马尼亚布加勒斯特国际计算机高中)
LAFR: Efficient Diffusion-based Blind Face Restoration via Latent Codebook Alignment Adapter
School of Electronic and Computer Engineering, Peking University(电子与计算机工程学院,北京大学);Computer Vision Lab, CAIDAS & IFI, University of Würzburg(计算机视觉实验室,乌尔姆大学)
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
ARC Lab, Tencent PCG(腾讯平台与内容事业群ARC实验室);University of Macau(澳门大学);Shenzhen Institute of Advanced Technology(深圳先进技术研究院);Peking University Shenzhen Graduate School(北京大学深圳研究生院)
Causal-Adapter: Taming Text-to-Image Diffusion for Faithful Counterfactual Generation
Centre for AI, DS\&AI, Astrazeneca, UK(英国阿斯利康人工智能中心);Institute for Imaging, Data and Communications (IDCOM), School of Engineering, University of Edinburgh, Edinburgh, UK(爱丁堡大学工程学院影像、数据与通信研究所);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
HiCAST: Highly Customized Arbitrary Style Transfer with Adapter Enhanced Diffusion Models
Harbin Institute of Technology(哈尔滨工业大学);Baidu Research(百度研究院);Baidu Inc.(百度公司);University of Chinese Academy of Sciences(中国科学院大学);ETH Zurich(苏黎世联邦理工学院)