arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.15977cs.CV
收藏

Motion-I2V:基于显式运动建模的一致性可控图像到视频生成

Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling

The Chinese University of Hong Kong(香港中文大学);SenseTime Research(商汤科技研究院);NVIDIA AI Technology Center(英伟达人工智能技术中心);Tsinghua University(清华大学);Centre for Perceptual and Interactive Intelligence (CPII)(感知与交互智能中心);Shanghai AI Laboratory(上海人工智能实验室);Avolution AI

Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, Hongsheng Li

AI总结:

本研究提出Motion-I2V框架,将图像到视频生成分解为显式运动预测与运动增强时序注意力两阶段,提升了生成视频的一致性与可控性,还支持零样本视频转换。

摘要:

我们提出了Motion-I2V,一种用于实现一致性可控图像到视频生成(I2V)的新型框架。与直接学习复杂图像到视频映射的现有方法不同,Motion-I2V将I2V任务分解为两个阶段,结合显式运动建模实现。第一阶段,我们提出了基于扩散的运动场预测器,专注于推导参考图像像素的运动轨迹。第二阶段,我们提出运动增强时序注意力模块,用于增强视频潜扩散模型中有限的一维时序注意力能力。该模块可在第一阶段预测轨迹的引导下,将参考图像特征有效传播到生成的帧中。与现有方法相比,即使存在大幅运动和视角变化,Motion-I2V也能生成一致性更强的视频。通过为第一阶段训练稀疏轨迹ControlNet(控制网络),Motion-I2V可支持用户通过稀疏轨迹和区域标注,精确控制运动轨迹与运动区域,相比仅依赖文本指令的方式,为I2V流程提供了更强的可控性。此外,Motion-I2V的第二阶段天然支持零样本视频到视频转换。定性与定量对比均表明,在一致性可控图像到视频生成任务上,Motion-I2V优于现有方法。项目页面详见https://xiaoyushi97.github.io/Motion-I2V/。

英文摘要:

We introduce Motion-I2V, a novel framework for consistent and controllable image-to-video generation (I2V). In contrast to previous methods that directly learn the complicated image-to-video mapping, Motion-I2V factorizes I2V into two stages with explicit motion modeling. For the first stage, we propose a diffusion-based motion field predictor, which focuses on deducing the trajectories of the reference image's pixels. For the second stage, we propose motion-augmented temporal attention to enhance the limited 1-D temporal attention in video latent diffusion models. This module can effectively propagate reference image's feature to synthesized frames with the guidance of predicted trajectories from the first stage. Compared with existing methods, Motion-I2V can generate more consistent videos even at the presence of large motion and viewpoint variation. By training a sparse trajectory ControlNet for the first stage, Motion-I2V can support users to precisely control motion trajectories and motion regions with sparse trajectory and region annotations. This offers more controllability of the I2V process than solely relying on textual instructions. Additionally, Motion-I2V's second stage naturally supports zero-shot video-to-video translation. Both qualitative and quantitative comparisons demonstrate the advantages of Motion-I2V over prior approaches in consistent and controllable image-to-video generation. Please see our project page at https://xiaoyushi97.github.io/Motion-I2V/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑