arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.09146cs.CV
收藏

基于连续分段仿射的运动模型用于图像动画

Continuous Piecewise-Affine Based Motion Model for Image Animation

Shanghai Jiao Tong University(上海交通大学);East China Normal University(华东师范大学)

Hexiang Wang, Fengqi Liu, Qianyu Zhou, Ran Yi, Xin Tan, Lizhuang Ma

AI总结:

本文提出基于连续分段仿射变换的图像动画运动模型,结合SAM引导关键点语义损失和结构对齐损失,在高表达性微分同胚空间中提升运动迁移质量,并在多个数据集上验证其优于现有方法。

摘要:

图像动画旨在根据驱动视频让静态图像动起来,并创建可用于动画、娱乐和教育等多种用途的引人入胜的视觉内容。最近的免监督方法利用基于关键点的仿射和薄板样条变换,将驱动帧中的运动迁移到源图像上。然而,受限于所用变换的表达能力,当驱动帧中的运动与源图像之间的差距较大时,这些方法往往产生较差的结果。为了解决这一问题,我们提出在高表达性的微分同胚空间中建模从源图像到驱动帧的运动。首先,我们引入基于连续分段仿射(CPAB)的变换来建模运动,并提出一种精心设计的推理算法,以从控制关键点生成CPAB变换。其次,我们提出一种SAM引导的关键点语义损失,以进一步约束关键点提取过程,并提高源图像和驱动图像上对应关键点之间的语义一致性。最后,我们设计了一种结构对齐损失,用于对齐从驱动图像和生成图像中提取的结构相关特征,从而帮助生成器生成与驱动动作更一致的结果。在四个数据集上的大量实验定量和定性地证明了我们的方法相对于最先进竞争对手的有效性。代码将公开在:https://github.com/DevilPG/AAAI2024-CPABMM。

英文摘要:

Image animation aims to bring static images to life according to driving videos and create engaging visual content that can be used for various purposes such as animation, entertainment, and education. Recent unsupervised methods utilize affine and thin-plate spline transformations based on keypoints to transfer the motion in driving frames to the source image. However, limited by the expressive power of the transformations used, these methods always produce poor results when the gap between the motion in the driving frame and the source image is large. To address this issue, we propose to model motion from the source image to the driving frame in highly-expressive diffeomorphism spaces. Firstly, we introduce Continuous Piecewise-Affine based (CPAB) transformation to model the motion and present a well-designed inference algorithm to generate CPAB transformation from control keypoints. Secondly, we propose a SAM-guided keypoint semantic loss to further constrain the keypoint extraction process and improve the semantic consistency between the corresponding keypoints on the source and driving images. Finally, we design a structure alignment loss to align the structure-related features extracted from driving and generated images, thus helping the generator generate results that are more consistent with the driving action. Extensive experiments on four datasets demonstrate the effectiveness of our method against state-of-the-art competitors quantitatively and qualitatively. Code will be publicly available at: https://github.com/DevilPG/AAAI2024-CPABMM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑