arXiv 2405.044962024-10-15cs.CV

Edit-Your-Motion：用于视频动作编辑的时空扩散解耦学习

Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editing

Xidian University（西安电子科技大学）；School of Artificial Intelligence（人工智能学院）；Key Laboratory of Intelligent Perception and Image Understanding of the Ministry of Education of China（智能感知与图像理解教育部重点实验室）；International Research Center of Intelligent Perception and Computation（智能感知与计算国际研究中心）

Yi Zuo, Lingling Li, Licheng Jiao, Fang Liu, Xu Liu, Wenping Ma, Shuyuan Yang, Yuwei Guo

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Edit-Your-Motion视频动作编辑方法，通过DDIM inversion初始化噪声、Motion Attention Module和时空两阶段学习策略(STL)解耦动作与外观，有效解决未见场景下的重影和身体扭曲问题，显著提升了编辑效果。

摘要：

现有的基于扩散的方法在人体动作编辑方面取得了令人瞩目的成果。然而，这些方法在未见过的真实场景中经常出现明显的重影和身体扭曲。本文提出了Edit-Your-Motion，这是一种通过对未见场景进行一次性微调来解决上述挑战的视频动作编辑方法。具体而言，首先利用DDIM inversion初始化噪声以保留源视频的外观，并设计了轻量级的动作注意力适配器模块来增强动作保真度。DDIM inversion旨在通过估计源视频的预测噪声来获取隐式表示，作为采样过程的起点，确保源视频与编辑后视频之间的外观一致性。Motion Attention Module (MA)通过解决骨架特征与外观特征之间的冲突，增强了模型的动作编辑能力。其次，为了有效解耦源视频的动作与外观，设计了时空两阶段学习策略(STL)。第一阶段专注于学习人体动作的时间特征，并提出recurrent causal attention (RCA)以确保视频帧之间的一致性。第二阶段将重点转向学习源视频的外观特征。借助Edit-Your-Motion，用户可以编辑源视频中人物的动作，创造更具吸引力和多样化的内容。大量定性和定量实验以及用户偏好研究表明，Edit-Your-Motion优于其他方法。

英文摘要：

Existing diffusion-based methods have achieved impressive results in human motion editing. However, these methods often exhibit significant ghosting and body distortion in unseen in-the-wild cases. In this paper, we introduce Edit-Your-Motion, a video motion editing method that tackles these challenges through one-shot fine-tuning on unseen cases. Specifically, firstly, we utilized DDIM inversion to initialize the noise, preserving the appearance of the source video and designed a lightweight motion attention adapter module to enhance motion fidelity. DDIM inversion aims to obtain the implicit representations by estimating the prediction noise from the source video, which serves as a starting point for the sampling process, ensuring the appearance consistency between the source and edited videos. The Motion Attention Module (MA) enhances the model's motion editing ability by resolving the conflict between the skeleton features and the appearance features. Secondly, to effectively decouple motion and appearance of source video, we design a spatio-temporal two-stage learning strategy (STL). In the first stage, we focus on learning temporal features of human motion and propose recurrent causal attention (RCA) to ensure consistency between video frames. In the second stage, we shift focus on learning the appearance features of the source video. With Edit-Your-Motion, users can edit the motion of humans in the source video, creating more engaging and diverse content. Extensive qualitative and quantitative experiments, along with user preference studies, show that Edit-Your-Motion outperforms other methods.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。