MotionFollower：通过轻量级分数引导扩散进行视频运动编辑
MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusion
AI总结：
本文提出MotionFollower，一种轻量级分数引导扩散模型，通过双分支架构和轻量级信号控制器实现视频运动编辑，在保留外观和背景的同时显著降低GPU内存占用并支持大相机移动。
摘要：
尽管基于扩散的视频编辑模型在改变视频属性方面取得了令人瞩目的进展，但在修改运动信息的同时保留原始主角外观和背景方面的探索仍然有限。在本文中，我们提出了MotionFollower，一种用于视频运动编辑的轻量级分数引导扩散模型。为了向去噪过程引入条件控制，MotionFollower利用了我们提出的两个轻量级信号控制器，一个用于姿态，另一个用于外观，两者均由卷积块组成，不涉及繁重的注意力计算。此外，我们设计了一种基于双分支架构（包括重建分支和编辑分支）的分数引导原则，显著增强了对纹理细节和复杂背景的建模能力。具体而言，我们在分数估计过程中施加了几种一致性正则化和损失。由此产生的梯度为中间潜变量注入适当的引导，迫使模型在不干扰运动修改的情况下保留原始背景细节和主角外观。实验定性和定量地证明了MotionFollower具有竞争力的运动编辑能力。与最先进的运动编辑模型MotionEditor相比，MotionFollower在提供卓越运动编辑性能的同时，实现了约80%的GPU内存减少，并且独家支持大型相机移动和动作。
英文摘要：
Despite impressive advancements in diffusion-based video editing models in altering video attributes, there has been limited exploration into modifying motion information while preserving the original protagonist's appearance and background. In this paper, we propose MotionFollower, a lightweight score-guided diffusion model for video motion editing. To introduce conditional controls to the denoising process, MotionFollower leverages two of our proposed lightweight signal controllers, one for poses and the other for appearances, both of which consist of convolution blocks without involving heavy attention calculations. Further, we design a score guidance principle based on a two-branch architecture, including the reconstruction and editing branches, which significantly enhance the modeling capability of texture details and complicated backgrounds. Concretely, we enforce several consistency regularizers and losses during the score estimation. The resulting gradients thus inject appropriate guidance to the intermediate latents, forcing the model to preserve the original background details and protagonists' appearances without interfering with the motion modification. Experiments demonstrate the competitive motion editing ability of MotionFollower qualitatively and quantitatively. Compared with MotionEditor, the most advanced motion editing model, MotionFollower achieves an approximately 80% reduction in GPU memory while delivering superior motion editing performance and exclusively supporting large camera movements and actions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
