arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.14864cs.CV
收藏

视频扩散模型是免训练的运动解释器与控制器

Video Diffusion Models are Training-free Motion Interpreter and Controller

S-Lab, Nanyang Technological University(南洋理工大学S-Lab);Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所)

Zeqi Xiao, Yifan Zhou, Shuai Yang, Xingang Pan

AI总结:

本文发现视频扩散模型中已存在鲁棒的运动感知特征,通过PCA提取并过滤出免训练的运动特征MOFT,提出了一个架构无关的免训练视频运动控制框架,在下游任务中表现出竞争力。

摘要:

视频生成主要旨在跨帧建模真实且定制的运动,因此理解和控制运动成为一个关键课题。大多数基于扩散模型的视频运动研究专注于采用基于训练的范式进行运动定制,然而这需要大量的训练资源,并且需要为不同的模型重新训练。至关重要的是,这些方法没有探讨视频扩散模型如何在其特征中编码跨帧运动信息,缺乏其有效性的可解释性和透明度。为了回答这个问题,本文引入了一种新颖的视角来理解、定位和操纵视频扩散模型中的运动感知特征。通过使用主成分分析(PCA)进行分析,我们的工作揭示了视频扩散模型中已经存在鲁棒的运动感知特征。我们通过消除内容相关信息并过滤运动通道,提出了一种新的运动特征(MOtion FeaTure, MOFT)。MOFT提供了一系列独特的优势,包括能够编码具有清晰可解释性的全面运动信息、无需训练即可提取,以及跨不同架构的泛化能力。利用MOFT,我们提出了一种新颖的免训练视频运动控制框架。我们的方法在生成自然且忠实的运动方面表现出具有竞争力的性能,为各种下游任务提供了与架构无关的见解和适用性。

英文摘要:

Video generation primarily aims to model authentic and customized motion across frames, making understanding and controlling the motion a crucial topic. Most diffusion-based studies on video motion focus on motion customization with training-based paradigms, which, however, demands substantial training resources and necessitates retraining for diverse models. Crucially, these approaches do not explore how video diffusion models encode cross-frame motion information in their features, lacking interpretability and transparency in their effectiveness. To answer this question, this paper introduces a novel perspective to understand, localize, and manipulate motion-aware features in video diffusion models. Through analysis using Principal Component Analysis (PCA), our work discloses that robust motion-aware feature already exists in video diffusion models. We present a new MOtion FeaTure (MOFT) by eliminating content correlation information and filtering motion channels. MOFT provides a distinct set of benefits, including the ability to encode comprehensive motion information with clear interpretability, extraction without the need for training, and generalizability across diverse architectures. Leveraging MOFT, we propose a novel training-free video motion control framework. Our method demonstrates competitive performance in generating natural and faithful motion, providing architecture-agnostic insights and applicability in a variety of downstream tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑