无需训练的运动因子分解用于组合视频生成
Training-free Motion Factorization for Compositional Video Generation
AI总结：
本文提出一种无需训练的运动因子分解框架，将复杂运动分解为静止、刚体和非刚体运动三类，通过规划先于生成的范式，提升视频生成中运动多样性的表现。
摘要：
组合视频生成旨在合成具有多样外观和运动的多个实例。然而，当前方法主要关注语义绑定，忽略了提示中指定的多样化运动类别。本文提出了一种运动因子分解框架，将复杂运动分解为三个主要类别：静止、刚体和非刚体运动。具体而言，我们的框架遵循规划先于生成的范式。在规划阶段，我们通过运动图推理解释运动规律，以获得每个实例的形状和位置的帧级变化。这通过将用户提示组织为实例及其相互作用的结构化表示，缓解了语义模糊性。在生成阶段，我们以解耦的方式调节不同运动类别的合成。在运动提示条件下，指导分支稳定静止区域的外观，保留刚体几何，并正则化局部非刚体变形。关键的是，我们的两个模块是模型无关的，可以无缝集成到各种扩散模型架构中。广泛的实验表明，我们的框架在现实世界基准上实现了出色的运动合成性能。代码可在https://github.com/ZixuanWang0525/MF-CVG获得。
英文摘要：
Compositional video generation aims to synthesize multiple instances with diverse appearance and motion. However, current approaches mainly focus on binding semantics, neglecting to understand diverse motion categories specified in prompts. In this paper, we propose a motion factorization framework that decomposes complex motion into three primary categories: motionlessness, rigid motion, and non-rigid motion. Specifically, our framework follows a planning before generation paradigm. (1) During planning, we reason about motion laws on the motion graph to obtain frame-wise changes in the shape and position of each instance. This alleviates semantic ambiguities in the user prompt by organizing it into a structured representation of instances and their interactions. (2) During generation, we modulate the synthesis of distinct motion categories in a disentangled manner. Conditioned on the motion cues, guidance branches stabilize appearance in motionless regions, preserve rigid-body geometry, and regularize local non-rigid deformations. Crucially, our two modules are model-agnostic, which can be seamlessly incorporated into various diffusion model architectures. Extensive experiments demonstrate that our framework achieves impressive performance in motion synthesis on real-world benchmarks. Code is available at https://github.com/ZixuanWang0525/MF-CVG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
