arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2504.02451cs.CV
收藏

ConMo:用于零样本运动迁移的可控运动解耦与重组

ConMo: Controllable Motion Disentanglement and Recomposition for Zero-Shot Motion Transfer

Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Beijing University of Posts and Telecommunications(北京邮电大学)

Jiayi Gao, Zijin Yin, Changcheng Hua, Yuxin Peng, Kongming Liang, Zhanyu Ma, Jun Guo, Yang Liu

AI总结:

提出零样本运动迁移框架ConMo,通过解耦重组主体与摄像机运动,结合软引导机制,显著提升多主体场景下的运动控制精度、保真度与语义一致性。

摘要:

文本到视频(T2V)生成技术的发展使得运动迁移成为可能,从而能够基于现有素材控制视频运动。然而,当前方法存在两个局限性:1)难以处理多主体视频,无法迁移特定主体的运动;2)在迁移至具有不同形状的主体时,难以保持运动的多样性与准确性。为克服这些问题,我们提出了ConMo,这是一个零样本框架,能够解耦并重组主体运动与摄像机运动。ConMo仅利用主体掩码,从源视频的复杂轨迹中分离出单个主体和背景运动线索,并将其重新组合以生成目标视频。该方法实现了跨不同主体的更准确运动控制,并提升了多主体场景下的性能。此外,我们在重组阶段提出了软引导机制,通过控制原始运动的保留程度来调整形状约束,从而辅助主体形状适应和语义转换。与以往方法不同,ConMo解锁了广泛的应用,包括主体大小和位置编辑、主体移除、语义修改以及摄像机运动模拟。大量实验表明,ConMo在运动保真度和语义一致性上显著优于当前最先进的方法。代码可在 https://github.com/Andyplus1/ConMo 获取。

英文摘要:

The development of Text-to-Video (T2V) generation has made motion transfer possible, enabling the control of video motion based on existing footage. However, current methods have two limitations: 1) struggle to handle multi-subjects videos, failing to transfer specific subject motion; 2) struggle to preserve the diversity and accuracy of motion as transferring to subjects with varying shapes. To overcome these, we introduce \textbf{ConMo}, a zero-shot framework that disentangle and recompose the motions of subjects and camera movements. ConMo isolates individual subject and background motion cues from complex trajectories in source videos using only subject masks, and reassembles them for target video generation. This approach enables more accurate motion control across diverse subjects and improves performance in multi-subject scenarios. Additionally, we propose soft guidance in the recomposition stage which controls the retention of original motion to adjust shape constraints, aiding subject shape adaptation and semantic transformation. Unlike previous methods, ConMo unlocks a wide range of applications, including subject size and position editing, subject removal, semantic modifications, and camera motion simulation. Extensive experiments demonstrate that ConMo significantly outperforms state-of-the-art methods in motion fidelity and semantic consistency. The code is available at https://github.com/Andyplus1/ConMo.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑