MoTrans：使用文本驱动视频扩散模型的定制化运动迁移
MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models
AI总结：
针对T2V模型难以建模复杂人体运动且运动与外观耦合的问题，本文提出MoTrans，通过MLLM重描述、外观注入和运动特定嵌入实现定制化运动迁移。
摘要：
现有的预训练文本到视频（T2V）模型在生成包含基础运动或相机运动的真实视频方面已展现出令人瞩目的能力。然而，这些模型在生成复杂的、以人为中心的运动时存在显著局限。当前的研究主要集中在包含特定运动的少量视频上对模型进行微调。它们往往无法在有限的参考视频中有效解耦运动与外观，从而削弱了运动模式的建模能力。为此，我们提出 MoTrans，这是一种定制化运动迁移方法，能够在新情境中生成具有相似运动的视频。具体而言，我们引入了一种基于多模态大语言模型（MLLM）的重描述器，用于扩展初始提示，使其更加关注外观；同时引入外观注入模块，将视频帧中的外观先验适配到运动建模过程中。来自重描述提示和视频帧的这些互补多模态表示促进了外观建模，并有助于外观与运动的解耦。此外，我们设计了运动特定嵌入，以进一步增强对特定运动的建模。实验结果表明，我们的方法能够从单个或多个参考视频中有效学习特定运动模式，并在定制化视频生成方面取得优于现有方法的表现。
英文摘要：
Existing pretrained text-to-video (T2V) models have demonstrated impressive abilities in generating realistic videos with basic motion or camera movement. However, these models exhibit significant limitations when generating intricate, human-centric motions. Current efforts primarily focus on fine-tuning models on a small set of videos containing a specific motion. They often fail to effectively decouple motion and the appearance in the limited reference videos, thereby weakening the modeling capability of motion patterns. To this end, we propose MoTrans, a customized motion transfer method enabling video generation of similar motion in new context. Specifically, we introduce a multimodal large language model (MLLM)-based recaptioner to expand the initial prompt to focus more on appearance and an appearance injection module to adapt appearance prior from video frames to the motion modeling process. These complementary multimodal representations from recaptioned prompt and video frames promote the modeling of appearance and facilitate the decoupling of appearance and motion. In addition, we devise a motion-specific embedding for further enhancing the modeling of the specific motion. Experimental results demonstrate that our method effectively learns specific motion pattern from singular or multiple reference videos, performing favorably against existing methods in customized video generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
