MotionCtrl:用于视频生成的统一灵活运动控制器
MotionCtrl: A Unified and Flexible Motion Controller for Video Generation
AI总结:
MotionCtrl是一种统一灵活的视频生成运动控制器,独立控制摄像机与物体运动,通过姿态与轨迹条件实现细粒度控制,实验证明优于现有方法。
摘要:
视频中的运动主要包括由摄像机移动引起的摄像机运动,以及由物体移动引起的物体运动。对摄像机运动和物体运动的精确控制对于视频生成至关重要。然而,现有工作要么主要关注一种类型的运动,要么没有明确区分这两种运动,限制了它们的控制能力和多样性。因此,本文提出了MotionCtrl,一种用于视频生成的统一灵活的运动控制器,旨在有效且独立地控制摄像机和物体的运动。MotionCtrl的架构和训练策略经过精心设计,考虑了摄像机运动、物体运动和不完美训练数据的固有属性。与以往方法相比,MotionCtrl具有三个主要优势:1)它有效且独立地控制摄像机运动和物体运动,实现更细粒度的运动控制,并促进两种运动类型的灵活多样组合。2)其运动条件由摄像机姿态和轨迹决定,这些条件无外观信息,对生成视频中物体的外观或形状影响最小。3)它是一个相对通用的模型,一旦训练,可以适应广泛的摄像机姿态和轨迹。已进行了广泛的定性和定量实验,以证明MotionCtrl相对于现有方法的优越性。项目页面:https://wzhouxiff.github.io/projects/MotionCtrl/
英文摘要:
Motions in a video primarily consist of camera motion, induced by camera movement, and object motion, resulting from object movement. Accurate control of both camera and object motion is essential for video generation. However, existing works either mainly focus on one type of motion or do not clearly distinguish between the two, limiting their control capabilities and diversity. Therefore, this paper presents MotionCtrl, a unified and flexible motion controller for video generation designed to effectively and independently control camera and object motion. The architecture and training strategy of MotionCtrl are carefully devised, taking into account the inherent properties of camera motion, object motion, and imperfect training data. Compared to previous methods, MotionCtrl offers three main advantages: 1) It effectively and independently controls camera motion and object motion, enabling more fine-grained motion control and facilitating flexible and diverse combinations of both types of motion. 2) Its motion conditions are determined by camera poses and trajectories, which are appearance-free and minimally impact the appearance or shape of objects in generated videos. 3) It is a relatively generalizable model that can adapt to a wide array of camera poses and trajectories once trained. Extensive qualitative and quantitative experiments have been conducted to demonstrate the superiority of MotionCtrl over existing methods. Project Page: https://wzhouxiff.github.io/projects/MotionCtrl/
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。