arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.03162cs.CV
收藏

Direct-a-Video:具有用户指导的摄像机运动和物体运动的定制视频生成

Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion

City University of Hong Kong(香港城市大学);Tianjin University(天津大学);Kuaishou Technology(快手科技)

Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, Jing Liao

AI总结:

本文提出 Direct-a-Video 系统,通过空间交叉注意力调制和新引入的时间交叉注意力层,分别解耦控制物体运动与摄像机平移缩放,并以自监督方式训练,无需运动标注,实现灵活可泛化的定制视频生成。

摘要:

近年来,文本到视频扩散模型取得了令人瞩目的进展。在实际应用中,用户通常希望能够独立控制物体运动和摄像机运动,以实现定制化的视频创作。然而,现有方法缺乏以解耦方式分别控制物体运动和摄像机运动的能力,这限制了文本到视频模型的可控性和灵活性。在本文中,我们介绍了 Direct-a-Video,这是一个允许用户独立指定多个物体的运动以及摄像机的平移和缩放运动的系统,如同导演一部视频。我们提出了一种简单而有效的策略,用于解耦控制物体运动和摄像机运动。物体运动通过利用模型固有先验的空间交叉注意力调制来控制,无需额外优化。对于摄像机运动,我们引入了新的时间交叉注意力层来解释定量的摄像机运动参数。我们进一步采用基于增强的方法,在小规模数据集上以自监督方式训练这些层,从而无需显式的运动标注。这两个组件独立运行,允许单独或组合控制,并且可以泛化到开放域场景。大量实验证明了我们方法的优越性和有效性。项目主页和代码可在 https://direct-a-video.github.io/ 获取。

英文摘要:

Recent text-to-video diffusion models have achieved impressive progress. In practice, users often desire the ability to control object motion and camera movement independently for customized video creation. However, current methods lack the focus on separately controlling object motion and camera movement in a decoupled manner, which limits the controllability and flexibility of text-to-video models. In this paper, we introduce Direct-a-Video, a system that allows users to independently specify motions for multiple objects as well as camera's pan and zoom movements, as if directing a video. We propose a simple yet effective strategy for the decoupled control of object motion and camera movement. Object motion is controlled through spatial cross-attention modulation using the model's inherent priors, requiring no additional optimization. For camera movement, we introduce new temporal cross-attention layers to interpret quantitative camera movement parameters. We further employ an augmentation-based approach to train these layers in a self-supervised manner on a small-scale dataset, eliminating the need for explicit motion annotation. Both components operate independently, allowing individual or combined control, and can generalize to open-domain scenarios. Extensive experiments demonstrate the superiority and effectiveness of our method. Project page and code are available at https://direct-a-video.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑