VideoComposer：具有运动可控性的组合式视频合成
VideoComposer: Compositional Video Synthesis with Motion Controllability
AI总结：
本文提出组合式视频生成框架 VideoComposer，通过压缩视频运动向量与统一的 STC-encoder 融合文本、空间和时间条件，实现高帧间一致性的多形式运动可控视频合成。
摘要：
将可控性作为视觉内容创作更高标准的追求，已在可定制图像合成方面取得显著进展。然而，由于时间动态变化巨大且需要跨帧时间一致性，实现可控的视频合成仍具挑战性。基于组合式生成范式，本工作提出 VideoComposer，允许用户利用文本条件、空间条件，以及更重要的时间条件，灵活地组合视频。具体而言，考虑到视频数据的特性，我们引入来自压缩视频的运动向量作为显式控制信号，以提供有关时间动态的引导。此外，我们开发了一个时空条件编码器（Spatio-Temporal Condition encoder，STC-encoder），作为统一接口来有效融入序列输入的空间与时间关系；借助该编码器，模型能够更好地利用时间条件，从而实现更高的帧间一致性。大量实验结果表明，VideoComposer 能够以多种形式同时控制合成视频中的空间与时间模式，例如文本描述、草图序列、参考视频，甚至仅仅是手工制作的运动。代码和模型将在 https://videocomposer.github.io 公开发布。
英文摘要：
The pursuit of controllability as a higher standard of visual content creation has yielded remarkable progress in customizable image synthesis. However, achieving controllable video synthesis remains challenging due to the large variation of temporal dynamics and the requirement of cross-frame temporal consistency. Based on the paradigm of compositional generation, this work presents VideoComposer that allows users to flexibly compose a video with textual conditions, spatial conditions, and more importantly temporal conditions. Specifically, considering the characteristic of video data, we introduce the motion vector from compressed videos as an explicit control signal to provide guidance regarding temporal dynamics. In addition, we develop a Spatio-Temporal Condition encoder (STC-encoder) that serves as a unified interface to effectively incorporate the spatial and temporal relations of sequential inputs, with which the model could make better use of temporal conditions and hence achieve higher inter-frame consistency. Extensive experimental results suggest that VideoComposer is able to control the spatial and temporal patterns simultaneously within a synthesized video in various forms, such as text description, sketch sequence, reference video, or even simply hand-crafted motions. The code and models will be publicly available at https://videocomposer.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
