ControlVideo：面向一次性文本驱动视频编辑及其扩展的条件控制
ControlVideo: Conditional Control for One-shot Text-driven Video Editing and Beyond
AI总结：
ControlVideo 通过在预训练文本到图像扩散模型中引入条件控制与注意力微调，实现一次性文本驱动视频编辑，并借助分段融合策略扩展至长视频编辑，显著提升保真度与时间一致性。
摘要：
本文提出了 ControlVideo，用于文本驱动视频编辑——生成与给定文本对齐同时保留源视频结构的视频。ControlVideo 建立在预训练的文本到图像扩散模型之上，通过引入额外条件（如边缘图）来增强保真度和时间一致性，并通过对设计空间的深入探索，在源视频-文本对上微调关键帧注意力和时间注意力。大量实验结果表明，ControlVideo 优于多种有竞争力的基线方法，能够生成在源内容方面具有高保真度且具备时间一致性的视频，同时与文本保持一致。通过在训练前将 Low-rank adaptation 层融入模型，ControlVideo 进一步具备生成与参考图像无缝对齐的视频的能力。更重要的是，ControlVideo 可以很容易地扩展到更具挑战性的长视频编辑任务（例如包含数百帧的视频），在该任务中保持长程时间一致性至关重要。为实现这一目标，我们提出构建融合 ControlVideo，方法是将基础 ControlVideo 应用于重叠的短视频片段和关键帧视频，然后通过预定义的权重函数将其合并。实证结果验证了其能够生成跨越 140 帧的视频，这大约是先前工作所达到帧数的 5.83 到 17.5 倍。代码可在 https://github.com/thu-ml/controlvideo 获取，可视化结果可在 https://drive.google.com/file/d/1wEgc2io3UwmoC5vTPbkccFvTkwVqsZlK/view?usp=drive_link 查看。
英文摘要：
This paper presents \emph{ControlVideo} for text-driven video editing -- generating a video that aligns with a given text while preserving the structure of the source video. Building on a pre-trained text-to-image diffusion model, ControlVideo enhances the fidelity and temporal consistency by incorporating additional conditions (such as edge maps), and fine-tuning the key-frame and temporal attention on the source video-text pair via an in-depth exploration of the design space. Extensive experimental results demonstrate that ControlVideo outperforms various competitive baselines by delivering videos that exhibit high fidelity w.r.t. the source content, and temporal consistency, all while aligning with the text. By incorporating Low-rank adaptation layers into the model before training, ControlVideo is further empowered to generate videos that align seamlessly with reference images. More importantly, ControlVideo can be readily extended to the more challenging task of long video editing (e.g., with hundreds of frames), where maintaining long-range temporal consistency is crucial. To achieve this, we propose to construct a fused ControlVideo by applying basic ControlVideo to overlapping short video segments and key frame videos and then merging them by pre-defined weight functions. Empirical results validate its capability to create videos across 140 frames, which is approximately 5.83 to 17.5 times more than what previous works achieved. The code is available at \href{https://github.com/thu-ml/controlvideo}{https://github.com/thu-ml/controlvideo} and the visualization results are available at \href{https://drive.google.com/file/d/1wEgc2io3UwmoC5vTPbkccFvTkwVqsZlK/view?usp=drive_link}{HERE}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
