VideoDirector：借助文本生成视频模型实现精准视频编辑
VideoDirector: Precise Video Editing via Text-to-Video Models
AI总结：
针对传统编辑范式迁移至T2V模型时的伪影问题，提出时空解耦引导、多帧空文本优化及自注意力控制策略，构建VideoDirector方法实现高精度、高保真且时序连贯的视频编辑。
摘要：
尽管采用文本生成图像（T2I）模型的典型“反演后编辑”范式已展现出可观效果，但将其直接拓展至文本生成视频（T2V）模型时，仍会出现色彩闪烁、内容失真等严重伪影。因此，当前视频编辑方法主要依赖T2I模型，而这类模型本身缺乏时间相干生成能力，往往导致编辑效果不佳。本文将典型编辑范式的失效归因于两点：1）时空耦合紧密。普通的基于关键向量（pivotal）的反演策略难以在视频扩散模型中解耦时空信息；2）时空布局复杂。普通的交叉注意力控制在保留未编辑内容方面存在不足。为解决这些局限，我们提出时空解耦引导（STDG）与多帧空文本优化策略，为更精准的关键向量反演提供关键时间线索。此外，我们引入自注意力控制策略，以在精准局部内容编辑中维持更高保真度。实验结果表明，我们提出的方法（命名为VideoDirector）有效利用了T2V模型强大的时间生成能力，所生成的编辑后视频在准确性、运动流畅度、真实感及未编辑内容保真度方面均达到当前最优水平。
英文摘要：
Despite the typical inversion-then-editing paradigm using text-to-image (T2I) models has demonstrated promising results, directly extending it to text-to-video (T2V) models still suffers severe artifacts such as color flickering and content distortion. Consequently, current video editing methods primarily rely on T2I models, which inherently lack temporal-coherence generative ability, often resulting in inferior editing results. In this paper, we attribute the failure of the typical editing paradigm to: 1) Tightly Spatial-temporal Coupling. The vanilla pivotal-based inversion strategy struggles to disentangle spatial-temporal information in the video diffusion model; 2) Complicated Spatial-temporal Layout. The vanilla cross-attention control is deficient in preserving the unedited content. To address these limitations, we propose a spatial-temporal decoupled guidance (STDG) and multi-frame null-text optimization strategy to provide pivotal temporal cues for more precise pivotal inversion. Furthermore, we introduce a self-attention control strategy to maintain higher fidelity for precise partial content editing. Experimental results demonstrate that our method (termed VideoDirector) effectively harnesses the powerful temporal generation capabilities of T2V models, producing edited videos with state-of-the-art performance in accuracy, motion smoothness, realism, and fidelity to unedited content.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
