Edit-A-Video：具有对象感知一致性的单视频编辑
Edit-A-Video: Single Video Editing with Object-Aware Consistency
AI总结：
提出Edit-A-Video框架，利用预训练文本到图像模型和单个文本-视频对实现视频编辑，通过两阶段扩展与反演及稀疏因果混合方法解决背景不一致问题，提升编辑质量。
摘要：
尽管文本到视频（TTV）模型最近取得了显著成功，但针对TTV扩展到视频编辑的方法却很少。受基于扩散的文本到图像（TTI）模型适应TTV方法的启发，我们提出了一种仅给定预训练TTI模型和单个<文本，视频>对的视频编辑框架，称为Edit-A-Video。该框架包括两个阶段：（1）通过附加时间模块并在源视频上进行微调，将2D模型扩展为3D模型；（2）将源视频反演为噪声，并使用目标文本提示和注意力图注入进行编辑。每个阶段都实现了源视频的时间建模和语义属性保持。视频编辑的关键挑战之一是背景不一致问题，即未包含在编辑中的区域遭受不期望且不一致的时间变化。为缓解此问题，我们还引入了一种新颖的掩码混合方法，称为稀疏因果混合（SC混合）。我们改进了先前的掩码混合方法以反映时间一致性，使得应用编辑的区域表现出平滑过渡，同时实现未编辑区域的时空一致性。我们展示了针对各种类型文本和视频的大量实验结果，并在背景一致性、文本对齐和视频编辑质量方面证明了所提方法相对于基线的优越性。
英文摘要：
Despite the fact that text-to-video (TTV) model has recently achieved remarkable success, there have been few approaches on TTV for its extension to video editing. Motivated by approaches on TTV models adapting from diffusion-based text-to-image (TTI) models, we suggest the video editing framework given only a pretrained TTI model and a single <text, video> pair, which we term Edit-A-Video. The framework consists of two stages: (1) inflating the 2D model into the 3D model by appending temporal modules and tuning on the source video (2) inverting the source video into the noise and editing with target text prompt and attention map injection. Each stage enables the temporal modeling and preservation of semantic attributes of the source video. One of the key challenges for video editing include a background inconsistency problem, where the regions not included for the edit suffer from undesirable and inconsistent temporal alterations. To mitigate this issue, we also introduce a novel mask blending method, termed as sparse-causal blending (SC Blending). We improve previous mask blending methods to reflect the temporal consistency so that the area where the editing is applied exhibits smooth transition while also achieving spatio-temporal consistency of the unedited regions. We present extensive experimental results over various types of text and videos, and demonstrate the superiority of the proposed method compared to baselines in terms of background consistency, text alignment, and video editing quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
