面向真实世界视频编辑的注意力控制下的混合潜在扩散
Blended Latent Diffusion under Attention Control for Real-World Video Editing
AI总结：
本文提出基于图像级混合潜在扩散模型的自适应局部视频编辑方法，通过DDIM反演保持背景、交叉注意力图自主生成掩码，并将U-Net自注意力改造为时序-空间模块以增强时间一致性。
摘要：
由于缺乏完全公开的文本到视频模型，当前的视频编辑方法倾向于构建在预训练的文本到图像生成模型之上，然而它们在处理带有时序信息的视频局部编辑时仍面临巨大挑战。首先，尽管现有方法试图通过预定义掩码聚焦于局部区域编辑，但由于每一帧都是空间上整体生成，掩码外部背景的保持效果并不理想。此外，由用户专门提供掩码是一项额外的昂贵工作，因此需要一种集成到编辑过程中的自主掩码策略。最后但同样重要的是，图像级预训练模型尚未学习视频帧间的时序信息，而这对表达运动和动态至关重要。本文提出将图像级混合潜在扩散模型适配用于执行局部视频编辑任务。具体而言，我们利用DDIM反演获取潜在表示作为背景潜在表示，而非随机加噪的潜在表示，以更好地保持输入视频的背景信息。我们进一步引入了一种源自扩散步骤中交叉注意力图的自主掩码生成机制。最后，我们通过将U-Net的自注意力模块转换为时序-空间模块来增强视频帧间的时间一致性。通过大量实验，我们提出的方法在不同的真实世界视频编辑任务中展示了有效性。
英文摘要：
Due to lack of fully publicly available text-to-video models, current video editing methods tend to build on pre-trained text-to-image generation models, however, they still face grand challenges in dealing with the local editing of video with temporal information. First, although existing methods attempt to focus on local area editing by a pre-defined mask, the preservation of the outside-area background is non-ideal due to the spatially entire generation of each frame. In addition, specially providing a mask by user is an additional costly undertaking, so an autonomous masking strategy integrated into the editing process is desirable. Last but not least, image-level pretrained model hasn't learned temporal information across frames of a video which is vital for expressing the motion and dynamics. In this paper, we propose to adapt a image-level blended latent diffusion model to perform local video editing tasks. Specifically, we leverage DDIM inversion to acquire the latents as background latents instead of the randomly noised ones to better preserve the background information of the input video. We further introduce an autonomous mask manufacture mechanism derived from cross-attention maps in diffusion steps. Finally, we enhance the temporal consistency across video frames by transforming the self-attention blocks of U-Net into temporal-spatial blocks. Through extensive experiments, our proposed approach demonstrates effectiveness in different real-world video editing tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
