InFusion：面向多概念零样本文本驱动视频编辑的注入与注意力融合
InFusion: Inject and Attention Fusion for Multi Concept Zero-Shot Text-based Video Editing
AI总结：
本文提出InFusion框架，利用预训练图像扩散模型实现零样本文本驱动视频编辑，通过U-Net特征差异注入、注意力融合与掩码提取支持多概念像素级控制，无需训练即可生成时间一致的编辑视频。
摘要：
大型文本到图像扩散模型在生成多样且高质量的图像方面取得了显著成功。此外，这些模型已被成功用于仅通过改变文本提示来编辑输入图像。但当这些模型应用于视频时，主要挑战是确保帧间的时间一致性和连贯性。在本文中，我们提出InFusion，一个利用大型预训练图像扩散模型的零样本文本驱动视频编辑框架。我们的框架特别支持对编辑提示中提到的多个概念进行像素级控制编辑。具体而言，我们从U-Net解码器层的残差块中注入源提示和编辑提示所获得特征的差异。当这些特征与注入的注意力特征相结合时，就可以查询源内容，并在注入未编辑部分的同时缩放已编辑概念。通过掩码提取和注意力融合，编辑进一步以细粒度方式控制，将编辑部分从源中剪切出来并粘贴到编辑提示的去噪流程中。由于不需要训练，我们的框架是单次微调模型编辑的低成本替代方案。我们使用LoRA在通用图像模型（Stable Diffusion v1.5）上演示了复杂概念编辑。该适配方式与所有现有图像扩散技术兼容。大量实验结果表明，现有方法在渲染高质量且时间一致的视频方面是有效的。
英文摘要：
Large text-to-image diffusion models have achieved remarkable success in generating diverse, high-quality images. Additionally, these models have been successfully leveraged to edit input images by just changing the text prompt. But when these models are applied to videos, the main challenge is to ensure temporal consistency and coherence across frames. In this paper, we propose InFusion, a framework for zero-shot text-based video editing leveraging large pre-trained image diffusion models. Our framework specifically supports editing of multiple concepts with pixel-level control over diverse concepts mentioned in the editing prompt. Specifically, we inject the difference in features obtained with source and edit prompts from U-Net residual blocks of decoder layers. When these are combined with injected attention features, it becomes feasible to query the source contents and scale edited concepts along with the injection of unedited parts. The editing is further controlled in a fine-grained manner with mask extraction and attention fusion, which cut the edited part from the source and paste it into the denoising pipeline for the editing prompt. Our framework is a low-cost alternative to one-shot tuned models for editing since it does not require training. We demonstrated complex concept editing with a generalised image model (Stable Diffusion v1.5) using LoRA. Adaptation is compatible with all the existing image diffusion techniques. Extensive experimental results demonstrate the effectiveness of existing methods in rendering high-quality and temporally consistent videos.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
