基于视觉引导和掩码增强的自适应去噪用于提示驱动的图像编辑
Vision-guided and Mask-enhanced Adaptive Denoising for Prompt-based Image Editing
AI总结：
提出ViMAEdit方法，通过引入图像嵌入引导、自注意力迭代定位编辑区域和空间自适应方差采样，解决文本提示引导不足、区域关系挖掘有限和统一编辑强度问题，显著提升图像编辑能力。
摘要：
文本到图像扩散模型在从文本提示合成高质量图像方面取得了显著进展，这推动了基于提示的图像编辑研究，即根据目标提示编辑源图像。尽管取得了进展，现有方法仍面临三个关键问题：1）文本提示在引导目标图像生成时的能力有限；2）对词到图像块和图像块到图像块关系的挖掘不足，难以定位编辑区域；3）在每个去噪步骤中对所有区域使用统一的编辑强度。为解决这些问题，我们提出了一种视觉引导和掩码增强的自适应编辑方法，包含三个关键创新设计。首先，我们提出利用图像嵌入作为显式引导来增强传统的基于文本提示的去噪过程，并引入了一种基于CLIP的目标图像嵌入估计策略。其次，我们设计了一种自注意力引导的迭代编辑区域定位策略，迭代地利用自注意力图传递的图像块到图像块关系，来细化交叉注意力图中包含的词到图像块关系。最后，我们提出了一种空间自适应的方差引导采样方法，为关键图像区域突出采样方差以提升编辑能力。实验结果表明，ViMAEdit在编辑能力上优于所有现有方法。
英文摘要：
Text-to-image diffusion models have demonstrated remarkable progress in synthesizing high-quality images from text prompts, which boosts researches on prompt-based image editing that edits a source image according to a target prompt. Despite their advances, existing methods still encounter three key issues: 1) limited capacity of the text prompt in guiding target image generation, 2) insufficient mining of word-to-patch and patch-to-patch relationships for grounding editing areas, and 3) unified editing strength for all regions during each denoising step. To address these issues, we present a Vision-guided and Mask-enhanced Adaptive Editing (ViMAEdit) method with three key novel designs. First, we propose to leverage image embeddings as explicit guidance to enhance the conventional textual prompt-based denoising process, where a CLIP-based target image embedding estimation strategy is introduced. Second, we devise a self-attention-guided iterative editing area grounding strategy, which iteratively exploits patch-to-patch relationships conveyed by self-attention maps to refine those word-to-patch relationships contained in cross-attention maps. Last, we present a spatially adaptive variance-guided sampling, which highlights sampling variances for critical image regions to promote the editing capability. Experimental results demonstrate the superior editing capacity of ViMAEdit over all existing methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
