arXiv 2309.156642023-09-27cs.CV

动态提示学习：解决基于文本的图像编辑中的交叉注意力泄漏问题

Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing

Computer Vision Center（计算机视觉中心）；Universitat Autònoma de Barcelona（巴塞罗那自治大学）

Kai Wang, Fei Yang, Shiqi Yang, Muhammad Atif Butt, Joost van de Weijer

备注：: Neurips 2023. The code page: https://github.com/wangkai930418/DPL

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对文本图像编辑中交叉注意力图失准导致背景或无关对象被误改的问题，本文提出基于泄漏修复损失更新名词动态标记的DPL方法，并在Stable Diffusion上验证其定量与定性优势。

摘要：

大规模文本到图像生成模型一直是生成式AI中的一项突破性进展，其中扩散模型展现出其根据输入文本提示合成令人信服图像的惊人能力。图像编辑研究的目标是通过修改文本提示，让用户能够控制生成的图像。当前的图像编辑技术容易对目标区域之外的区域造成非预期修改，例如背景或干扰对象，这些对象与目标对象存在某种语义或视觉关系。根据我们的实验发现，不准确的交叉注意力图是这一问题的根源。基于这一观察，我们提出动态提示学习（Dynamic Prompt Learning，DPL），以促使交叉注意力图聚焦于文本提示中正确的名词。通过利用所提出的泄漏修复损失更新文本输入中名词的动态标记，我们实现了针对特定对象的细粒度图像编辑，同时避免对其他图像区域产生不期望的改变。我们的方法DPL基于公开可用的Stable Diffusion，在广泛的图像上进行了大量评估，并在定量指标（CLIP score、Structure-Dist）和定性结果（用户评估）上持续取得更优表现。我们展示了其在Word-Swap、Prompt Refinement和Attention Re-weighting方面改进的提示编辑结果，尤其适用于复杂的多对象场景。

英文摘要：

Large-scale text-to-image generative models have been a ground-breaking development in generative AI, with diffusion models showing their astounding ability to synthesize convincing images following an input text prompt. The goal of image editing research is to give users control over the generated images by modifying the text prompt. Current image editing techniques are susceptible to unintended modifications of regions outside the targeted area, such as on the background or on distractor objects which have some semantic or visual relationship with the targeted object. According to our experimental findings, inaccurate cross-attention maps are at the root of this problem. Based on this observation, we propose Dynamic Prompt Learning (DPL) to force cross-attention maps to focus on correct noun words in the text prompt. By updating the dynamic tokens for nouns in the textual input with the proposed leakage repairment losses, we achieve fine-grained image editing over particular objects while preventing undesired changes to other image regions. Our method DPL, based on the publicly available Stable Diffusion, is extensively evaluated on a wide range of images, and consistently obtains superior results both quantitatively (CLIP score, Structure-Dist) and qualitatively (on user-evaluation). We show improved prompt editing results for Word-Swap, Prompt Refinement, and Attention Re-weighting, especially for complex multi-object scenes.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文