arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.03431cs.CV
收藏

理解Stable Diffusion中用于文本引导图像编辑的交叉注意力与自注意力

Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing

South China University of Technology（华南理工大学）；Alibaba Group（阿里巴巴集团）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））

Bingyan Liu, Chengyu Wang, Tingfeng Cao, Kui Jia, Jun Huang

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

该研究深入分析了Stable Diffusion中交叉注意力与自注意力在文本引导图像编辑中的作用，发现交叉注意力易致编辑失败、自注意力可保留图像几何细节，并据此提出更稳定高效的免调优编辑方法，性能在多数据集上超越主流方案。

摘要：

深度文本到图像合成（Text-to-Image Synthesis, TIS）模型（如Stable Diffusion）近年来在创意文本生成图像领域广受欢迎。但对于特定领域场景，免调优的文本引导图像编辑（Text-guided Image Editing, TIE）对应用开发者而言更为重要，该技术通过在生成过程中操控注意力层的特征分量，修改图像中的对象或对象属性。然而，目前人们对这些注意力层学习到了哪些语义含义、以及注意力图的哪些部分促成了图像编辑的成功，仍知之甚少。\n本文开展了深入的探测分析，证明Stable Diffusion中的交叉注意力图通常包含会导致编辑失败的对象属性信息。相比之下，自注意力图在源图像向目标图像转换的过程中，对保留源图像的几何与形状细节起着关键作用。我们的分析为理解扩散模型中的交叉注意力图与自注意力图提供了宝贵见解。此外，基于研究发现，我们简化了主流图像编辑方法，提出了一种更直接但更稳定高效的免调优流程，该流程仅在去噪过程中修改指定注意力层的自注意力图。实验结果表明，我们的简化方法在多个数据集上的性能始终优于主流方法。

英文摘要：

Deep Text-to-Image Synthesis (TIS) models such as Stable Diffusion have recently gained significant popularity for creative Text-to-image generation. Yet, for domain-specific scenarios, tuning-free Text-guided Image Editing (TIE) is of greater importance for application developers, which modify objects or object properties in images by manipulating feature components in attention layers during the generation process. However, little is known about what semantic meanings these attention layers have learned and which parts of the attention maps contribute to the success of image editing. In this paper, we conduct an in-depth probing analysis and demonstrate that cross-attention maps in Stable Diffusion often contain object attribution information that can result in editing failures. In contrast, self-attention maps play a crucial role in preserving the geometric and shape details of the source image during the transformation to the target image. Our analysis offers valuable insights into understanding cross and self-attention maps in diffusion models. Moreover, based on our findings, we simplify popular image editing methods and propose a more straightforward yet more stable and efficient tuning-free procedure that only modifies self-attention maps of the specified attention layers during the denoising process. Experimental results show that our simplified method consistently surpasses the performance of popular approaches on multiple datasets.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑