arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.09256cs.CV
收藏

LIME:基于扩散模型中注意力正则化的局部图像编辑

LIME: Localized Image Editing via Attention Regularization in Diffusion Models

ETH Zürich(苏黎世联邦理工学院);Google Switzerland(谷歌瑞士分部);Technical University of Munich(慕尼黑工业大学)

Enis Simsar, Alessio Tonioni, Yongqin Xian, Thomas Hofmann, Federico Tombari

AI总结:

本文提出 LIME,通过预训练特征、聚类、交叉注意力图细化及新型注意力正则化,在无需用户指定区域、额外文本或训练微调的情况下实现精确局部图像编辑。

摘要:

随着文本到图像生成的最新进展,扩散模型(DMs)因其生成高质量多样化图像的能力而受到广泛关注。研究重点目前正转向 DMs 的可控性。该领域的一个重大挑战是局部编辑,即在修改图像特定区域的同时不影响其余内容。本文提出了用于扩散模型局部图像编辑的 LIME。LIME 不需要用户指定的感兴趣区域(RoI)或额外文本输入,而是利用来自预训练方法的特征和一种简单的聚类方法来获得精确的编辑掩码。随后,它借助交叉注意力图细化这些分割区域,以寻找可获得局部编辑的区域。最后,我们提出了一种新颖的交叉注意力正则化技术,该技术在去噪步骤中惩罚 RoI 内不相关的交叉注意力分数,从而确保局部化编辑。我们的方法无需重新训练、微调或额外用户输入,在各种编辑基准测试中持续提升了现有方法的性能。项目页面可参见 https://enisimsar.github.io/LIME/。

英文摘要:

Diffusion models (DMs) have gained prominence due to their ability to generate high-quality varied images with recent advancements in text-to-image generation. The research focus is now shifting towards the controllability of DMs. A significant challenge within this domain is localized editing, where specific areas of an image are modified without affecting the rest of the content. This paper introduces LIME for localized image editing in diffusion models. LIME does not require user-specified regions of interest (RoI) or additional text input, but rather employs features from pre-trained methods and a straightforward clustering method to obtain precise editing mask. Then, by leveraging cross-attention maps, it refines these segments for finding regions to obtain localized edits. Finally, we propose a novel cross-attention regularization technique that penalizes unrelated cross-attention scores in the RoI during the denoising steps, ensuring localized edits. Our approach, without re-training, fine-tuning and additional user inputs, consistently improves the performance of existing methods in various editing benchmarks. The project page can be found at https://enisimsar.github.io/LIME/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑