你不需要所有注意力:文本到图像扩散模型中的外科记忆缓解
You Don't Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models
AI总结:
提出GUARD框架,通过吸引-排斥动力学调整去噪过程,结合交叉注意力衰减机制,在不损害图像质量的前提下有效缓解文本到图像扩散模型中的记忆问题。
摘要:
生成模型已被证明会“记忆”某些训练数据,导致生成逐字或近乎逐字的图像,这可能引发隐私问题或版权侵权。我们引入了使用吸引-排斥动力学的引导(GUARD),一种用于文本到图像扩散模型中记忆缓解的新框架。GUARD调整图像去噪过程,引导生成远离原始训练图像,朝向与训练数据不同但仍与提示对齐的图像,防止复制训练数据,同时不损害图像生成质量。我们提出了该框架的一个具体实例,其中我们引导的正向目标由一种新的(交叉)注意力衰减方法给出,该方法基于(i)一种新颖的统计机制,自动识别需要衰减交叉注意力的提示位置,以及(ii)在这些每个提示的位置衰减交叉注意力。由此产生的GUARD提供了一种外科手术式的、动态的、每个提示的推理时方法,我们发现,在两种架构以及逐字和模板记忆方面,它是最稳健的方法,始终产生最先进的记忆缓解结果,同时在图像质量方面也优于或产生可比的结果。
英文摘要:
Generative models have been shown to "memorize" certain training data, leading to verbatim or near-verbatim generating images, which may cause privacy concerns or copyright infringement. We introduce Guidance Using Attractive-Repulsive Dynamics (GUARD), a novel framework for memorization mitigation in text-to-image diffusion models. GUARD adjusts the image denoising process to guide the generation away from an original training image and towards one that is distinct from training data while remaining aligned with the prompt, guarding against reproducing training data, without hurting image generation quality. We propose a concrete instantiation of this framework, where the positive target that we steer towards is given by a novel method for (cross) attention attenuation based on (i) a novel statistical mechanism that automatically identifies the prompt positions where cross attention must be attenuated and (ii) attenuating cross-attention in these per-prompt locations. The resulting GUARD offers a surgical, dynamic per-prompt inference-time approach that, we find, is by far the most robust method in terms of consistently producing state-of-the-art results for memorization mitigation across two architectures and for both verbatim and template memorization, while also improving upon or yielding comparable results in terms of image quality.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。