arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.00922cs.CV
收藏

迈向无记忆化扩散模型

Towards Memorization-Free Diffusion Models

School of Computer Science, Faculty of Engineering, The University of Sydney(悉尼大学工程学部计算机科学学院)

Chen Chen, Daochang Liu, Chang Xu

AI总结:

本文提出抗记忆化引导(AMG)框架,通过三种针对性引导策略和自动检测系统,在保持图像质量与文本对齐的同时,首次成功消除扩散模型的所有记忆化现象。

摘要:

预训练扩散模型及其输出因其合成高质量图像的卓越能力和开源特性而被广泛获取。然而,用户可能面临诉讼风险,因为模型在推理过程中倾向于记忆并复现训练数据。为了解决这一问题,我们提出了抗记忆化引导(AMG),这是一个新颖的框架,针对记忆化的主要成因采用了三种有针对性的引导策略:图像与文本描述的重复,以及高度具体的用户提示。因此,AMG 在保持高图像质量和文本对齐的同时,确保了无记忆化的输出,这得益于其引导方法的协同作用,每一种方法本身都是不可或缺的。AMG 还具备一个创新的自动检测系统,可在推理过程的每一步检测潜在的记忆化,允许选择性地应用引导策略,最小程度地干扰原始采样过程以保持输出实用性。我们将 AMG 应用于预训练的去噪扩散概率模型(DDPM)和 Stable Diffusion,并在各种生成任务中进行了验证。结果表明,AMG 是第一种成功消除所有记忆化实例的方法,且对图像质量和文本对齐没有影响或仅有边际影响,这一点通过 FID 和 CLIP 分数得到了证实。

英文摘要:

Pretrained diffusion models and their outputs are widely accessible due to their exceptional capacity for synthesizing high-quality images and their open-source nature. The users, however, may face litigation risks owing to the models' tendency to memorize and regurgitate training data during inference. To address this, we introduce Anti-Memorization Guidance (AMG), a novel framework employing three targeted guidance strategies for the main causes of memorization: image and caption duplication, and highly specific user prompts. Consequently, AMG ensures memorization-free outputs while maintaining high image quality and text alignment, leveraging the synergy of its guidance methods, each indispensable in its own right. AMG also features an innovative automatic detection system for potential memorization during each step of inference process, allows selective application of guidance strategies, minimally interfering with the original sampling process to preserve output utility. We applied AMG to pretrained Denoising Diffusion Probabilistic Models (DDPM) and Stable Diffusion across various generation tasks. The results demonstrate that AMG is the first approach to successfully eradicates all instances of memorization with no or marginal impacts on image quality and text-alignment, as evidenced by FID and CLIP scores.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑