arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.20086cs.LGcs.CRcs.CV
收藏

理解并缓解扩散模型中的复制现象

Understanding and Mitigating Copying in Diffusion Models

University of Maryland, College Park(马里兰大学帕克分校);New York University(纽约大学)

Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, Tom Goldstein

AI总结:

本文分析文本到图像扩散模型复制训练数据的成因,发现文本条件与重复图像同样关键,并提出通过随机化和增强图像描述在训练与推理阶段减少复制的方法。

摘要:

由 Stable Diffusion 等扩散模型生成的图像日益普及。最近的研究甚至诉讼表明,这些模型容易在用户不知情的情况下复制其训练数据。本文首先分析了文本到图像扩散模型中的这种记忆问题。尽管人们普遍认为训练集中的重复图像是推理时内容复制的成因,但我们观察到模型的文本条件起着同样重要的作用。事实上,我们在实验中发现,无条件模型通常不会发生数据复制,而文本条件模型则普遍存在该现象。基于这些发现,我们随后提出了几种在训练和推理阶段通过随机化和增强训练集中的图像描述来减少数据复制的技术。

英文摘要:

Images generated by diffusion models like Stable Diffusion are increasingly widespread. Recent works and even lawsuits have shown that these models are prone to replicating their training data, unbeknownst to the user. In this paper, we first analyze this memorization problem in text-to-image diffusion models. While it is widely believed that duplicated images in the training set are responsible for content replication at inference time, we observe that the text conditioning of the model plays a similarly important role. In fact, we see in our experiments that data replication often does not happen for unconditional models, while it is common in the text-conditional case. Motivated by our findings, we then propose several techniques for reducing data replication at both training and inference time by randomizing and augmenting image captions in the training set.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑