arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.07345cs.CV
收藏

从扩散模型中擦除概念

Erasing Concepts from Diffusion Models

Northeastern University(东北大学);Massachusetts Institute of Technology(麻省理工学院)

Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, David Bau

AI总结:

针对扩散模型可能被滥用于生成不当内容的问题,提出一种基于负向引导的微调方法,仅凭风格名称即可永久擦除模型中的视觉概念,性能与Safe Latent Diffusion相当,且无法被绕过。

摘要:

受近期文本到图像扩散模型进展的推动,我们研究了从模型权重中擦除特定概念的方法。尽管Stable Diffusion在生成露骨或写实艺术作品方面展现出潜力,但也引发了对其可能被滥用的担忧。我们提出一种微调方法,仅需给定风格名称,并利用负向引导作为教师,即可从预训练扩散模型中擦除一个视觉概念。我们将我们的方法与先前移除露骨性内容的方法进行基准测试,并展示其有效性,性能与Safe Latent Diffusion和审查训练相当。为评估艺术风格移除效果,我们进行了从网络中擦除五位现代艺术家的实验,并开展用户研究以评估人类对移除风格的感知。与先前方法不同,我们的方法能够永久性地从扩散模型中移除概念,而非在推理时修改输出,因此即使用户能够访问模型权重,也无法绕过该移除。我们的代码、数据和结果可在https://erasing.baulab.info/获取。

英文摘要:

Motivated by recent advancements in text-to-image diffusion, we study erasure of specific concepts from the model's weights. While Stable Diffusion has shown promise in producing explicit or realistic artwork, it has raised concerns regarding its potential for misuse. We propose a fine-tuning method that can erase a visual concept from a pre-trained diffusion model, given only the name of the style and using negative guidance as a teacher. We benchmark our method against previous approaches that remove sexually explicit content and demonstrate its effectiveness, performing on par with Safe Latent Diffusion and censored training. To evaluate artistic style removal, we conduct experiments erasing five modern artists from the network and conduct a user study to assess the human perception of the removed styles. Unlike previous methods, our approach can remove concepts from a diffusion model permanently rather than modifying the output at the inference time, so it cannot be circumvented even if a user has access to model weights. Our code, data, and results are available at https://erasing.baulab.info/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑