Capacity Control is an Effective Memorization Mitigation Mechanism in Text-Conditional Diffusion Models
School of Informatics, University of Edinburgh(爱丁堡大学信息学院);School of Engineering, University of Edinburgh(爱丁堡大学工程学院);Samsung AI Center, Cambridge(三星剑桥人工智能中心)
Raman Dutt, Pedro Sanchez, Ondrej Bohdal, Sotirios A. Tsaftaris, Timothy Hospedales
备注:
Accepted at the GenLaw (Generative AI + Law) workshop at ICML'24
In this work, we present compelling evidence that controlling model capacity during fine-tuning can effectively mitigate memorization in diffusion models. Specifically, we demonstrate that adopting Parameter-Efficient Fine-Tuning (PEFT) within the pre-train fine-tune paradigm significantly reduces memorization compared to traditional full fine-tuning approaches. Our experiments utilize the MIMIC dataset, which comprises image-text pairs of chest X-rays and their corresponding reports. The results, evaluated through a range of memorization and generation quality metrics, indicate that PEFT not only diminishes memorization but also enhances downstream generation quality. Additionally, PEFT methods can be seamlessly combined with existing memorization mitigation techniques for further improvement. The code for our experiments is available at: https://github.com/Raman1121/Diffusion_Memorization_HPO
Demystifying Foreground-Background Memorization in Diffusion Models
Cheriton School of Computer Science, University of Waterloo and Vector Institute(查尔顿计算机科学学院,滑铁卢大学和向量研究所);University of Ottawa(渥太华大学);CISPA Helmholtz Center for Information Security(信息安全赫尔姆霍茨中心CISPA)
Filtering Memorization from Parameter-Space in Diffusion Models
RIKEN AIP(理化学研究所Advanced Institute for Peripheral Research);Science of Tokyo(东京科学大学);University of California, Berkeley(加州大学伯克利分校);Zhejiang University(浙江大学)
Finding DoRI: Discovery of Retained Images in Diffusion Models
CISPA Helmholtz Center for Information Security(CISPA信息安全研究中心);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心(DFKI));Technical University of Darmstadt(达姆施塔特技术大学);Hessian Center for AI (Hessian.AI)(黑森人工智能中心(Hessian.AI));Centre for Cognitive Science, Technical University of Darmstadt(达姆施塔特技术大学认知科学中心)
Investigating Memorization in Video Diffusion Models
School of Computer Science, Faculty of Engineering, The University of Sydney, Australia(悉尼大学计算机科学学院,工程学院);School of Physics, Mathematics and Computing, The University of Western Australia, Australia(西澳大学物理、数学与计算学院);Center for Research in Computer Vision, University of Central Florida, USA(佛罗里达中央大学计算机视觉研究中心)
Redistribute Ensemble Training for Mitigating Memorization in Diffusion Models
School of Computer Science, Wuhan University(武汉大学计算机学院);School of Cyber Science and Technology, Sun Yat-sen University(中山大学网络空间科学与技术学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancy
School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);PKU-OCTA Laboratory for Blockchain and Privacy Computing, Peking University(北京大学PKU-OCTA区块链与隐私计算实验室);Dept. of Comp. Sci. and Tech., Institute for AI, BNRist Center, THBI Lab, Tsinghua University(清华大学计算机科学与技术系、人工智能研究院、BNRist中心、THBI实验室);RealAI;The University of Western Australia(西澳大学);Zhongguancun Laboratory, Beijing, China(中关村实验室);Nanyang Technological University(南洋理工大学)