Diffusion models, known for their tremendous ability to generate novel and high-quality samples, have recently raised concerns due to their data memorization behavior, which poses privacy risks. Recent approaches for memory mitigation either only focused on the text modality problem in cross-modal generation tasks or utilized data augmentation strategies. In this paper, we propose a novel training framework for diffusion models from the perspective of visual modality, which is more generic and fundamental for mitigating memorization. To facilitate forgetting of stored information in diffusion model parameters, we propose an iterative ensemble training strategy by splitting the data into multiple shards for training multiple models and intermittently aggregating these model parameters. Moreover, practical analysis of losses illustrates that the training loss for easily memorable images tends to be obviously lower. Thus, we propose an anti-gradient control method to exclude the sample with a lower loss value from the current mini-batch to avoid memorizing. Extensive experiments and analysis on four datasets are conducted to illustrate the effectiveness of our method, and results show that our method successfully reduces memory capacity while even improving the performance slightly. Moreover, to save the computing cost, we successfully apply our method to fine-tune the well-trained diffusion models by limited epochs, demonstrating the applicability of our method. Code is available in https://github.com/liuxiao-guan/IET_AGC.
Redistribute Ensemble Training for Mitigating Memorization in Diffusion Models
School of Computer Science, Wuhan University(武汉大学计算机学院);School of Cyber Science and Technology, Sun Yat-sen University(中山大学网络空间科学与技术学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
Filtering Memorization from Parameter-Space in Diffusion Models
RIKEN AIP(理化学研究所Advanced Institute for Peripheral Research);Science of Tokyo(东京科学大学);University of California, Berkeley(加州大学伯克利分校);Zhejiang University(浙江大学)
Unconsciously Forget: Mitigating Memorization; Without Knowing What is being Memorized
RWTH Aachen University(亚琛工业大学);National University of Singapore(新加坡国立大学);Fraunhofer Institute for Applied Information Technology FIT(弗劳恩霍夫应用信息技术研究所FIT)
CrossSplit: Mitigating Label Noise Memorization through Data Splitting
Samsung Advanced Institute of Technology (SAIT)(三星高级技术研究院(SAIT));SAIT AI Lab(三星高级技术研究院人工智能实验室);Mila(米拉研究所);Université de Montreal(蒙特利尔大学);Canada CIFAR AI Chair(加拿大CIFAR人工智能讲席)
Towards Application Aligned Synthetic Surgical Image Synthesis
Department of Translational Surgical Oncology, NCT/UCC Dresden(转化外科肿瘤学系);DKFZ, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden,HZDR, Germany(德累斯顿技术大学医学院);The Centre for Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden(人机交互 tactile 网络中心)
VGMShield: Mitigating Misuse of Video Generative Models
University of Virginia(弗吉尼亚大学);CISPA Helmholtz Center for Information Security(CISPA亥姆霍兹信息安全中心);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training
Johns Hopkins University(约翰斯·霍普金斯大学);University of California, Berkeley(加州大学伯克利分校);University of California San Diego(加州大学圣地亚哥分校);Duke University(杜克大学)