Images generated by diffusion models like Stable Diffusion are increasingly widespread. Recent works and even lawsuits have shown that these models are prone to replicating their training data, unbeknownst to the user. In this paper, we first analyze this memorization problem in text-to-image diffusion models. While it is widely believed that duplicated images in the training set are responsible for content replication at inference time, we observe that the text conditioning of the model plays a similarly important role. In fact, we see in our experiments that data replication often does not happen for unconditional models, while it is common in the text-conditional case. Motivated by our findings, we then propose several techniques for reducing data replication at both training and inference time by randomizing and augmenting image captions in the training set.
TWIG: Two-Step Image Generation using Segmentation Masks in Diffusion Models
BRAC University(布拉克大学);University of Delaware(德雷塞尔大学);University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);ADAPT SFI Research Center(ADAPT SFI研究中心);Insight SFI Research Center for Data Analytics(Insight SFI数据分析研究中心);Dublin City University(都柏林城市大学)
Mitigating Exposure Bias in Discriminator Guided Diffusion Models
Artificial Intelligence and Learning Systems Laboratory(人工智能与学习系统实验室);School of Electrical and Computer Engineering(电气与计算机工程学院);National Technical University of Athens(雅典国立技术大学)
ProstateGAN: Mitigating Data Bias via Prostate Diffusion Imaging Synthesis with Generative Adversarial Networks
University of Waterloo(滑铁卢大学);Waterloo Artificial Intelligence Institute(滑铁卢人工智能研究所);Lunenfeld-Tanenbaum Research Institute (LTRI)(卢嫩菲尔德-塔南鲍姆研究所(LTRI));Sinai Health System(西奈卫生系统);University of Toronto(多伦多大学)
Improving the Euclidean Diffusion Generation of Manifold Data by Mitigating Score Function Singularity
Center for Data Science, Peking University(北京大学数据科学中心);Zuse Institute Berlin(柏林泽尼茨研究所);Center for Machine Learning Research, Peking University(北京大学机器学习研究中心);LMAM and School of Mathematical Sciences, Peking University(北京大学数学科学学院)
MeDi: Metadata-Guided Diffusion Models for Mitigating Biases in Tumor Classification
Machine Learning Group, Technische Universität Berlin, Germany The Konrad Zuse School of Excellence in Learning;Intelligent Systems (ELIZA), Germany BIFOLD – Berlin Institute for the Foundations of Learning
Mitigating Illumination-Induced Domain Shift in Night-Time Pedestrian Detection for Intelligent Vehicles using Annotation-Preserving Diffusion Augmentation
Understanding and Mitigating Memorization in Generative Models via Sharpness of Probability Landscapes
Department of Computer Science, Yonsei University, Seoul, Korea(计算机科学系,延世大学,首尔,韩国);Department of Artificial Intelligence, Yonsei University, Seoul, Korea(人工智能系,延世大学,首尔,韩国)
Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
Department of Computer;Data Sciences, Case Western Reserve University, Cleveland, USA;Department of Computer Science \& Engineering, Texas A\&M University, College Station, USA;Department of Biochemistry, Case Western Reserve University, Cleveland, USA;Center for RNA Science;Therapeutics, Case Western Reserve University, Cleveland, USA;Department of Biomedical Engineering, Case Western Reserve University, Cleveland, USA
Redistribute Ensemble Training for Mitigating Memorization in Diffusion Models
School of Computer Science, Wuhan University(武汉大学计算机学院);School of Cyber Science and Technology, Sun Yat-sen University(中山大学网络空间科学与技术学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
Mitigating Barren Plateaus in Quantum Denoising Diffusion Probabilistic Model
School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算机科学与数据科学学院);Research Institute of Quantum Technology, The Hong Kong Polytechnic University(香港理工大学量子技术研究所)
FOCUS: Frequency-Optimized Conditioning of DiffUSion Models for mitigating catastrophic forgetting during Test-Time Adaptation
Centre for Frontier AI Research (CFAR)(前沿人工智能研究中心);Agency for Science, Technology and Research (A*STAR)(科技研究局);College of Computing and Data Science(计算与数据科学学院);Nanyang Technological University(南洋理工大学);Institute of Infocomm Research (I2R)(信息通信研究所);School of Computer Science and Technology(计算机科学与技术学院)
DiffInfinite: Large Mask-Image Synthesis via Parallel Random Patch Diffusion in Histopathology
University of Glasgow(格拉斯哥大学);Dotphoton;Fraunhofer HHI(弗劳恩霍夫海因里希·赫兹研究所);Aignostics;Institute of Pathology, LMU Munich(慕尼黑大学病理学研究所);UC Berkeley(加州大学伯克利分校);TU Berlin(柏林工业大学)
Losing dimensions: Geometric memorization in generative diffusion
Computing Sciences Department, Bocconi University(博科尼大学计算科学系);Bocconi Institute for Data Science and Analytics (BIDSA)(博科尼数据科学与分析研究所);Donders Institute, Radboud University(拉德堡德大学多纳尔斯研究所);OnePlanet Research Center, Wageningen(瓦赫宁根大学OnePlanet研究中心);Rensselaer Polytechnic Institute(伦斯勒理工学院);Jheronimus Academy of Data Science (JADS)(杰霍尼穆斯数据科学学院);IBM Research, Cambridge(IBM研究院,剑桥)