Recent breakthroughs in diffusion models have exhibited exceptional image-generation capabilities. However, studies show that some outputs are merely replications of training data. Such replications present potential legal challenges for model owners, especially when the generated content contains proprietary information. In this work, we introduce a straightforward yet effective method for detecting memorized prompts by inspecting the magnitude of text-conditional predictions. Our proposed method seamlessly integrates without disrupting sampling algorithms, and delivers high accuracy even at the first generation step, with a single generation per prompt. Building on our detection strategy, we unveil an explainable approach that shows the contribution of individual words or tokens to memorization. This offers an interactive medium for users to adjust their prompts. Moreover, we propose two strategies i.e., to mitigate memorization by leveraging the magnitude of text-conditional predictions, either through minimization during inference or filtering during training. These proposed strategies effectively counteract memorization while maintaining high-generation quality. Code is available at https://github.com/YuxinWenRick/diffusion_memorization.
Redistribute Ensemble Training for Mitigating Memorization in Diffusion Models
School of Computer Science, Wuhan University(武汉大学计算机学院);School of Cyber Science and Technology, Sun Yat-sen University(中山大学网络空间科学与技术学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
Filtering Memorization from Parameter-Space in Diffusion Models
RIKEN AIP(理化学研究所Advanced Institute for Peripheral Research);Science of Tokyo(东京科学大学);University of California, Berkeley(加州大学伯克利分校);Zhejiang University(浙江大学)
Unconsciously Forget: Mitigating Memorization; Without Knowing What is being Memorized
RWTH Aachen University(亚琛工业大学);National University of Singapore(新加坡国立大学);Fraunhofer Institute for Applied Information Technology FIT(弗劳恩霍夫应用信息技术研究所FIT)
CrossSplit: Mitigating Label Noise Memorization through Data Splitting
Samsung Advanced Institute of Technology (SAIT)(三星高级技术研究院(SAIT));SAIT AI Lab(三星高级技术研究院人工智能实验室);Mila(米拉研究所);Université de Montreal(蒙特利尔大学);Canada CIFAR AI Chair(加拿大CIFAR人工智能讲席)
Towards Application Aligned Synthetic Surgical Image Synthesis
Department of Translational Surgical Oncology, NCT/UCC Dresden(转化外科肿瘤学系);DKFZ, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden,HZDR, Germany(德累斯顿技术大学医学院);The Centre for Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden(人机交互 tactile 网络中心)
Test-Time Training with KV Binding Is Secretly Linear Attention
NVIDIA, Toronto, Ontario, Canada(NVIDIA,多伦多,安大略省,加拿大);University of Toronto, Toronto, Ontario, Canada(多伦多大学,多伦多,安大略省,加拿大);Vector Institute, Toronto, Ontario, Canada(向量研究所,多伦多,安大略省,加拿大);Technion -- Israel Institute of Technology, Haifa, Israel(技术ion -- 以色列理工学院,海法,以色列)
NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training
Johns Hopkins University(约翰斯·霍普金斯大学);University of California, Berkeley(加州大学伯克利分校);University of California San Diego(加州大学圣地亚哥分校);Duke University(杜克大学)