arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.15328cs.CV
收藏

基于反梯度控制的迭代集成训练用于缓解扩散模型中的记忆化问题

Iterative Ensemble Training with Anti-Gradient Control for Mitigating Memorization in Diffusion Models

School of Computer Science, Wuhan University(武汉大学计算机学院);School of Cyber Science and Technology, Sun Yat-sen University(中山大学网络空间安全学院)

Xiao Liu, Xiaoliu Guan, Yu Wu, Jiaxu Miao

AI总结:

本文提出迭代集成训练与反梯度控制方法,通过多分片训练、参数聚合及排除低损失样本,在视觉模态上有效缓解扩散模型的数据记忆化问题,并兼顾性能与计算成本。

摘要:

扩散模型以其生成新颖且高质量样本的强大能力而闻名,但近期因其数据记忆化行为引发了隐私风险方面的担忧。现有的记忆缓解方法要么只关注跨模态生成任务中的文本模态问题,要么采用数据增强策略。本文从视觉模态的角度提出了一种新颖的扩散模型训练框架,该框架在缓解记忆化方面更具通用性和基础性。为了促进扩散模型参数中已存储信息的遗忘,我们提出了一种迭代集成训练策略,将数据划分为多个分片以训练多个模型,并间歇性地聚合这些模型参数。此外,对损失的实际分析表明,容易记忆的图像其训练损失往往明显更低。因此,我们提出了一种反梯度控制方法,将损失值较低的样本从当前小批量中排除,以避免对其产生记忆。在四个数据集上进行了大量实验和分析,以说明我们方法的有效性,结果表明我们的方法成功降低了记忆容量,同时甚至略微提升了性能。此外,为了节省计算成本,我们成功地将该方法应用于在有限轮数内微调已训练好的扩散模型,证明了我们方法的适用性。代码可在 https://github.com/liuxiao-guan/IET_AGC 获取。

英文摘要:

Diffusion models, known for their tremendous ability to generate novel and high-quality samples, have recently raised concerns due to their data memorization behavior, which poses privacy risks. Recent approaches for memory mitigation either only focused on the text modality problem in cross-modal generation tasks or utilized data augmentation strategies. In this paper, we propose a novel training framework for diffusion models from the perspective of visual modality, which is more generic and fundamental for mitigating memorization. To facilitate forgetting of stored information in diffusion model parameters, we propose an iterative ensemble training strategy by splitting the data into multiple shards for training multiple models and intermittently aggregating these model parameters. Moreover, practical analysis of losses illustrates that the training loss for easily memorable images tends to be obviously lower. Thus, we propose an anti-gradient control method to exclude the sample with a lower loss value from the current mini-batch to avoid memorizing. Extensive experiments and analysis on four datasets are conducted to illustrate the effectiveness of our method, and results show that our method successfully reduces memory capacity while even improving the performance slightly. Moreover, to save the computing cost, we successfully apply our method to fine-tune the well-trained diffusion models by limited epochs, demonstrating the applicability of our method. Code is available in https://github.com/liuxiao-guan/IET_AGC.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑