arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.22149cs.CV
收藏

容量控制是文本条件扩散模型中一种有效的记忆缓解机制

Capacity Control is an Effective Memorization Mitigation Mechanism in Text-Conditional Diffusion Models

School of Informatics, University of Edinburgh(爱丁堡大学信息学院);School of Engineering, University of Edinburgh(爱丁堡大学工程学院);Samsung AI Center, Cambridge(三星剑桥人工智能中心)

Raman Dutt, Pedro Sanchez, Ondrej Bohdal, Sotirios A. Tsaftaris, Timothy Hospedales

AI总结:

该研究发现在文本条件扩散模型微调中采用PEFT控制模型容量,可在使用MIMIC数据集的实验中同时减少记忆问题并提升下游生成质量,还可与现有缓解技术结合进一步优化。

摘要:

在本研究中,我们提供了令人信服的证据,证明在微调期间控制模型容量可有效缓解扩散模型中的记忆问题。具体而言,我们证明在预训练-微调范式下采用参数高效微调(Parameter-Efficient Fine-Tuning, PEFT),与传统全微调方法相比能显著减少记忆。我们的实验使用MIMIC数据集,该数据集包含胸部X光片及其对应报告的图文对。通过一系列记忆指标和生成质量指标评估的结果表明,PEFT不仅能减少记忆,还能提升下游生成质量。此外,PEFT方法可与现有记忆缓解技术无缝结合以实现进一步提升。我们的实验代码可在以下网址获取:https://github.com/Raman1121/Diffusion_Memorization_HPO

英文摘要:

In this work, we present compelling evidence that controlling model capacity during fine-tuning can effectively mitigate memorization in diffusion models. Specifically, we demonstrate that adopting Parameter-Efficient Fine-Tuning (PEFT) within the pre-train fine-tune paradigm significantly reduces memorization compared to traditional full fine-tuning approaches. Our experiments utilize the MIMIC dataset, which comprises image-text pairs of chest X-rays and their corresponding reports. The results, evaluated through a range of memorization and generation quality metrics, indicate that PEFT not only diminishes memorization but also enhances downstream generation quality. Additionally, PEFT methods can be seamlessly combined with existing memorization mitigation techniques for further improvement. The code for our experiments is available at: https://github.com/Raman1121/Diffusion_Memorization_HPO

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑