arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.16145cs.CVcs.AIcs.LG
收藏

一维适配器统御一切:概念、扩散模型与擦除应用

One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications

Tsinghua University(清华大学);BNRist(北京国家信息科学技术研究院);Alibaba Group(阿里巴巴集团)

Mengyao Lyu, Yuhong Yang, Haiwen Hong, Hui Chen, Xuan Jin, Yuan He, Hui Xue, Jungong Han, Guiguang Ding

AI总结:

本研究针对现有扩散模型概念擦除方法存在的生成侵蚀、迁移性差等问题,提出基于一维适配器的半透膜结构SPM及潜在锚定微调、易化转运机制,实现了非侵入式、可迁移的多概念精准擦除。

摘要:

商用和开源扩散模型(DMs)在文本生成图像领域的广泛应用促使人们开展风险缓解工作,以防范非预期行为。学术界现有的概念擦除方法均基于全参数或基于特定规范的微调,我们从中发现了以下问题:1)生成效果出现侵蚀性偏移:目标消除过程中的参数漂移会导致所有生成结果出现偏移和潜在变形,甚至会不同程度地侵蚀其他概念,在多概念擦除场景下这一问题更为明显;2)迁移能力不足与部署效率低下:以往针对特定模型的擦除方法阻碍了概念的灵活组合,也无法向其他模型进行免训练迁移,导致部署场景增加时成本呈线性增长。为实现非侵入式、精准、可定制且可迁移的概念消除,我们将擦除框架建立在一维适配器之上,可在多种擦除应用中同时从大多数扩散模型中擦除多个概念。我们将概念半透膜结构(SPM,SemiPermeable Membrane)注入任意扩散模型中以学习针对性擦除,同时通过一种新颖的潜在锚定(Latent Anchoring)微调策略有效缓解偏移与侵蚀现象。一旦训练完成,SPM可灵活组合,且能即插即用于其他扩散模型,无需专门重新微调,能够及时高效地适配多样化场景。在生成过程中,我们的易化转运(Facilitated Transport)机制会动态调节每个SPM的通透性,以响应不同的输入提示,进一步降低对其他概念的影响。在约40个概念、7个扩散模型和4种擦除应用上的定量与定性结果均证明了SPM具备卓越的擦除性能。我们的代码和预调优SPM可在项目页面https://lyumengyao.github.io/projects/spm获取。

英文摘要:

The prevalent use of commercial and open-source diffusion models (DMs) for text-to-image generation prompts risk mitigation to prevent undesired behaviors. Existing concept erasing methods in academia are all based on full parameter or specification-based fine-tuning, from which we observe the following issues: 1) Generation alternation towards erosion: Parameter drift during target elimination causes alternations and potential deformations across all generations, even eroding other concepts at varying degrees, which is more evident with multi-concept erased; 2) Transfer inability & deployment inefficiency: Previous model-specific erasure impedes the flexible combination of concepts and the training-free transfer towards other models, resulting in linear cost growth as the deployment scenarios increase. To achieve non-invasive, precise, customizable, and transferable elimination, we ground our erasing framework on one-dimensional adapters to erase multiple concepts from most DMs at once across versatile erasing applications. The concept-SemiPermeable structure is injected as a Membrane (SPM) into any DM to learn targeted erasing, and meantime the alteration and erosion phenomenon is effectively mitigated via a novel Latent Anchoring fine-tuning strategy. Once obtained, SPMs can be flexibly combined and plug-and-play for other DMs without specific re-tuning, enabling timely and efficient adaptation to diverse scenarios. During generation, our Facilitated Transport mechanism dynamically regulates the permeability of each SPM to respond to different input prompts, further minimizing the impact on other concepts. Quantitative and qualitative results across ~40 concepts, 7 DMs and 4 erasing applications have demonstrated the superior erasing of SPM. Our code and pre-tuned SPMs are available on the project page https://lyumengyao.github.io/projects/spm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑