arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.00773cs.LGcs.CV
收藏

扩散调优:通过遗忘链迁移扩散模型

Diffusion Tuning: Transferring Diffusion Models via Chain of Forgetting

Tsinghua University(清华大学);BNRist(北京信息科学与技术国家研究中心)

Jincheng Zhong, Xingzhuo Guo, Jiaxiang Dong, Mingsheng Long

AI总结:

本文发现扩散模型可迁移性沿逆向过程呈遗忘链趋势,据此提出 Diff-Tuning 方法,在去噪链末端保留预训练知识、丢弃噪声侧知识,在多个下游生成任务上相比标准微调提升 26%,并加速 ControlNet 收敛 24%。

摘要:

扩散模型已显著推动生成建模领域的发展。然而,训练扩散模型的计算成本高昂,因此迫切需要将现成的扩散模型适配到下游生成任务中。当前的微调方法侧重于参数高效迁移学习,但忽视了扩散模型的基本迁移特性。在本文中,我们研究了扩散模型的可迁移性,并观察到可迁移性沿逆向过程呈现出单调的遗忘链趋势。基于这一观察和新的理论见解,我们提出了 Diff-Tuning,这是一种极其简单的迁移方法,利用遗忘链趋势。Diff-Tuning 促使微调后的模型在去噪链末端(接近生成数据处)保留预训练知识,同时丢弃另一侧(噪声侧)的知识。我们进行了全面的实验来评估 Diff-Tuning,包括将预训练的 Diffusion Transformer 模型迁移到八个下游生成任务,以及使用 ControlNet 将 Stable Diffusion 适配到五种控制条件。Diff-Tuning 相比标准微调实现了 26% 的提升,并将 ControlNet 的收敛速度提高了 24%。值得注意的是,针对扩散模型的参数高效迁移学习技术也能从 Diff-Tuning 中受益。

英文摘要:

Diffusion models have significantly advanced the field of generative modeling. However, training a diffusion model is computationally expensive, creating a pressing need to adapt off-the-shelf diffusion models for downstream generation tasks. Current fine-tuning methods focus on parameter-efficient transfer learning but overlook the fundamental transfer characteristics of diffusion models. In this paper, we investigate the transferability of diffusion models and observe a monotonous chain of forgetting trend of transferability along the reverse process. Based on this observation and novel theoretical insights, we present Diff-Tuning, a frustratingly simple transfer approach that leverages the chain of forgetting tendency. Diff-Tuning encourages the fine-tuned model to retain the pre-trained knowledge at the end of the denoising chain close to the generated data while discarding the other noise side. We conduct comprehensive experiments to evaluate Diff-Tuning, including the transfer of pre-trained Diffusion Transformer models to eight downstream generations and the adaptation of Stable Diffusion to five control conditions with ControlNet. Diff-Tuning achieves a 26% improvement over standard fine-tuning and enhances the convergence speed of ControlNet by 24%. Notably, parameter-efficient transfer learning techniques for diffusion models can also benefit from Diff-Tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑