arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.04891cs.LG
收藏

扩散模型的低秩持续个性化

Low-Rank Continual Personalization of Diffusion Models

Warsaw University of Technology(华沙理工大学);IDEAS NCBR

Łukasz Staniszewski, Katarzyna Zaleska, Kamil Deja

AI总结:

本文研究扩散模型在连续个性化任务中的适配器干扰与遗忘问题,提出并比较多种 LoRA 适配器初始化、合并及选择性更新策略,证明其可改善持续学习的稳定性与可塑性。

摘要:

近期针对扩散模型的个性化方法(如 DreamBooth 和 LoRA)允许对预训练模型进行微调,以生成新概念。然而,为纳入新对象或风格等内容而在连续任务上应用这些技术时,由于其适配器之间的相互干扰,会导致对先前知识的遗忘。在本工作中,我们在无法访问过去任务适配器的严格设定下,解决持续定制问题。在这种场景中,我们研究不同适配器的初始化和合并方法如何提升最终模型的质量。为此,我们评估了定制模型的朴素持续微调,并将该方法与三种连续适配器训练方法进行比较:顺序合并新适配器、合并正交初始化的适配器,以及仅更新相关的任务特定权重。实验表明,与朴素方法相比,所提出的技术能够缓解遗忘。研究还展示了所选技术的不同特征,及其对持续适配模型可塑性和稳定性的影响。包含代码的仓库可在 https://github.com/luk-st/continual-lora 获取。

英文摘要:

Recent personalization methods for diffusion models, such as Dreambooth and LoRA, allow fine-tuning pre-trained models to generate new concepts. However, applying these techniques across consecutive tasks in order to include, e.g., new objects or styles, leads to a forgetting of previous knowledge due to mutual interference between their adapters. In this work, we tackle the problem of continual customization under a rigorous regime with no access to past tasks' adapters. In such a scenario, we investigate how different adapters' initialization and merging methods can improve the quality of the final model. To that end, we evaluate the naive continual fine-tuning of customized models and compare this approach with three methods for consecutive adapters' training: sequentially merging new adapters, merging orthogonally initialized adapters, and updating only relevant task-specific weights. In our experiments, we show that the proposed techniques mitigate forgetting when compared to the naive approach. In our studies, we show different traits of selected techniques and their effect on the plasticity and stability of the continually adapted model. Repository with the code is available at https://github.com/luk-st/continual-lora.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑