Recent personalization methods for diffusion models, such as Dreambooth and LoRA, allow fine-tuning pre-trained models to generate new concepts. However, applying these techniques across consecutive tasks in order to include, e.g., new objects or styles, leads to a forgetting of previous knowledge due to mutual interference between their adapters. In this work, we tackle the problem of continual customization under a rigorous regime with no access to past tasks' adapters. In such a scenario, we investigate how different adapters' initialization and merging methods can improve the quality of the final model. To that end, we evaluate the naive continual fine-tuning of customized models and compare this approach with three methods for consecutive adapters' training: sequentially merging new adapters, merging orthogonally initialized adapters, and updating only relevant task-specific weights. In our experiments, we show that the proposed techniques mitigate forgetting when compared to the naive approach. In our studies, we show different traits of selected techniques and their effect on the plasticity and stability of the continually adapted model. Repository with the code is available at https://github.com/luk-st/continual-lora.
LoRA Diffusion: Zero-Shot LoRA Synthesis for Diffusion Model Personalization
Leonardo AI Research Lab(莱昂纳多人工智能研究实验室);Lucid Simulations, Corp.(Lucid Simulations公司);Precigenetics Inc(Precigenetics公司);SimpleBerry Research Lab(SimpleBerry研究实验室);University of California, Merced(加州大学默塞德分校)
Revisiting Weight Regularization for Low-Rank Continual Learning
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家重点实验室);Institute of Artificial Intelligence and Robotics(人工智能与机器人研究院);Computer Vision Center(计算机视觉中心);Universitat Autònoma de Barcelona(巴塞罗那自治大学);School of Instrument Science and Engineering(仪器科学与工程学院);Bernoulli Institute(伯努利学院);University of Groningen(格罗宁根大学);School of Software Engineering(软件工程学院)
C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models
Institute of Intelligent Information Processing, Shanxi University(山西大学智能信息处理研究所);Alliance Manchester Business School, The University of Manchester(曼彻斯特大学联盟曼彻斯特商学院)
Continual Gradient Low-Rank Projection Fine-Tuning for LLMs
Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence(北京交通大数据挖掘与具身智能重点实验室);School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院);State Key Laboratory of Advanced Rail Autonomous Operation(先进轨道交通自主运行国家重点实验室)
G2LoRA: Gradient Orthogonal Low-Rank Adaptation Framework for Graph Continual Learning on Text-Attributed Graphs
School of Computer Science and Engineering, Beihang University(北航计算机科学与工程学院);Department of Statistics, Columbia University(哥伦比亚大学统计系);College of Computer Science, Beijing University of Technology(北京理工大学计算机学院)
Hessian Aware Low-Rank Perturbation for Order-Robust Continual Learning
University of Western Ontario(西安大略大学);Vector Institute(矢量研究院);Beihang University(北京航空航天大学);Mcgill University(麦吉尔大学);Laval University(拉瓦尔大学);Mila(米拉研究所)
Energy-Structured Low-Rank Adaptation for Continual Learning
School of Computer Science and Engineering, Southeast University, Nanjing, China(东南大学计算机科学与工程学院,南京,中国);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其交叉应用重点实验室(东南大学),教育部,中国);Huawei Technologies, Shenzhen, China(华为技术有限公司,深圳,中国)
CLoRA: Parameter-Efficient Continual Learning with Low-Rank Adaptation
Augmented Vision Group, German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心增强视觉组);RPTU – University of Kaiserslautern-Landau(凯撒斯劳滕-兰道大学)