Fine-tuning large language models (LLMs) with Low-Rank adaption (LoRA) is widely acknowledged as an effective approach for continual learning for new tasks. However, it often suffers from catastrophic forgetting when dealing with multiple tasks sequentially. To this end, we propose Attentional Mixture of LoRAs (AM-LoRA), a continual learning approach tailored for LLMs. Specifically, AM-LoRA learns a sequence of LoRAs for a series of tasks to continually learn knowledge from different tasks. The key of our approach is that we devise an attention mechanism as a knowledge mixture module to adaptively integrate information from each LoRA. With the attention mechanism, AM-LoRA can efficiently leverage the distinctive contributions of each LoRA, while mitigating the risk of mutually negative interactions among them that may lead to catastrophic forgetting. Moreover, we further introduce $L1$ norm in the learning process to make the attention vector more sparse. The sparse constraints can enable the model to lean towards selecting a few highly relevant LoRAs, rather than aggregating and weighting all LoRAs collectively, which can further reduce the impact stemming from mutual interference. Experimental results on continual learning benchmarks indicate the superiority of our proposed method.
Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
College of Health Solutions, Arizona State University（亚利桑那州立大学健康解决方案学院）；Department of Electrical and Computer Engineering, The University of Texas at Austin（德克萨斯大学奥斯汀分校电气与计算机工程系）；School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）；The GAME School, Arizona State University（亚利桑那州立大学GAME学院）
Efficient Continual Learning in Neural Machine Translation: A Low-Rank Adaptation Approach
Pattern Recognition and Human Language Technology, Universitat Politècnica de València（模式识别与人类语言技术，瓦伦西亚理工大学）；Valencian Graduate School and Research Network of Artificial Intelligence, Universitat Politècnica de València（瓦伦西亚人工智能研究生学院与研究网络，瓦伦西亚理工大学）
Is Parameter Collision Hindering Continual Learning in LLMs?
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Peng Cheng Laboratory（鹏城实验室）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Hupan Lab（湖畔实验室）
Enhancing Multimodal Continual Instruction Tuning with BranchLoRA
Mohamed bin Zayed University of Artificial Intelligence（Mohamed bin Zayed大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Kyoto University（京都大学）；Tencent AI Lab（腾讯AI实验室）；Tomorrow Advancing Life（明天生命科技）
Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning
College of Information Sciences and Technology, Pennsylvania State University（宾夕法尼亚州立大学信息科学与技术学院）；School of Computer Science and Technology, Singapore Management University（新加坡管理大学计算机科学与技术学院）；School of Computer Science, Hefei Univeristy of Technology（合肥工业大学计算机学院）
Efficient Training of Robust Traditional Chinese LLaMA-1B on a Single Consumer GPU: Continual Pre-training, SFT, and DPO
National Tsing Hua University（台湾清华大学）；Accton Technology Corporation（智邦科技公司）；National Yang Ming Chiao Tung University（阳明交通大学）；University of Taipei（台北市立大学）
MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
Department of Data Science & AI, Monash University, Australia（数据科学与人工智能系，墨尔本大学，澳大利亚）；Department of Human Centred Computing, Monash University, Australia（以人为中心计算系，墨尔本大学，澳大利亚）
Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models
Zhejiang University（浙江大学）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新区（滨江）区块链与数据安全研究院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；National University of Singapore（新加坡国立大学）；Guangdong OPPO Mobile Telecommunications Corp., Ltd.（广东欧珀移动通信有限公司）
SECURA: Sigmoid-Enhanced CUR Decomposition with Uninterrupted Retention and Low-Rank Adaptation in Large Language Models
Aberdeen Institute of Data Science and Artificial Intelligence, South China Normal University（华南师范大学阿伯丁数据科学与人工智能学院）；Department of Computing Science, University of Aberdeen（阿伯丁大学计算科学系）
