在遗忘前合并：通过持续合并实现单个LoRA持续学习
Merge before Forget: A Single LoRA Continual Learning via Continual Merging
AI总结：
本文提出了一种通过持续合并实现单个LoRA持续学习的方法，利用正交初始化和时间感知缩放机制减少任务干扰并提升性能。
摘要：
参数高效持续学习已成为大型语言模型（LLMs）缓解灾难性遗忘并使模型适应新任务的一种有希望的方法。当前的低秩适应（LoRA）持续学习技术通常保留并冻结先前学习的LoRAs或生成数据表示以克服遗忘，通常利用这些来支持新LoRAs学习新任务。然而，这些方法不仅忽略了随着任务增长而增加的计算内存和存储空间的限制，还由于缺乏有效的LoRA合并机制而可能产生任务干扰。在本文中，我们提出了一种新的持续学习方法，该方法正交初始化并依次将LoRAs的更新合并到一个统一的LoRA中。我们的方法利用从先前学习的LoRA中提取的正交基来初始化新任务的学习，进一步利用LoRA组件的内在非对称性属性，通过时间感知的缩放机制在持续合并过程中平衡新旧知识。我们的方法保持了内存复杂度与任务数量之间的恒定关系，通过正交基初始化最小化过去任务和新任务之间的干扰，并通过自适应缩放改进了非对称LoRA合并的性能。我们提供了理论分析以证明我们的设计，并在各种持续学习基准上使用各种Llama模型进行了广泛的实验，证明了我们方法的有效性和效率。
英文摘要：
Parameter-efficient continual learning has emerged as a promising approach for large language models (LLMs) to mitigate catastrophic forgetting while enabling adaptation to new tasks. Current Low-Rank Adaptation (LoRA) continual learning techniques often retain and freeze previously learned LoRAs or generate data representations to overcome forgetting, typically utilizing these to support new LoRAs learn new tasks. However, these methods not only ignore growing computational memory with tasks and limited storage space but also suffer from potential task interference due to the lack of effective LoRA merging mechanisms. In this paper, we propose a novel continual learning method that orthogonally initializes and sequentially merges LoRAs updates into a single unified LoRA. Our method leverages orthogonal basis extraction from previously learned LoRA to initialize the learning of new tasks, further exploits the intrinsic asymmetry property of LoRA components by using a time-aware scaling mechanism to balance new and old knowledge during continual merging. Our approach maintains constant memory complexity with respect to the number of tasks, minimizes interference between past and new tasks via orthogonal basis initialization, and improves performance over asymmetric LoRA merging via adaptive scaling. We provide theoretical analysis to justify our design and conduct extensive experiments across diverse continual learning benchmarks using various Llama models, demonstrating the effectiveness and efficiency of our method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
