arXiv 2605.284952026-05-27cs.CV

Janus-LoRA：面向持续学习的平衡低秩适配

Janus-LoRA: A Balanced Low-Rank Adaptation for Continual Learning

School of Computer Science and Technology, Tongji University, Shanghai, China（同济大学计算机科学与技术学院，上海，中国）；School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China（电子科技大学计算机科学与工程学院，成都，中国）；Shanghai Innovation Institute, Shanghai, China（上海创新研究院，上海，中国）；Independent Researcher（独立研究者）

Cheng Chen, Pengpeng Zeng, Yuyu Guo, Lianli Gao, Hengtao Shen, Jingkuan Song

备注：: 9pages, International Conference on Machine Learning

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出Janus-LoRA框架，通过梯度修正实现参数级正交性以克服灾难性遗忘，并利用解耦边际损失增强特征级分离，从而在持续学习中平衡稳定性与可塑性。

摘要：

低秩适配（LoRA）已成为持续学习的一种有前景的范式。它独立更新其低秩因子（$A$和$B$），通过它们的相互作用对完整权重矩阵产生复合更新。为了防止灾难性遗忘，该更新应保持与包含先前学习知识的任务特定子空间正交。然而，我们发现这种复合更新系统性地违反了这种正交性，重新引入了干扰并破坏了稳定性。此外，天真地强制执行这种正交性会损害可塑性，破坏微妙的稳定性-可塑性权衡。为了解决这些问题，我们提出了 extbf{Janus-LoRA}框架，通过两个新颖的组件恢复这种平衡。具体来说，我们首先引入梯度修正，这是一种闭式解，数学上解耦LoRA的因子更新，针对通过高效在线估计识别的历史知识子空间强制执行正交性。接下来，为了增强可塑性，我们引入解耦边际损失，通过将新特征表示推离旧特征表示来促进特征级分离，从而为新学习创建独特、低干扰的区域。在具有挑战性的基准上的全面实验表明，通过协调参数级正交性与特征级分离，Janus-LoRA实现了优越的平衡，并建立了新的最先进性能。

英文摘要：

Low-Rank Adaptation (LoRA) has emerged as a promising paradigm for Continual Learning. It independently updates its low-rank factors ($A$ and $B$), creating a composite update to the full weight matrix through their interaction. To prevent catastrophic forgetting, this update should remain orthogonal to the task-specific subspace that contains previously learned knowledge. However, we identify that this composite update systematically violates this orthogonality, reintroducing interference and undermining stability. Furthermore, naively enforcing this orthogonality compromises plasticity, disrupting the delicate stability-plasticity trade-off. To resolve these issues, we propose \textbf{Janus-LoRA}, a framework that restores this balance through two novel components. Specifically, we first introduce Gradient Rectification, a closed-form solution that mathematically decouples LoRA's factor updates, enforcing orthogonality against the historical knowledge subspace identified by an efficient Online Estimation. Next, to enhance plasticity, we introduce a Decoupled Margin Loss that promotes feature-level separation by pushing new feature representations away from old ones, thus creating distinct, low-interference regions for new learning. Comprehensive experiments on challenging benchmarks demonstrate that by harmonizing parameter-level orthogonality with feature-level separation, Janus-LoRA achieves a superior balance and establishes new state-of-the-art performance.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。