arXiv 2603.001912026-05-24cs.LGcs.CV

基于LoRA的持续学习中任务驱动的子空间分解用于知识共享与隔离

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

Department of XXX, University of YYY, Location, Country（XXX部门，YYY大学，地点，国家）；School of ZZZ, Institute of WWW, Location, Country（ZZZ学院，WWW研究所，地点，国家）；State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi'an, China（信息服务网络国家重点实验室，电信工程学院，西安电子科技大学，西安，中国）；School of Electronic Engineering, Xidian University, Xi'an, China（电子工程学院，西安电子科技大学，西安，中国）

Lingfeng He, De Cheng, Huaijie Wang, Xi Yang, Nannan Wang, Xinbo Gao

备注：: Accepted by ICML 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出LoDA方法，通过任务驱动分解构建通用和任务特定LoRA子空间，结合梯度对齐优化和闭式重校准，实现知识共享与隔离，提升持续学习性能。

摘要：

持续学习要求模型在不遗忘旧知识的情况下顺序适应新任务。最近，低秩适应（LoRA）作为一种代表性的参数高效微调方法，在持续学习中受到越来越多的关注。几种基于LoRA的持续学习方法通过分离更新空间来减少任务间的干扰，通常从过去任务的估计零空间中构建新空间。然而，它们（i）忽略了任务共享方向，抑制了知识迁移；（ii）未能捕获真正有效的任务特定方向，因为旧任务的这些“零基”在相关任务下对新任务几乎保持不活跃。为了解决这个问题，我们从投影能量的角度研究LoRA的学习能力，并提出了低秩分解与适应（LoDA）。它通过解决两个基于能量的目标，执行任务驱动分解以构建通用和真正的任务特定LoRA子空间，解耦知识共享和隔离的方向。LoDA固定两个子空间上的LoRA下投影，并通过梯度对齐优化方法学习鲁棒的上投影。在每个任务之后，在将LoRA更新集成到主干之前，LoDA为通用更新推导出一个闭式重校准，沿着这个任务共享方向近似特征级联合最优。实验表明，LoDA优于现有的持续学习方法。我们的代码可在https://github.com/HHHLF/LoDA_ICML2026获取。

英文摘要：

Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL. Several LoRA-based CL methods reduce interference across tasks by separating their update spaces, typically building the new space from the estimated null space of past tasks. However, they (i) overlook task-shared directions, which suppresses knowledge transfer, and (ii) fail to capture truly effective task-specific directions since these ``null bases" of old tasks can remain nearly inactive for new task under correlated tasks. To address this, we study LoRA learning capability from a projection energy perspective, and propose Low-rank Decomposition and Adaptation (LoDA). It performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based objectives, decoupling directions for knowledge sharing and isolation. LoDA fixes LoRA down-projections on two subspaces and learns robust up-projections via a Gradient-Aligned Optimization (GAO) approach. After each task, before integrating the LoRA updates into the backbone, LoDA derives a closed-form recalibration for the general update, approximating a feature-level joint optimum along this task-shared direction. Experiments indicate that LoDA outperforms existing CL methods. Our code is available at https://github.com/HHHLF/LoDA_ICML2026.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。