发表机构
Shanghai Jiao Tong University; Shanghai Innovation Institute; Beijing Zhongguancun Academy; StepFun; Huazhong University of Science and Technology; IAAR（上海交通大学; 上海创新研究院; 北京中关村学院; 阶跃星辰; 华中科技大学; 智能自动化研究院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
ChainLoRA提出一种基于链式更新任务向量几何的无重放持续合并框架，通过自适应SVD合并与Procrustes适应实现几何近似分离，在多个基准上达到先进性能。
AI 中文摘要
大语言模型（LLMs）的持续参数高效微调必须平衡先前知识的保留、对新任务的适应以及严格的参数预算。我们提出了ChainLoRA，一种基于链式更新任务向量几何的无重放持续合并框架。从参数合并的角度，我们通过任务更新之间的可测量交互构建了遗忘的几何视图，将方向重叠与系数耦合区分开来。基于这一视图，ChainLoRA将链式更新训练与流后自适应SVD合并相结合。在训练期间，初始化和单侧正交性代理仅使用最后一个载体，随着任务流的增长，其历史状态足迹和正则化开销保持不变。在合并时，自适应SVD提取共享载体，并通过Procrustes适应将其对齐到最新任务。我们的理论分析表明，Procrustes适应有助于共享组件和任务特定组件的几何近似分离。单侧代理进一步限制了任务间干扰。有效秩惩罚还促进了持续学习期间任务子空间的高效利用。实验表明，在Large和SuperNI基准上，ChainLoRA在评估的无重放方法中实现了最先进的性能，同时在Standard CL上保持竞争力，并在所有三个基准上获得了与评估的基于重放方法几乎最接近的平均分数。
英文摘要
Continual parameter-efficient fine-tuning for large language models (LLMs) must balance retention of previously acquired knowledge, adaptation to new tasks, and strict parameter budgets. We present \textbf{ChainLoRA}, a replay-free continual merging framework built on chain-updated task-vector geometry. From a parameter-merging perspective, we formulate a geometric view of forgetting through a measurable interaction between task updates, separating directional overlap from coefficient coupling. Building on this view, ChainLoRA combines chain-updated training with post-stream adaptive SVD merging. During training, initialization and a one-sided orthogonality proxy use only the last carrier, keeping their historical-state footprint and regularization overhead constant as the task stream grows. At merging time, Adaptive SVD extracts a shared carrier and aligns it to the latest task through Procrustes adaptation. Our theoretical analysis shows that Procrustes adaptation facilitates geometric approximate separation of shared and task-specific components. The one-sided proxy further bounds inter-task interference. An effective-rank penalty additionally promotes efficient utilization of the task subspace during continual learning. Experiments show that ChainLoRA achieves state-of-the-art performance among the evaluated replay-free methods on the Large and SuperNI benchmarks, while remaining competitive on Standard CL and attaining almost the closest average scores to the evaluated replay-based method across all three benchmarks.
Comments18 pages
