SPACE-LoRA：为持续学习分配激活子空间保护
SPACE-LoRA: Allocating Activation-Subspace Protection for Continual Learning
- Korea University（高丽大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对LoRA持续学习中的灾难性遗忘，提出SPACE-LoRA，通过抑制新残差分支在旧任务敏感输入方向的响应并自适应分配保护覆盖，有效缓解干扰。
AI中文摘要：
本研究从终身学习的视角，探讨了在使用低秩适配（LoRA）顺序学习连续任务时出现的灾难性遗忘问题。现有方法主要通过约束参数更新或学习子空间来减少对过去知识的干扰，但未充分考虑加性干扰。这种干扰发生在固定过去模型之上新添加的残差适配器，沿对旧任务重要的输入方向产生非零响应，从而改变先前的预测。为此，我们提出了具有分配容量的子空间保护以实现高效持续适配（SPACE-LoRA）。SPACE-LoRA直接抑制新残差分支沿对旧任务重要的输入激活方向的响应，并基于通过通用Fisher敏感性覆盖目标估计的过去任务敏感性，自适应地确定每个模块的保护覆盖范围。在固定LoRA秩下，该方法自适应调整模块特定的保护覆盖范围，同时抑制沿对旧任务敏感输入方向的干扰。我们评估了激活子空间保护在缓解灾难性遗忘方面的有效性，并考察了敏感性引导保护在多样化任务持续学习中的作用。代码可在该https URL获取。
英文摘要：
This study addresses the catastrophic forgetting problem that occurs when sequentially learning successive tasks using Low-Rank Adaptation (LoRA) from a lifelong learning perspective. While existing approaches have primarily constrained parameter updates or learning subspaces to reduce interference with past knowledge, they have not fully considered additive interference. This occurs when a newly added residual adapter on top of a fixed past model generates non-zero responses along input directions important for old tasks, thereby altering previous predictions. To this end, we propose Subspace Protection with Allocated Capacity for Efficient Continual Adaptation (SPACE-LoRA). SPACE-LoRA directly suppresses the responses of the new residual branch along input activation directions that are important for old tasks and adaptively determines the protection coverage for each module based on past-task sensitivity estimated via a common Fisher sensitivity-based coverage target. Under a fixed LoRA rank, this approach adaptively adjusts module-specific protection coverage while suppressing interference along input directions sensitive to old tasks. We assess the effectiveness of activation-subspace protection in mitigating catastrophic forgetting and examine the role of sensitivity-guided protection in continual learning across diverse tasks. Code is available at https://anonymous.4open.science/r/SPACE-LoRA-7864.
