SplitLoRA：通过梯度空间划分平衡持续学习中的稳定性与可塑性
SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting
AI总结：
针对持续学习中梯度投影法难平衡稳定性与可塑性的问题，提出基于LoRA的SplitLoRA方法，通过理论分析推导梯度空间最优划分，在多数据集上达到SOTA性能。
摘要：
持续学习要求模型按顺序学习多个任务，同时兼具稳定性（保留此前已学任务的知识）与可塑性（有效学习新任务）。梯度投影已成为持续学习领域一种有效且流行的范式，它将此前已学任务的梯度空间划分为两个正交子空间：主子空间和次要子空间。新任务在次要子空间内被有效学习，从而减少对已获取知识的干扰。然而，现有梯度投影方法难以在可塑性与稳定性之间实现最优平衡，因为很难恰当地划分梯度空间。在本研究中，我们关注一种基于低秩适配（Low-Rank Adaptation, LoRA）的持续学习范式——该范式因效率高、适用范围广而受到广泛关注，并提出了一种名为SplitLoRA的新型持续学习方法。我们首先从理论层面分析了子空间划分如何影响模型的稳定性与可塑性。基于这一分析，我们随后提出了一种有效方法，可推导得出此前已学任务的梯度空间最优划分。该方法有效平衡了持续学习中的稳定性与可塑性。在多个数据集上的实验结果表明，所提方法达到了当前最优（state-of-the-art）性能。
英文摘要：
Continual Learning requires a model to learn multiple tasks in sequence while maintaining both stability:preserving knowledge from previously learned tasks, and plasticity:effectively learning new tasks. Gradient projection has emerged as an effective and popular paradigm in CL, where it partitions the gradient space of previously learned tasks into two orthogonal subspaces: a primary subspace and a minor subspace. New tasks are learned effectively within the minor subspace, thereby reducing interference with previously acquired knowledge. However, existing Gradient Projection methods struggle to achieve an optimal balance between plasticity and stability, as it is hard to appropriately partition the gradient space. In this work, we consider a continual learning paradigm based on Low-Rank Adaptation, which has gained considerable attention due to its efficiency and wide applicability, and propose a novel approach for continual learning, called SplitLoRA. We first provide a theoretical analysis of how subspace partitioning affects model stability and plasticity. Informed by this analysis, we then introduce an effective method that derives the optimal partition of the gradient space for previously learned tasks. This approach effectively balances stability and plasticity in continual learning. Experimental results on multiple datasets demonstrate that the proposed method achieves state-of-the-art performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
