KeepLoRA++: 基于层级缩放残差梯度适应的持续学习
KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation
AI总结：
针对预训练视觉语言模型持续学习中保留预训练知识、旧任务知识和学习新知识的冲突，提出KeepLoRA++，通过层级缩放残差梯度适应方法，限制LoRA参数更新到残差子空间并采用浅到深层缩放，平衡三者，在图像分类、视觉问答和视频理解任务上优于基线。
摘要：
预训练视觉语言模型的持续学习需要平衡三个相互竞争的目标：保留预训练知识、保留一系列已学习任务的知识以及保持获取新知识的可塑性。本文提出KeepLoRA++，通过统一的二维知识保留机制来平衡这些目标。我们从层间和层内两个角度分析Transformer架构的知识分布。层间视角考察知识保留如何跨层分布，而层内视角关注每层内的参数空间。我们的分析揭示了一个结构特性：通用可迁移知识主要编码在浅层和参数的主子空间中，而任务特定适应则定位于深层和残差子空间。受此启发，KeepLoRA++引入了一种层级缩放残差梯度适应方法。新任务的学习通过将LoRA参数更新限制在残差子空间，并结合从浅到深的层级缩放来实现，以防止干扰先前获得的能力。具体而言，新任务的梯度被投影到与预训练模型主子空间以及先前任务特征主导方向正交的子空间上，同时为浅层分配较小的更新幅度，为深层分配较大的更新幅度。我们的理论分析和实证评估证实，KeepLoRA++成功平衡了这三个相互竞争的目标，在图像分类、视觉问答和视频理解任务上持续优于代表性基线。
英文摘要：
Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents KeepLoRA++, balancing these objectives through a unified dual-dimensional knowledge retention mechanism. We analyze knowledge distribution of Transformer architecture from both inter-layer and intra-layer perspectives. The inter-layer perspective examines how retention is distributed across layers, while the intra-layer perspective focuses on the parameter space within each layer. Our analysis reveals a structural property: general transferable knowledge is mainly encoded in the shallow layers and the principal subspace of the parameters, while task-specific adaptations are localized in the deep layers and the residual subspace. Motivated by this insight, KeepLoRA++ introduces a layer-scaled residual gradient adaptation method. New tasks are learned by restricting LoRA parameter updates to the residual subspace, combined with a shallow-to-deep layer scaling, to prevent interference with previously acquired capabilities. Specifically, the gradient of a new task is projected onto a subspace orthogonal to both the principal subspace of the pre-trained model and the dominant directions of previous task features, while simultaneously assigning smaller update magnitudes to shallow layers and larger ones to deeper layers. Our theoretical analysis and empirical evaluations confirm that KeepLoRA++ successfully balances these three competing objectives, consistently outperforming representative baselines across image classification, visual question answering, and video understanding tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
