KeepLoRA: 基于残差梯度适应的持续学习
KeepLoRA: Continual Learning with Residual Gradient Adaptation
AI总结：
KeepLoRA通过残差梯度适应方法实现持续学习，有效平衡知识保留、任务知识保持和新知识获取，取得最佳性能。
摘要：
预训练视觉-语言模型的持续学习需要平衡三个相互竞争的目标：保留预训练知识、保持一系列学习任务的知识以及保持可塑性以获取新知识。本文提出了一种简单但有效的方法，称为KeepLoRA，以有效平衡这些目标。我们首先分析了模型参数空间内的知识保留机制，发现通用知识主要编码在主子空间中，而任务特定知识编码在残差子空间中。受这一发现的启发，KeepLoRA通过限制LoRA参数更新到残差子空间来学习新任务，以防止干扰之前学习的能力。具体来说，我们通过将新任务的梯度投影到一个正交于预训练模型主子空间和先前任务特征主导方向的子空间来注入知识。我们的理论和实证分析证实，KeepLoRA平衡了这三个目标并实现了最先进的性能。实现代码可在https://github.com/MaolinLuo/KeepLoRA上获得。
英文摘要：
Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents a simple but effective approach called KeepLoRA to effectively balance these objectives. We first analyze the knowledge retention mechanism within the model parameter space and find that general knowledge is mainly encoded in the principal subspace, while task-specific knowledge is encoded in the residual subspace. Motivated by this finding, KeepLoRA learns new tasks by restricting LoRA parameter updates in the residual subspace to prevent interfering with previously learned capabilities. Specifically, we infuse knowledge for a new task by projecting its gradient onto a subspace orthogonal to both the principal subspace of pre-trained model and the dominant directions of previous task features. Our theoretical and empirical analyses confirm that KeepLoRA balances the three objectives and achieves state-of-the-art performance. The implementation code is available at https://github.com/MaolinLuo/KeepLoRA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
