参数高效持续学习与动态低秩适应
Parameter Efficient Continual Learning with Dynamic Low-Rank Adaptation
AI总结：
PEARL通过动态低秩适应实现参数高效的持续学习，有效解决灾难性遗忘问题，适用于多种视觉架构和持续学习场景。
摘要：
灾难性遗忘一直是深度神经网络在持续学习（CL）中的关键挑战，因为它在学习新任务时会破坏已积累的知识。参数高效的微调持续学习技术正在获得 traction，因其在轻量级训练计划下有效解决灾难性遗忘，同时避免在预训练模型中已积累知识的退化。然而，这些方法中的低秩适配器（LoRA）对秩选择高度敏感，可能导致资源分配不优和性能下降。为此，我们引入PEARL，一种无需回放的持续学习框架，其在持续学习训练中动态分配LoRA组件的秩。具体而言，PEARL利用参考任务权重，并根据参数空间中当前任务与参考任务权重的接近程度，自适应确定任务特定LoRA组件的秩。为了展示PEARL的通用性，我们评估了它在三种视觉架构（ResNet、分离卷积网络和视觉Transformer）和多种持续学习场景中的表现，并证明PEARL在所有考虑的基线中表现显著优于。
英文摘要：
Catastrophic forgetting has remained a critical challenge for deep neural networks in Continual Learning (CL) as it undermines consolidated knowledge when learning new tasks. Parameter efficient fine tuning CL techniques are gaining traction for their effectiveness in addressing catastrophic forgetting with a lightweight training schedule while avoiding degradation of consolidated knowledge in pre-trained models. However, low rank adapters (LoRA) in these approaches are highly sensitive to rank selection which can lead to sub-optimal resource allocation and performance. To this end, we introduce PEARL, a rehearsal-free CL framework that entails dynamic rank allocation for LoRA components during CL training. Specifically, PEARL leverages reference task weights and adaptively determines the rank of task-specific LoRA components based on the current tasks' proximity to reference task weights in parameter space. To demonstrate the versatility of PEARL, we evaluate it across three vision architectures (ResNet, Separable Convolutional Network and Vision Transformer) and a multitude of CL scenarios, and show that PEARL outperforms all considered baselines by a large margin.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
