CL-LoRA：用于无排练类增量学习的持续低秩适应
CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning
AI总结：
提出CL-LoRA双适配器架构，结合任务共享与任务特定适配器，利用随机正交矩阵和知识蒸馏保留共享知识，缓解任务间干扰，在多个基准上实现高效可扩展的无排练类增量学习。
摘要：
类增量学习（CIL）旨在按顺序学习新类的同时保留已学类的知识。近期，预训练模型（PTMs）结合参数高效微调（PEFT）在无排练CIL中展现出卓越性能，无需使用以往任务的样本。然而，现有的基于适配器的方法为CIL在PTMs中引入轻量级可学习模块，为每个新任务创建新适配器，导致参数冗余且无法利用任务间的共享知识。在本研究中，我们提出持续低秩适应（CL-LoRA），引入一种新颖的双适配器架构，结合任务共享适配器学习跨任务知识和任务特定适配器捕获每个新任务的独特特征。具体而言，共享适配器利用随机正交矩阵，并采用带梯度重新分配的知识蒸馏来保留关键的共享知识。此外，我们为任务特定适配器引入可学习的分块权重，在维持模型可塑性的同时缓解任务间干扰。我们证明CL-LoRA在多个基准下持续取得有前景的性能，且减少了训练和推理计算，为基于预训练模型的持续学习建立了更高效且可扩展的范式。
英文摘要：
Class-Incremental Learning (CIL) aims to learn new classes sequentially while retaining the knowledge of previously learned classes. Recently, pre-trained models (PTMs) combined with parameter-efficient fine-tuning (PEFT) have shown remarkable performance in rehearsal-free CIL without requiring exemplars from previous tasks. However, existing adapter-based methods, which incorporate lightweight learnable modules into PTMs for CIL, create new adapters for each new task, leading to both parameter redundancy and failure to leverage shared knowledge across tasks. In this work, we propose ContinuaL Low-Rank Adaptation (CL-LoRA), which introduces a novel dual-adapter architecture combining \textbf{task-shared adapters} to learn cross-task knowledge and \textbf{task-specific adapters} to capture unique features of each new task. Specifically, the shared adapters utilize random orthogonal matrices and leverage knowledge distillation with gradient reassignment to preserve essential shared knowledge. In addition, we introduce learnable block-wise weights for task-specific adapters, which mitigate inter-task interference while maintaining the model's plasticity. We demonstrate CL-LoRA consistently achieves promising performance under multiple benchmarks with reduced training and inference computation, establishing a more efficient and scalable paradigm for continual learning with pre-trained models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
