发表机构
The University of Hong Kong; Hong Kong Generative AI Research and Development Center（香港大学; 香港生成式人工智能研发中心）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出稀疏驱动持续学习框架CLARE，通过两阶段稀疏掩码微调缓解任务间干扰，在Omnibenchmark-1k上显著提升最终准确率，实现可扩展的类增量学习。
AI 中文摘要
持续学习必须在学习新知识与保留先前学到的知识之间取得平衡，以便从数据流中增量学习任务，而不会发生灾难性遗忘。虽然利用预训练模型已显著推进了持续学习，但现有方法在按顺序训练许多任务时表现出可扩展性瓶颈，由于任务间干扰和可塑性丧失而遭受性能下降。受稀疏微调能达到与全量微调相当性能的证据启发，本文提出了一种新颖的稀疏驱动持续学习框架。我们的持续学习方法名为CLARE，分两个阶段运行：首先通过稀疏性诱导目标识别一个稀疏的、任务关键的参数掩码，然后仅优化掩码选中的参数进行掩码约束微调。这种两阶段稀疏适配器机制使得所有任务能够在共享适配器空间内累积，同时减少跨任务的破坏性干扰。大量实验证明了CLARE的可扩展性。在长任务序列基准Omnibenchmark-1k上，CLARE在最终准确率上大幅超越强基线，例如在学习100个任务后分别将EASE提升了4.64%和13.34%。
英文摘要
Continual learning must balance the learning of new knowledge with the retention of previously learned knowledge to incrementally learn tasks from a data stream without catastrophic forgetting. While leveraging pretrained models has significantly advanced continual learning, existing methods exhibit a scalability bottleneck when trained sequentially on many tasks, suffering from performance degradation due to inter-task interference and loss of plasticity. Inspired by evidence that sparse fine-tuning achieves performance comparable to full fine-tuning, this paper presents a novel sparsity-driven continual learning framework. Our continual learning method, termed CLARE, operates in two stages: it first identifies a sparse, task-critical parameter mask via a sparsity-inducing objective, then performs mask-constrained fine-tuning by only optimizing parameters selected by the mask. This two-stage sparse adapter mechanism enables all tasks to be accumulated within a shared adapter space while reducing destructive interference across tasks. Extensive experiments demonstrate the scalability of CLARE. On the long task-sequence benchmark Omnibenchmark-1k, CLARE outperforms strong baselines in final accuracy by a large margin, e.g, improving EASE by 4.64% and 13.34% after learning 100 tasks, respectively.
CommentsBMVC2026
