arXiv 2602.175592026-02-19cs.LG

重新审视低秩持续学习中的权重正则化

Revisiting Weight Regularization for Low-Rank Continual Learning

State Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家重点实验室）；Institute of Artificial Intelligence and Robotics（人工智能与机器人研究院）；Computer Vision Center（计算机视觉中心）；Universitat Autònoma de Barcelona（巴塞罗那自治大学）；School of Instrument Science and Engineering（仪器科学与工程学院）；Bernoulli Institute（伯努利学院）；University of Groningen（格罗宁根大学）；School of Software Engineering（软件工程学院）

Yaoyue Zheng, Yin Zhang, Joost van de Weijer, Gido M van de Ven, Shaoyi Du, Xuetao Zhang, Zhiqiang Tian

备注：: Accepted by ICLR 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出EWC-LoRA方法，通过EWC正则化共享低秩更新来减轻PECL中的任务干扰，提供高效且稳定的持续学习解决方案。

摘要：

持续学习（CL）在大规模预训练模型（PTMs）中的应用最近引起了广泛关注，研究重点从从头开始训练转向持续适应PTMs。这催生了一个有前景的范式：参数高效持续学习（PECL），其中通常通过在训练过程中分配任务特定模块来减轻任务干扰，例如低秩适配器。然而，权重正则化技术，如弹性权重巩固（EWC）——持续学习中的关键策略——在这一新范式中仍被低估。在本文中，我们重新审视低秩持续学习中的权重正则化，作为减轻PECL中任务干扰的新视角。与现有低秩持续学习方法不同，我们通过EWC正则化共享低秩更新来减轻任务干扰，从而在任务数量不受限制的情况下保持存储需求和推理成本不变。我们提出的方法EWC-LoRA利用低秩表示来估计参数在全空间中的重要性。这种设计为PTMs的持续学习提供了实用的、计算和内存高效解决方案，并提供了可能影响PECL中正则化技术广泛应用的见解。在各种基准上的广泛实验表明了EWC-LoRA的有效性，实现了优于现有低秩持续学习方法的稳定性-可塑性权衡。这些结果表明，即使在低秩参数化下，权重正则化仍然是减轻任务干扰的有效机制。代码可在：https://github.com/yaoyz96/low-rank-cl获得。

英文摘要：

Continual Learning (CL) with large-scale pre-trained models (PTMs) has recently gained wide attention, shifting the focus from training from scratch to continually adapting PTMs. This has given rise to a promising paradigm: parameter-efficient continual learning (PECL), where task interference is typically mitigated by assigning a task-specific module during training, such as low-rank adapters. However, weight regularization techniques, such as Elastic Weight Consolidation (EWC)-a key strategy in CL-remain underexplored in this new paradigm. In this paper, we revisit weight regularization in low-rank CL as a new perspective for mitigating task interference in PECL. Unlike existing low-rank CL methods, we mitigate task interference by regularizing a shared low-rank update through EWC, thereby keeping the storage requirement and inference costs constant regardless of the number of tasks. Our proposed method EWC-LoRA leverages a low-rank representation to estimate parameter importance over the full-dimensional space. This design offers a practical, computational- and memory-efficient solution for CL with PTMs, and provides insights that may inform the broader application of regularization techniques within PECL. Extensive experiments on various benchmarks demonstrate the effectiveness of EWC-LoRA, achieving a stability-plasticity trade-off superior to existing low-rank CL approaches. These results indicate that, even under low-rank parameterizations, weight regularization remains an effective mechanism for mitigating task interference. Code is available at: https://github.com/yaoyz96/low-rank-cl.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。