持续学习（CL）旨在从非平稳数据流中持续积累知识，同时避免灾难性遗忘已学知识，这需要在稳定性和适应性之间取得平衡。基于预训练模型（PTMs）的通用表示，PTM-based CL方法通过在冻结的PTMs上添加可学习的适配器或提示，对下游任务进行有效的持续适应。然而，许多现有的PTM-based CL方法在固定的模块集合上使用受限的适应以避免遗忘，导致CL能力受限。周期性地添加任务特定模块会导致模型线性增长和知识重用受损。我们提出了SEMA（Self-Expansion of pre-trained models with Modularized Adaptation），一种新颖的方法，以增强PTM-based CL中稳定性-塑性平衡的控制。SEMA根据在不同表示级别上是否检测到无法处理的显著分布偏移，自动决定在CL中重用或按需添加适配器模块。我们设计了模块化适配器，由功能适配器和表示描述符组成。表示描述符被训练为分布偏移指示器，并用于触发自扩展信号。为了更好地组合适配器，我们联合学习了一个可扩展的加权路由器，用于混合适配器输出。SEMA实现了更好的知识重用和次线性扩展率。大量实验证明了所提出的自扩展方法的有效性，在无记忆重放的PTM-based CL方法中达到了最先进的性能。代码可在https://github.com/huiyiwang01/SEMA-CL获取。
英文摘要：
Continual learning (CL) aims to continually accumulate knowledge from a non-stationary data stream without catastrophic forgetting of learned knowledge, requiring a balance between stability and adaptability. Relying on the generalizable representation in pre-trained models (PTMs), PTM-based CL methods perform effective continual adaptation on downstream tasks by adding learnable adapters or prompts upon the frozen PTMs. However, many existing PTM-based CL methods use restricted adaptation on a fixed set of these modules to avoid forgetting, suffering from limited CL ability. Periodically adding task-specific modules results in linear model growth rate and impaired knowledge reuse. We propose Self-Expansion of pre-trained models with Modularized Adaptation (SEMA), a novel approach to enhance the control of stability-plasticity balance in PTM-based CL. SEMA automatically decides to reuse or add adapter modules on demand in CL, depending on whether significant distribution shift that cannot be handled is detected at different representation levels. We design modular adapter consisting of a functional adapter and a representation descriptor. The representation descriptors are trained as a distribution shift indicator and used to trigger self-expansion signals. For better composing the adapters, an expandable weighting router is learned jointly for mixture of adapter outputs. SEMA enables better knowledge reuse and sub-linear expansion rate. Extensive experiments demonstrate the effectiveness of the proposed self-expansion method, achieving state-of-the-art performance compared to PTM-based CL methods without memory rehearsal. Code is available at https://github.com/huiyiwang01/SEMA-CL.
Adapter-Enhanced Semantic Prompting for Continual Learning
Beijing Key Laboratory of Multimedia and Intelligent Software Technology（北京多媒体与智能软件技术重点实验室）；Faculty of Information Technology, Beijing University of Technology（信息科技学院，北京理工大学）；University of California at Merced（加州大学默塞德分校）；Macquarie University（麦考瑞大学）
Continual Adapter Tuning with Semantic Shift Compensation for Class-Incremental Learning
HUST AI & Visual Learning Lab (HAIV Lab)（华中科技大学人工智能与视觉学习实验室）；Huazhong University of Science and Technology (HUST)（华中科技大学）；Department of Computer Science（计算机科学系）；Boston University（波士顿大学）；ByteDance, Inc.（字节跳动公司）
University of Oulu（奥卢大学）；Nanyang Technological University（南洋理工大学）；ByteDance（字节跳动）；National University of Defense Technology（国防科技大学）
The goal of Few-Shot Continual Learning (FSCL) is to incrementally learn novel tasks with limited labeled samples and preserve previous capabilities simultaneously, while current FSCL methods are all for the class-incremental purpose. Moreover, the evaluation of FSCL solutions is only the cumulative performance of all encountered tasks, but there is no work on exploring the domain generalization ability. Domain generalization is a challenging yet practical task that aims to generalize beyond training domains. In this paper, we set up a Generalized FSCL (GFSCL) protocol involving both class- an…
Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
College of Health Solutions, Arizona State University（亚利桑那州立大学健康解决方案学院）；Department of Electrical and Computer Engineering, The University of Texas at Austin（德克萨斯大学奥斯汀分校电气与计算机工程系）；School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）；The GAME School, Arizona State University（亚利桑那州立大学GAME学院）
ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation
National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；The Chinese University of Hong Kong（香港中文大学）；Hong Kong University of Science and Technology（香港科技大学）；School of Computer Science Peking University（北京大学计算机学院）；AIRobotics
CLARE: Continual Learning for Vision-Language-Action Models via Autonomous Adapter Routing and Expansion
Technical University of Munich（慕尼黑技术大学）；TUM School of Computation, Information and Technology（TUM计算、信息与技术学院）；Department of Computer Engineering, Learning Systems and Robotics Lab（计算机工程系、学习系统与机器人实验室）；Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与机器智能研究所（MIRMI））；Robotics Institute Germany（德国机器人研究所）；Munich Center for Machine Learning（慕尼黑机器学习中心）
In many real-world scenarios, data to train machine learning models become available over time. However, neural network models struggle to continually learn new concepts without forgetting what has been learnt in the past. This phenomenon is known as catastrophic forgetting and it is often difficult to prevent due to practical constraints, such as the amount of data that can be stored or the limited computation sources that can be used. Moreover, training large neural networks, such as Transformers, from scratch is very costly and requires a vast amount of training data, which might not be ava…
LLMs Can Evolve Continually on Modality for X-Modal Reasoning
Dalian University of Technology（大连理工大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology（香港科技大学）
Few-Shot Continual Learning for 3D Brain MRI with Frozen Foundation Models
Independent Researcher, USA（美国独立研究者）；Indiana University at Bloomington, United States（印第安纳大学布卢明顿分校）；Independent Researcher, Taiwan（台湾独立研究者）；National Taiwan University, Taiwan（台湾国立大学）；Boise State University, United States（博伊西州立大学）；National Yang Ming Chiao Tung University, Taiwan（台湾阳明交通大学）
C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models
Institute of Intelligent Information Processing, Shanxi University（山西大学智能信息处理研究所）；Alliance Manchester Business School, The University of Manchester（曼彻斯特大学联盟曼彻斯特商学院）
Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts
School of Computing and Data Science, The University of Hong Kong（计算与数据科学学院，香港大学）；Center for Embodied Artificial Intelligence and Computer Vision, Shenzhen Loop Area Institute（具身人工智能与计算机视觉中心，深圳环宇区研究所）
Continual Learning for Segment Anything Model Adaptation
Southern University of Science and Technology（南方科技大学）；City University of Hong Kong（香港城市大学）；Hong Kong University of Science and Technology（香港科技大学）；Tencent YouTu Lab（腾讯优图实验室）
针对动态流式数据场景，提出持续 SAM 适应基准 CoSAM 和域适配器混合算法 MoDA，通过全局特征与辅助令牌提取分离特征，实现持续分割并保持知识，优于现有方法。
