Continual learning remains challenging due to catastrophic forgetting, where neural networks lose previously acquired knowledge when learning new tasks. Inspired by memory consolidation in neuroscience, we propose FSC-Net (Fast-Slow Consolidation Networks), a dual-network architecture that separates rapid task learning from gradual knowledge consolidation. Our method employs a fast network (NN1) for immediate adaptation to new tasks and a slow network (NN2) that consolidates knowledge through distillation and replay. Within the family of MLP-based NN1 variants we evaluated, consolidation effectiveness is driven more by methodology than architectural embellishments -- a simple MLP outperforms more complex similarity-gated variants by 1.2pp. Through systematic hyperparameter analysis, we observed empirically that pure replay without distillation during consolidation achieves superior performance, consistent with the hypothesis that distillation from the fast network introduces recency bias. On Split-MNIST (30 seeds), FSC-Net achieves 91.71% +/- 0.62% retention accuracy, a +4.27pp gain over the fast network alone (87.43% +/- 1.27%, paired t=23.585, p < 1e-10). On Split-CIFAR-10 (5 seeds), our method achieves 33.31% +/- 0.38% retention with an +8.20pp gain over the fast network alone (25.11% +/- 1.61%, paired t=9.75, p < 1e-3), demonstrating +8.20pp gain, though absolute performance (33.31%) remains modest and below random expectation, highlighting need for stronger backbones. Our results provide empirical evidence that the dual-timescale consolidation mechanism, rather than architectural complexity, is central to mitigating catastrophic forgetting in this setting.
In this report, we present a theoretical support of the continual learning method \textbf{Elastic Weight Consolidation}, introduced in paper titled `Overcoming catastrophic forgetting in neural networks'. Being one of the most cited paper in regularized methods for continual learning, this report disentangles the underlying concept of the proposed objective function. We assume that the reader is aware of the basic terminologies of continual learning.
Warsaw University of Technology（华沙理工大学）；Jagiellonian University（雅盖隆大学）；Tooploox
We propose a new method for unsupervised generative continual learning through realignment of Variational Autoencoder's latent space. Deep generative models suffer from catastrophic forgetting in the same way as other neural structures. Recent generative continual learning works approach this problem and try to learn from new data without forgetting previous knowledge. However, those methods usually focus on artificial scenarios where examples share almost no similarity between subsequent portions of data - an assumption not realistic in the real-life applications of continual learning. In thi…
Tsinghua University（清华大学）；Tsinghua-Peking Center for Life Sciences（清华-北大生命科学联合中心）
It is an important yet challenging setting to continually learn new tasks from a few examples. Although numerous efforts have been devoted to either continual learning or few-shot learning, little work has considered this new setting of few-shot continual learning (FSCL), which needs to minimize the catastrophic forgetting to the old tasks and gradually improve the ability of few-shot generalization. In this paper, we provide a first systematic study on FSCL and present an effective solution with deep neural networks. Our solution is based on the observation that continual learning of a task s…
Advanced Research Lab, NavInfo Europe（四维图新欧洲高级研究实验室）
Continual learning (CL) in the brain is facilitated by a complex set of mechanisms. This includes the interplay of multiple memory systems for consolidating information as posited by the complementary learning systems (CLS) theory and synaptic consolidation for protecting the acquired knowledge from erasure. Thus, we propose a general CL method that creates a synergy between SYNaptic consolidation and dual memory Experience Replay (SYNERgy). Our method maintains a semantic memory that accumulates and consolidates information across the tasks and interacts with the episodic memory for effective…
Semi-parametric Memory Consolidation: Towards Brain-like Deep Continual Learning
Centre for Artificial Intelligence and Robotics（人工智能与机器人中心）；Hong Kong Institute of Science and Innovation（香港科学与创新研究院）；Chinese Academy of Sciences（中国科学院）；Department of Computer Science（计算机科学系）；City University of Hong Kong（香港城市大学）；The Department of Neurosurgery（神经外科部门）；Peking University First Hospital（北京大学第一医院）；State Key Laboratory of Multimodal Artificial Intelligence Systems（多模态人工智能系统国家重点实验室）；Institute of Automation（自动化研究所）；Hong Kong SAR, China（香港特别行政区，中国）
University of Pisa（比萨大学）；Scuola Normale Superiore（比萨高等师范学校）；Computer Vision Center（计算机视觉中心）
Distributed learning on the edge often comprises self-centered devices (SCD) which learn local tasks independently and are unwilling to contribute to the performance of other SDCs. How do we achieve forward transfer at zero cost for the single SCDs? We formalize this problem as a Distributed Continual Learning scenario, where SCD adapt to local tasks and a CL model consolidates the knowledge from the resulting stream of models without looking at the SCD's private data. Unfortunately, current CL methods are not directly applicable to this scenario. We propose Data-Agnostic Consolidation (DAC), …
Continual learning (CL) over non-stationary data streams remains one of the long-standing challenges in deep neural networks (DNNs) as they are prone to catastrophic forgetting. CL models can benefit from self-supervised pre-training as it enables learning more generalizable task-agnostic features. However, the effect of self-supervised pre-training diminishes as the length of task sequences increases. Furthermore, the domain shift between pre-training data distribution and the task distribution reduces the generalizability of the learned representations. To address these limitations, we propo…
Self-paced Weight Consolidation for Continual Learning
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences（中国科学院机器人与智能制造创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）
CEL: A Continual Learning Model for Disease Outbreak Prediction by Leveraging Domain Adaptation via Elastic Weight Consolidation
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；University of Macau（澳门大学）
DESIRE: Dynamic Knowledge Consolidation for Rehearsal-Free Continual Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Centre for Artificial Intelligence and Robotics, HKISI-CAS（香港中文大学（深圳）人工智能与机器人中心（中国科学院深圳先进技术研究院））；Department of Computer Science, University of Illinois at Chicago（芝加哥大学计算机科学系）
Task-Core Memory Management and Consolidation for Long-term Continual Learning
School of Computer Science and Technology, East China Normal University（东华师范大学计算机科学与技术学院）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）；Computation and Artificial Intelligence Innovative College, Fudan University（复旦大学计算与人工智能创新学院）
Synaptic memory consolidation has been heralded as one of the key mechanisms for supporting continual learning in neuromorphic Artificial Intelligence (AI) systems. Here we report that a Fowler-Nordheim (FN) quantum-tunneling device can implement synaptic memory consolidation similar to what can be achieved by algorithmic consolidation models like the cascade and the elastic weight consolidation (EWC) models. The proposed FN-synapse not only stores the synaptic weight but also stores the synapse's historical usage statistic on the device itself. We also show that the operation of the FN-synaps…
Mitigating Task-Order Sensitivity and Forgetting via Hierarchical Second-Order Consolidation
Department of Computer Science and Engineering, University of South Carolina, SC, USA（南卡罗来纳大学计算机科学与工程系）；Argonne National Laboratory, Illinois, USA（阿贡国家实验室）
Elastic Weight Consolidation Done Right for Continual Learning
School of Artificial Intelligence, Sun Yat-sen University, China（中山大学人工智能学院，中国）；Key Laboratory of Intelligent Assessment Technology for Sustainable Tourism, Ministry of Culture and Tourism, Sun Yat-sen University（文化旅游可持续发展智能评估技术重点实验室，文化部，中山大学）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China（机器智能与高级计算重点实验室，教育部，中国）
Few-Shot Domain Incremental Learning via Continual Vision-Language Consolidation
School of Computer Science and IT, Adelaide University, Australia（阿德莱德大学计算机科学与信息技术学院，澳大利亚）；Faculty of Engineering and IT, University of Technology Sydney (UTS), Australia（悉尼大学工程与信息技术学院（UTS），澳大利亚）；School of Artificial Intelligence and Computer Science, Nantong University, China（南通大学人工智能与计算机科学学院，中国）；College of Computing and Data Science, Nanyang Technological University, Singapore（南洋理工大学计算与数据科学学院，新加坡）
University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Google（谷歌）
Continual learning in neural networks suffers from a phenomenon called catastrophic forgetting, in which a network quickly forgets what was learned in a previous task. The human brain, however, is able to continually learn new tasks and accumulate knowledge throughout life. Neuroscience findings suggest that continual learning success in the human brain is potentially associated with its modular structure and memory consolidation mechanisms. In this paper we propose a novel topological regularization that penalizes cycle structure in a neural network during training using principled theory fro…
Despite advances in deep learning, neural networks can only learn multiple tasks when trained on them jointly. When tasks arrive sequentially, they lose performance on previously learnt tasks. This phenomenon called catastrophic forgetting is a fundamental challenge to overcome before neural networks can learn continually from incoming data. In this work, we derive inspiration from human memory to develop an architecture capable of learning continuously from sequentially incoming tasks, while averting catastrophic forgetting. Specifically, our contributions are: (i) a dual memory architecture …
National Institute of Mental Health（美国国家精神卫生研究所）；Massachusetts Institute of Technology（麻省理工学院）
Collecting the large datasets needed to train deep neural networks can be very difficult, particularly for the many applications for which sharing and pooling data is complicated by practical, ethical, or legal concerns. However, it may be the case that derivative datasets or predictive models developed within individual sites can be shared and combined with fewer restrictions. Training on distributed data and combining the resulting networks is often viewed as continual learning, but these methods require networks to be trained sequentially. In this paper, we introduce distributed weight cons…
This work investigates continual learning of two segmentation tasks in brain MRI with neural networks. To explore in this context the capabilities of current methods for countering catastrophic forgetting of the first task when a new one is learned, we investigate elastic weight consolidation, a recently proposed method based on Fisher information, originally evaluated on reinforcement learning of Atari games. We use it to sequentially learn segmentation of normal brain structures and then segmentation of white matter lesions. Our findings show this recent method reduces catastrophic forgettin…
University of Guelph（圭尔夫大学）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）；Uber AI
Continual learning is the problem of sequentially learning new tasks or knowledge while protecting previously acquired knowledge. However, catastrophic forgetting poses a grand challenge for neural networks performing such learning process. Thus, neural networks that are deployed in the real world often struggle in scenarios where the data distribution is non-stationary (concept drift), imbalanced, or not always fully available, i.e., rare edge cases. We propose a Differentiable Hebbian Consolidation model which is composed of a Differentiable Hebbian Plasticity (DHP) Softmax layer that adds a…
ENSTA ParisTech（巴黎高等先进技术学院）；INRIA（法国国家信息与自动化研究所）；Thales（泰雷兹集团）；Fulda University of Applied Sciences（富尔达应用科学大学）
We present a new replay-based method of continual classification learning that we term "conditional replay" which generates samples and labels together by sampling from a distribution conditioned on the class. We compare conditional replay to another replay-based continual learning paradigm (which we term "marginal replay") that generates samples independently of their class and assigns labels in a separate step. The main improvement in conditional replay is that labels for generated samples need not be inferred, which reduces the margin for error in complex continual classification learning t…
Continual learning deals with training models on new tasks and datasets in an online fashion. One strand of research has used probabilistic regularization for continual learning, with two of the main approaches in this vein being Online Elastic Weight Consolidation (Online EWC) and Variational Continual Learning (VCL). VCL employs variational inference, which in other settings has been improved empirically by applying likelihood-tempering. We show that applying this modification to VCL recovers Online EWC as a limiting case, allowing for interpolation between the two approaches. We term the ge…
In this paper, we propose a continual learning (CL) technique that is beneficial to sequential task learners by improving their retained accuracy and reducing catastrophic forgetting. The principal target of our approach is the automatic extraction of modular parts of the neural network and then estimating the relatedness between the tasks given these modular components. This technique is applicable to different families of CL methods such as regularization-based (e.g., the Elastic Weight Consolidation) or the rehearsal-based (e.g., the Gradient Episodic Memory) approaches where episodic memor…
Continual Policy Consolidation for Lifelong Robot Learning
Department of XXX, University of YYY, Location, Country；School of ZZZ, Institute of WWW, Location, Country；Zhejiang University, Zhejiang, China；South University of Science；Hong Kong University, Hong Kong, China；INFIFORCE Intelligent Technology Co., Ltd., Hangzhou, China；The Hong Kong Polytechnic University, Hong Kong, China；School of Computer Science, Carnegie Mellon University
Natural language generation (NLG) is an essential component of task-oriented dialog systems. Despite the recent success of neural approaches for NLG, they are typically developed in an offline manner for particular domains. To better fit real-life applications where new data come in a stream, we study NLG in a "continual learning" setting to expand its knowledge to new domains or functionalities incrementally. The major challenge towards this goal is catastrophic forgetting, meaning that a continually trained model tends to forget the knowledge it has learned before. To this end, we propose a …
In recent years, the use of lithium-ion batteries has greatly expanded into products from many industrial sectors, e.g. cars, power tools or medical devices. An early prediction and robust understanding of battery faults could therefore greatly increase product quality in those fields. While current approaches for data-driven fault prediction provide good results on the exact processes they were trained on, they often lack the ability to flexibly adapt to changes, e.g. in operational or environmental parameters. Continual learning promises such flexibility, allowing for an automatic adaption o…
Online Continual Learning with Dynamic Label Hierarchies
College of Computer Science（计算机科学学院）；Technology, Nanjing University of Aeronautics（技术学院，南京航空航天大学）；MIIT Key Laboratory of Pattern Analysis（模式分析 MIIT 实验室）；Computer Vision Center, Spain（西班牙计算机视觉中心）；Computer Sciences Department, Universitat Autonoma de Barcelona, Spain（巴塞罗那自治大学计算机科学系，西班牙）；State Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；IDEAS Research Institute, Warsaw, Poland（华沙 IDEAS 研究院，波兰）；Joint Laboratory of Spatial intelligent Perception（空间智能感知联合实验室）
