A Comparative Empirical Study of Catastrophic Forgetting Mitigation in Sequential Task Adaptation for Continual Natural Language Processing Systems
Zaven P. and Sonia Akian College of Science and Engineering（泽文·皮和索尼亚·阿基安科学与工程学院）；American University of Armenia（亚美尼亚美国大学）；Yerevan（亚美尼亚共和国首都耶莱范）；Armenia（亚美尼亚）
部署在现实应用中的神经语言模型必须持续适应新任务和领域，同时不遗忘之前获得的知识。本文通过CLINC150数据集构建了一个10任务标签不重叠场景，评估了三种基础架构（前馈人工神经网络、门控循环单元和Transformer编码器）在多种连续学习策略下的表现。考虑了每种主要连续学习家族的代表性方法：基于重放的Maximally Interfered Retrieval（MIR）、基于正则化的Learning without Forgetting（LwF）以及通过Hard Attention to Task（HAT）进行参数隔离，分别单独和成对及三重组合。通过平均准确率、宏F1和反向迁移评估性能，捕捉任务序列中的稳定性-可塑性权衡。结果表明，朴素的顺序微调对所有架构都导致严重的遗忘，且没有单一的连续学习方法能完全防止。重放成为关键成分：MIR是单个最可靠的策略，包含重放的组合（MIR+HAT、MIR+LwF、MIR+LwF+HAT）在不同架构上均取得高最终性能，反向迁移接近零或轻微正向。最优配置依赖于架构。MIR+HAT对ANN和Transformer效果最佳，MIR+LwF+HAT对GRU效果最佳，某些情况下连续学习方法甚至超过联合训练，表明正则化效应。这些发现突显了在设计连续意图分类系统时联合选择基础架构和连续学习机制的重要性。
英文摘要：
Neural language models deployed in real-world applications must continually adapt to new tasks and domains without forgetting previously acquired knowledge. This work presents a comparative empirical study of catastrophic forgetting mitigation in continual intent classification. Using the CLINC150 dataset, we construct a 10-task label-disjoint scenario and evaluate three backbone architectures: a feed-forward Artificial Neural Network (ANN), a Gated Recurrent Unit (GRU), and a Transformer encoder, under a range of continual learning (CL) strategies. We consider one representative method from each major CL family: replay-based Maximally Interfered Retrieval (MIR), regularization-based Learning without Forgetting (LwF), and parameter-isolation via Hard Attention to Task (HAT), both individually and in all pairwise and triple combinations. Performance is assessed with average accuracy, macro F1, and backward transfer, capturing the stability-plasticity trade-off across the task sequence. Our results show that naive sequential fine-tuning suffers from severe forgetting for all architectures and that no single CL method fully prevents it. Replay emerges as a key ingredient: MIR is the most reliable individual strategy, and combinations that include replay (MIR+HAT, MIR+LwF, MIR+LwF+HAT) consistently achieve high final performance with near-zero or mildly positive backward transfer. The optimal configuration is architecture-dependent. MIR+HAT yields the best result for ANN and Transformer, MIR+LwF+HAT, on the other hand, works the best for GRU, and in several cases CL methods even surpass joint training, indicating a regularization effect. These findings highlight the importance of jointly selecting backbone architecture and CL mechanism when designing continual intent-classification systems.
Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives
Tsinghua University（清华大学）；Microsoft Research（微软研究院）；Georgia Institute of Technology（佐治亚理工学院）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Pattern Recognition Center, WeChat AI, Tencent Inc（腾讯公司微信AI模式识别中心）；School of Engineering, Westlake University（西湖大学工程学院）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖高等研究院先进技术研究所）
When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
MPI for Intelligent Systems, Tübingen（马克斯·普朗克智能系统研究所（蒂宾根））；Jinesis Lab, University of Toronto & Vector Institute（Jinesis实验室，多伦多大学与向量研究所）；EuroSafeAI；Hector Foundation（赫克托基金会）；Stanford University（斯坦福大学）；ELLIS Institute Tübingen（ELLIS研究所（蒂宾根））
FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning
The Hong Kong Polytechnic University（香港理工大学）；ShineMo Ltd., China（中国 ShineMo 公司）；Solar System of OVB, Tencent, China（腾讯 OVB 太阳系中国）；Peking University（北京大学）；University College London（伦敦大学学院）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）
