Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal
School of Informatics, Xiamen University（厦门大学信息学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Tencent AI Lab（腾讯AI实验室）；Zhejiang University（浙江大学）
Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, Jinsong Su
Large language models (LLMs) suffer from catastrophic forgetting during continual learning. Conventional rehearsal-based methods rely on previous training data to retain the model's ability, which may not be feasible in real-world applications. When conducting continual learning based on a publicly-released LLM checkpoint, the availability of the original training data may be non-existent. To address this challenge, we propose a framework called Self-Synthesized Rehearsal (SSR) that uses the LLM to generate synthetic instances for rehearsal. Concretely, we first employ the base LLM for in-context learning to generate synthetic instances. Subsequently, we utilize the latest LLM to refine the instance outputs based on the synthetic inputs, preserving its acquired ability. Finally, we select diverse high-quality synthetic instances for rehearsal in future stages. Experimental results demonstrate that SSR achieves superior or comparable performance compared to conventional rehearsal-based approaches while being more data-efficient. Besides, SSR effectively preserves the generalization capabilities of LLMs in general domains.
An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Pattern Recognition Center, WeChat AI, Tencent Inc（腾讯公司微信AI模式识别中心）；School of Engineering, Westlake University（西湖大学工程学院）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖高等研究院先进技术研究所）
When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
MPI for Intelligent Systems, Tübingen（马克斯·普朗克智能系统研究所（蒂宾根））；Jinesis Lab, University of Toronto & Vector Institute（Jinesis实验室，多伦多大学与向量研究所）；EuroSafeAI；Hector Foundation（赫克托基金会）；Stanford University（斯坦福大学）；ELLIS Institute Tübingen（ELLIS研究所（蒂宾根））
Rehearsal-Free Modular and Compositional Continual Learning for Language Models
Bosch Center for Artificial Intelligence（博世人工智能中心）；LMU Munich（慕尼黑大学）；Hochschule der Medien（斯图加特媒体大学）；Karlsruhe University of Applied Sciences（卡尔斯鲁厄应用技术大学）
Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives
Tsinghua University（清华大学）；Microsoft Research（微软研究院）；Georgia Institute of Technology（佐治亚理工学院）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
