Continual Named Entity Recognition without Catastrophic Forgetting
Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Baidu Inc(百度公司);Hong Kong Baptist University(香港浸会大学);Australian Artificial Intelligence Institute, University of Technology Sydney(悉尼科技大学澳大利亚人工智能研究所)
Continual Named Entity Recognition (CNER) is a burgeoning area, which involves updating an existing model by incorporating new entity types sequentially. Nevertheless, continual learning approaches are often severely afflicted by catastrophic forgetting. This issue is intensified in CNER due to the consolidation of old entity types from previous steps into the non-entity type at each step, leading to what is known as the semantic shift problem of the non-entity type. In this paper, we introduce a pooled feature distillation loss that skillfully navigates the trade-off between retaining knowledge of old entity types and acquiring new ones, thereby more effectively mitigating the problem of catastrophic forgetting. Additionally, we develop a confidence-based pseudo-labeling for the non-entity type, \emph{i.e.,} predicting entity types using the old model to handle the semantic shift of the non-entity type. Following the pseudo-labeling process, we suggest an adaptive re-weighting type-balanced learning strategy to handle the issue of biased type distribution. We carried out comprehensive experiments on ten CNER settings using three different datasets. The results illustrate that our method significantly outperforms prior state-of-the-art approaches, registering an average improvement of $6.3$\% and $8.0$\% in Micro and Macro F1 scores, respectively.
Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives
Tsinghua University(清华大学);Microsoft Research(微软研究院);Georgia Institute of Technology(佐治亚理工学院);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心)
An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Pattern Recognition Center, WeChat AI, Tencent Inc(腾讯公司微信AI模式识别中心);School of Engineering, Westlake University(西湖大学工程学院);Institute of Advanced Technology, Westlake Institute for Advanced Study(西湖高等研究院先进技术研究所)
When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
MPI for Intelligent Systems, Tübingen(马克斯·普朗克智能系统研究所(蒂宾根));Jinesis Lab, University of Toronto & Vector Institute(Jinesis实验室,多伦多大学与向量研究所);EuroSafeAI;Hector Foundation(赫克托基金会);Stanford University(斯坦福大学);ELLIS Institute Tübingen(ELLIS研究所(蒂宾根))
FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning
The Hong Kong Polytechnic University(香港理工大学);ShineMo Ltd., China(中国 ShineMo 公司);Solar System of OVB, Tencent, China(腾讯 OVB 太阳系中国);Peking University(北京大学);University College London(伦敦大学学院);University of Illinois Chicago(伊利诺伊大学芝加哥分校)