Selective forgetting or removing information from deep neural networks (DNNs) is essential for continual learning and is challenging in controlling the DNNs. Such forgetting is crucial also in a practical sense since the deployed DNNs may be trained on the data with outliers, poisoned by attackers, or with leaked/sensitive information. In this paper, we formulate selective forgetting for classification tasks at a finer level than the samples' level. We specify the finer level based on four datasets distinguished by two conditions: whether they contain information to be forgotten and whether they are available for the forgetting procedure. Additionally, we reveal the need for such formulation with the datasets by showing concrete and practical situations. Moreover, we introduce the forgetting procedure as an optimization problem on three criteria; the forgetting, the correction, and the remembering term. Experimental results show that the proposed methods can make the model forget to use specific information for classification. Notably, in specific cases, our methods improved the model's accuracy on the datasets, which contains information to be forgotten but is unavailable in the forgetting procedure. Such data are unexpectedly found and misclassified in actual situations.
Enhancing Continual Learning for Software Vulnerability Prediction: Addressing Catastrophic Forgetting via Hybrid-Confidence-Aware Selective Replay for Temporal LLM Fine-Tuning
School of Computer Science, University of Nottingham, Nottingham, United Kingdom(诺丁汉大学计算机科学学院);School of Informatics and Computing, Northern Arizona University, Flagstaff, United States of America(北亚利桑那大学信息学院);School of Computer Science, University of Nottingham, Ningbo, China(诺丁汉大学宁波校区)
College of Electric and Information, Northeast Agricultural University(东北农业大学电气与信息学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Divide and not forget: Ensemble of selectively trained experts in Continual Learning
IDEAS-NCBR;Warsaw University of Technology(华沙工业大学);Gdańsk University of Technology(格但斯克工业大学);Tooploox;Jagiellonian University(雅盖隆大学);Computer Vision Center, Barcelona(巴塞罗那计算机视觉中心);Universitat Autònoma de Barcelona(巴塞罗那自治大学)
Accurate Forgetting for Heterogeneous Federated Continual Learning
Institute for Artificial Intelligence, Tsinghua University(清华大学人工智能研究院);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);The University of Auckland(奥克兰大学);RIKEN(理化学研究所);Hong Kong Baptist University(香港浸会大学);DataCanvas Technology Co., Ltd.(数帆科技有限公司);The University of Tokyo(东京大学)
Practical Continual Forgetting for Pre-trained Vision Models
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences(人工智能与机器人中心,香港科学创新研究院,中国科学院);SenseTime Research(商汤科技研究院)
SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language Models
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所基础模型研究中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Chongqing University(重庆大学);Peng Cheng Laboratory(鹏城实验室);Wuhan AI Research(武汉人工智能研究院)
Rethinking Quantum Continual Learning with Quantum Fisher Information
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院电气工程学院);Cross College Elite Program, National Cheng Kung University(成功学院精英计划,成功大学);National Center for High-Performance Computing , National Institutes of Applied Research (NIAR)(高性能计算中心,应用研究国立机构);Department of Electrophysics, National Yang Ming Chiao Tung University(电子物理系,交通大学)