Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem
IDEAS NCBR；University of Warsaw（华沙大学）；Warsaw University of Technology（华沙理工大学）；Jagiellonian University（雅盖隆大学）；Google DeepMind（谷歌DeepMind）；Institute of Mathematics, Polish Academy of Sciences（波兰科学院数学研究所）
Maciej Wołczyk, Bartłomiej Cupiał, Mateusz Ostaszewski, Michał Bortkiewicz, Michał Zając, Razvan Pascanu, Łukasz Kuciński, Piotr Miłoś
Fine-tuning is a widespread technique that allows practitioners to transfer pre-trained capabilities, as recently showcased by the successful applications of foundation models. However, fine-tuning reinforcement learning (RL) models remains a challenge. This work conceptualizes one specific cause of poor transfer, accentuated in the RL setting by the interplay between actions and observations: forgetting of pre-trained capabilities. Namely, a model deteriorates on the state subspace of the downstream task not visited in the initial phase of fine-tuning, on which the model behaved well due to pre-training. This way, we lose the anticipated transfer benefits. We identify conditions when this problem occurs, showing that it is common and, in many cases, catastrophic. Through a detailed empirical analysis of the challenging NetHack and Montezuma's Revenge environments, we show that standard knowledge retention techniques mitigate the problem and thus allow us to take full advantage of the pre-trained capabilities. In particular, in NetHack, we achieve a new state-of-the-art for neural models, improving the previous best score from $5$K to over $10$K points in the Human Monk scenario.
Princeton University（普林斯顿大学）；The Hong Kong University of Science and Technology（香港科技大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；NVIDIA（英伟达）
MLFCIL: A Multi-Level Forgetting Mitigation Framework for Federated Class-Incremental Learning in LEO Satellites
Gannan University of Science and Technology（赣南科技学院）；Tsinghua University（清华大学）；Queen University（女王大学）；Lingnan University（岭南大学）；Central South University（中南大学）
MEGA: Second-Order Gradient Alignment for Catastrophic Forgetting Mitigation in GFSCIL
Beijing Institute of Technology（北京理工大学）；Department of Biomedical Engineering, Hefei University of Technology（合肥工业大学生物医学工程学院）；School of Artificial Intelligence and Computer Science, Nantong University（南通大学人工智能与计算机科学学院）；Faculty of Data Science, City University of Macau（澳门城市大学数据科学学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
Dynamic Proxy-Mixing: Transferring Replay Controllers from Small to Large Models for Continual Instruction Tuning
Department of Computer Science, Iowa State University（爱荷华州立大学计算机科学系）；Department of Computer Science, Kalinga Institute of Industrial Technology（卡林加工业技术学院计算机科学系）；Department of Civil, Construction & Environmental Engineering, Iowa State University（爱荷华州立大学土木、建筑与环境工程系）
Surgical Knowledge Rewrite in Compact LLMs: An 'Unlearn-then-Learn' Strategy with ($IA^3$) for Localized Factual Modulation and Catastrophic Forgetting Mitigation
Catastrophic Forgetting Mitigation via Discrepancy-Weighted Experience Replay
Institute of Software Chinese Academy of Science, Beijing, China（中国科学院软件研究所）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学）；Advanced Institute of Big Data, Beijing, China（大数据先进研究院）
Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide
Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital（人工智能在医疗与医学中的chair，技术大学慕尼黑（TUM）和慕尼黑技术大学医院）；Department of Orthopaedics and Sports Orthopaedics, TUM University Hospital（骨科与运动骨科，慕尼黑技术大学医院）；Munich Center for Machine Learning (MCML), Munich, Germany（慕尼黑机器学习中心（MCML））；Department of Computing, Imperial College London, UK（计算系，帝国理工学院伦敦）
