Fine-tuning large language models on new data improves task performance but degrades capabilities learned during pretraining, a phenomenon known as catastrophic forgetting. Existing methods mitigate this by modifying the fine-tuning objective to suppress high-loss tokens or sequences, but these tokens are essential for learning new tasks, especially those with poor pretraining coverage. In such settings, hard tokens should still contribute to learning, so forgetting must be controlled without suppressing them. We identify a simple mechanism for doing so: per-step forgetting is bounded by the product of the learning rate and the square root of the current training loss. This suggests that high-loss batches are especially prone to inducing forgetting. Motivated by this observation, we introduce FINCH, a loss-adaptive learning-rate schedule that reduces the learning rate on high-loss batches and increases it as the model converges, while leaving the fine-tuning objective unchanged. Across knowledge acquisition, science, and low-resource language adaptation benchmarks, FINCH reduces forgetting by 93% on average while matching the task performance of standard fine-tuning. On Qwen3-4B knowledge acquisition, FINCH cuts TruthfulQA degradation by 5x and reverses HaluEval degradation, while better preserving confidence calibration. Overall, our results show that learning-rate schedules are an effective tool to shape model behavior during fine-tuning, beyond just target-task optimization.
On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning
Department of Electrical and Computer Engineering, The University of British Columbia（英国不列颠哥伦比亚大学电气与计算机工程系）；Department of Mathematics, The University of British Columbia（英国不列颠哥伦比亚大学数学系）；Department of Chemical and Biological Engineering, The University of British Columbia（英国不列颠哥伦比亚大学化学与生物工程系）
Forgetting: A New Mechanism Towards Better Large Language Model Fine-tuning
Department of Electrical and Computer Engineering, Isfahan University of Technology（伊斯法罕理工大学电气与计算机工程系）；RIKEN Center for Advanced Intelligence Project (AIP)（理化学研究所先进智能项目中心）；Australian Artificial Intelligence Institute, University of Technology Sydney（悉尼科技大学澳大利亚人工智能研究所）；School of Computer Science, Simon Fraser University（西蒙菲莎大学计算机科学学院）；Sydney AI Centre, The University of Sydney（悉尼大学悉尼人工智能中心）；Department of Computer Science, Hong Kong Baptist University（香港浸会大学计算机科学系）
Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem
IDEAS NCBR；University of Warsaw（华沙大学）；Warsaw University of Technology（华沙理工大学）；Jagiellonian University（雅盖隆大学）；Google DeepMind（谷歌DeepMind）；Institute of Mathematics, Polish Academy of Sciences（波兰科学院数学研究所）
MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning
The Chinese University of Hong Kong, Shenzhen, China（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Huawei Technologies Co.,Ltd, China（华为技术有限公司）；Shenzhen International Center for Industrial and Applied Mathematics（深圳国际工业与应用数学中心）
Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning
The University of Hong Kong（香港大学）；The Hong Kong University of Science and Technology（香港科学与技术大学）；Hong Kong Generative AI Research and Development Center（香港生成式人工智能研究与开发中心）
Measuring Catastrophic Forgetting in Cross-Lingual Transfer Paradigms: Exploring Tuning Strategies
Jožef Stefan Institute（约热夫· Stefan 研究所）；Jožef Stefan International Postgraduate School（约热夫· Stefan 国际研究生学院）；Faculty of Computer and Information Science, University of Ljubljana（卢布尔雅那大学计算机与信息科学学院）
本研究比较大语言模型在跨语言零样本与全样本设置下的 adapter/全参数微调及 IT、CLV 迁移策略，发现 IT 更利于目标语言，CLV 更能保留英语知识。
Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China（计算机科学与技术学院，中国科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China（苏州市先进研究 institute，中国科学技术大学）；School of Computer Science and Technology, China University of Mining and Technology（计算机科学与技术学院，中国矿业大学）；Mine Digitization Engineering Research Center of the Ministry of Education（教育部矿山数字化工程研究中心）
Effective Unbounded Drift Control in Heterogeneous Federated Learning: Bayesian Fine-Tuning Using a Reference Model
School of Computer Science and Engineering（计算机科学与工程学院）；Center for Biosystems and Biotech Data Science（生物系统与生物技术数据科学中心）；Ghent University Global Campus, Incheon, Republic of Korea（因斯布鲁克大学全球校区，韩国）
Mila - Quebec AI Institute（魁北克人工智能研究所）；Polytechnique Montréal（蒙特利尔理工学院）；Université de Montréal（蒙特利尔大学）；McGill University（麦吉尔大学）；CIFAR AI Chair（CIFAR人工智能主席）；Google DeepMind（谷歌DeepMind）
Early Data Exposure Improves Robustness to Subsequent Fine-Tuning
Department of Computer Science（计算机科学系）；Cranberry-Lemon University（Cranberry-Lemon 大学）；Department of Computational Neuroscience（计算神经科学系）；University of the Witwatersrand（沃茨沃斯兰德大学）
HKUST（香港科技大学）；University of Cambridge（剑桥大学）；Central South University（中南大学）；Shanghai Jiaotong University（上海交通大学）；SenseTime Research（商汤科技研究院）；Tsinghua University（清华大学）
Learning to Forget: Continual Learning with Adaptive Weight Decay
The Swiss AI Lab, IDSIA USI-SUPSI（瑞士人工智能实验室，IDSIA USI-SUPSI）；University of Alberta, Alberta Machine Intelligence Institute（阿尔伯塔大学，阿尔伯塔机器智能研究所）；Center of Excellence for Generative AI, KAUST（生成人工智能卓越中心，KAUST）
Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Division of Statistics and Machine Learning (STIMA), Department of Computer and Information Science (IDA), Linköping University（统计与机器学习系（STIMA）、计算机与信息科学系（IDA）、利厄普堡大学）
