缓解大型语言模型灾难性遗忘的遗忘感知剪枝方法
Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning
AI总结：
提出遗忘感知剪枝度量（FAPM），通过任务向量与预训练参数比率量化灾难性遗忘并融入剪枝标准，在八个数据集上实现仅0.25%遗忘和99.67%下游准确率，无需修改训练或架构。
摘要：
近期大型语言模型（LLMs）的进展在各种下游任务中展示了令人印象深刻的能力，但在微调过程中通常会面临灾难性遗忘（CF）问题。在本文中，我们提出了遗忘感知剪枝度量（FAPM），一种新颖的基于剪枝的方法，以平衡CF和下游任务性能。我们的调查揭示，任务向量（即从下游任务微调权重中减去预训练权重）与预训练模型参数的重叠程度是CF的关键因素。基于这一发现，FAPM采用任务向量与预训练模型参数的比率作为量化CF的度量，并将该度量整合到剪枝标准中。重要的是，FAPM不需要修改训练过程或模型架构，也不需要任何辅助数据。我们在八个数据集上进行了广泛实验，涵盖自然语言推理、通用问答、医学问答、数学问答、阅读理解和小型测试。结果表明，FAPM将CF限制在仅0.25%，同时在下游任务上保持99.67%的准确率。我们提供了复现我们结果的代码。
英文摘要：
Recent advancements in large language models (LLMs) have shown impressive capabilities in various downstream tasks but typically face Catastrophic Forgetting (CF) during fine-tuning. In this paper, we propose the Forgetting-Aware Pruning Metric (FAPM), a novel pruning-based approach to balance CF and downstream task performance. Our investigation reveals that the degree to which task vectors (i.e., the subtraction of pre-trained weights from the weights fine-tuned on downstream tasks) overlap with pre-trained model parameters is a critical factor for CF. Based on this finding, FAPM employs the ratio of the task vector to pre-trained model parameters as a metric to quantify CF, integrating this measure into the pruning criteria. Importantly, FAPM does not necessitate modifications to the training process or model architecture, nor does it require any auxiliary data. We conducted extensive experiments across eight datasets, covering natural language inference, General Q&A, Medical Q&A, Math Q&A, reading comprehension, and cloze tests. The results demonstrate that FAPM limits CF to just 0.25\% while maintaining 99.67\% accuracy on downstream tasks. We provide the code to reproduce our results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
