在微调中提高简单样本权重可缓解遗忘问题
Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting
AI总结:
针对无预训练数据时微调预训练模型易发生灾难性遗忘的问题,提出基于预训练损失提高简单样本权重的样本加权方案,从样本空间限制模型偏移,经理论与多模态实验验证可有效缓解遗忘。
摘要:
在下游任务上微调预训练模型通常会削弱其原有能力,这一现象被称为“灾难性遗忘”。当无法获取用于训练预训练模型的数据和流程时,该问题尤为突出。在这一约束下,大多数现有的缓解遗忘方法都无法适用。为应对这一挑战,我们提出了一种仅基于预训练模型损失的微调数据样本加权方案。具体而言,我们提高预训练模型在其上损失较低的简单样本的权重,反之则降低权重,以此限制模型偏离预训练状态。我们的方法与现有方法正交且互补:现有方法大多在参数或梯度空间上操作,而我们聚焦于样本空间。我们在线性设置下从理论上分析了使用我们的方法进行微调的影响,表明它会在某个子空间中停滞学习,从而抑制对目标任务的过拟合。我们在语言和视觉任务上通过实验验证了该方法的有效性。例如,在MetaMathQA数据集上微调Gemma 2 2B时,与标准微调相比,我们的方法在GSM8K(另一个数学数据集)上的准确率仅下降0.8%,同时在预训练数据集上多保留了5.4%的准确率。我们的代码已公开在https://github.com/sanyalsunny111/FLOW_finetuning 。
英文摘要:
Fine-tuning a pre-trained model on a downstream task often degrades its original capabilities, a phenomenon known as "catastrophic forgetting". This is especially an issue when one does not have access to the data and recipe used to develop the pre-trained model. Under this constraint, most existing methods for mitigating forgetting are inapplicable. To address this challenge, we propose a sample weighting scheme for the fine-tuning data solely based on the pre-trained model's losses. Specifically, we upweight the easy samples on which the pre-trained model's loss is low and vice versa to limit the drift from the pre-trained model. Our approach is orthogonal and yet complementary to existing methods; while such methods mostly operate on parameter or gradient space, we concentrate on the sample space. We theoretically analyze the impact of fine-tuning with our method in a linear setting, showing that it stalls learning in a certain subspace which inhibits overfitting to the target task. We empirically demonstrate the efficacy of our method on both language and vision tasks. As an example, when fine-tuning Gemma 2 2B on MetaMathQA, our method results in only a $0.8\%$ drop in accuracy on GSM8K (another math dataset) compared to standard fine-tuning, while preserving $5.4\%$ more accuracy on the pre-training datasets. Our code is publicly available at https://github.com/sanyalsunny111/FLOW_finetuning .
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。