谱遗忘:无需重新训练的后验能力恢复
Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining
AI总结:
研究探讨了语言模型在目标任务微调过程中因训练数据未显式威胁而退化的能力现象,提出了一种仅使用预训练检查点和微调后检查点的后验修复方法,通过谱修复技术恢复受损能力并保留目标任务收益。
摘要:
对语言模型进行目标任务微调通常会退化那些训练数据从未显式威胁的能力。我们研究这种现象,称为灾难性遗忘,并提出一种后验修复解决方案,仅使用预训练检查点W_base和其微调后代W_ft。目标不仅是将模型回退到基础检查点,而是恢复微调损坏的能力,同时保留目标任务的收益和任何有益的未显式改进。我们引入了DG-Hard,一种仅使用检查点的谱修复方法,用于微调更新Δ= W_ft - W_base。DG-Hard将Δ视为嵌入在IID-like噪声残差中的低秩任务对齐信号,该信号梯度下降没有动力去除,并对每个权重-增量矩阵应用Donoho-Gavish硬奇异值阈值,保留更新的结构高能部分并去除谱体。这将修复简化为一个闭合形式的SVD过滤步骤,无需数据依赖的调优。一个核心困难是评估:平均准确率隐藏了每个基准的失败,而朴素恢复分数奖励那些简单回退到基础的模型。因此,我们引入了一个分区条件度量,分别跟踪愈合、保留、非损坏和目标任务保留。在14(模型,任务)设置和九个跨领域未显式基准上,DG-Hard在后验基线中实现了最强的平衡修复。DG-Hard还恢复了由良性微调退化的三个独立安全轴的安全对齐,尽管不使用任何对齐数据。这些结果表明,部分微调引起的能力建设损失并非专业化不可避免的后果,而是在权重更新本身中可去除的谱残余。
英文摘要:
Fine-tuning a language model for a target task routinely degrades capabilities the training data never explicitly threatened. We study this phenomenon, known as catastrophic forgetting, and propose a post-hoc repair solution that uses only the pretrained checkpoint $W_{\mathrm{base}}$ and its fine-tuned descendant $W_{\mathrm{ft}}$. The goal is not merely to revert the model toward the base checkpoint, but to recover capabilities damaged by fine-tuning while preserving both the target-task gains and any beneficial held-out improvements. We introduce DG-Hard, a checkpoint-only spectral repair method for the fine-tuning update $Δ= W_{\mathrm{ft}} - W_{\mathrm{base}}$. DG-Hard treats $Δ$ as a low-rank task-aligned signal embedded in an IID-like noise residual that gradient descent has no incentive to remove, and applies the Donoho-Gavish hard singular-value threshold to each weight-delta matrix, keeping the structured high-energy part of the update and removing the spectral bulk. This reduces repair to a closed-form SVD filtering step requiring no data-dependent tuning. A central difficulty is evaluation: average accuracy hides per-benchmark failures, while naive recovery scores reward models that simply revert toward the base. We therefore introduce a partition-conditional metric that separately tracks healing, preservation, non-damage, and target-task retention. Across $14$ (model, task) settings and nine cross-domain held-out benchmarks, DG-Hard achieves the strongest balanced repair among post-hoc baselines. DG-Hard also restores safety alignment degraded by benign fine-tuning on three independent safety axes, despite using no alignment data. These results suggest that part of fine-tuning-induced capability loss is not an unavoidable consequence of specialization, but a removable spectral residue in the weight update itself.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。