arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.08647cs.LG
收藏

生长,而非覆盖:无需遗忘的微调

Grow, Don't Overwrite: Fine-tuning Without Forgetting

University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Google Research(谷歌研究)

Dyah Adila, Hanna Mazzawi, Benoit Dherin, Xavier Gonzalvo

AI总结:

本文提出了一种功能保持扩展方法,通过扩展模型容量并保持原有能力,实现稳定训练与高效重用预训练知识的平衡。

摘要:

在适应预训练模型到专门任务时,往往会导致灾难性遗忘,即新知识会覆盖基础能力。现有方法要么在新任务上的性能有所妥协,要么在平衡训练稳定性与高效重用预训练知识之间存在困难。我们引入了一种新的功能保持扩展方法,解决了这一矛盾。我们的技术通过在transformer子模块内复制预训练参数,并应用一种缩放校正,确保扩展后的模型在初始化时与原始模型数学上完全一致,从而在稳定训练的同时利用现有知识。实证表明,我们的方法消除了塑性与稳定性之间的权衡,能够在下游任务上与完整微调表现相当,而不会降低模型原有能力。此外,我们展示了该方法的模块性,证明通过选择性地扩展一小部分层,可以在计算成本仅为完整微调分数之一的情况下实现相同性能。

英文摘要:

Adapting pre-trained models to specialized tasks often leads to catastrophic forgetting, where new knowledge overwrites foundational capabilities. Existing methods either compromise performance on the new task or struggle to balance training stability with efficient reuse of pre-trained knowledge. We introduce a novel function-preserving expansion method that resolves this dilemma. Our technique expands model capacity by replicating pre-trained parameters within transformer submodules and applying a scaling correction that guarantees the expanded model is mathematically identical to the original at initialization, enabling stable training while exploiting existing knowledge. Empirically, our method eliminates the trade-off between plasticity and stability, matching the performance of full fine-tuning on downstream tasks without any degradation of the model's original capabilities. Furthermore, we demonstrate the modularity of our approach, showing that by selectively expanding a small subset of layers we can achieve the same performance as full fine-tuning at a fraction of the computational cost.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑