重新思考权重衰减以实现基础模型的稳健微调
Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models
AI总结：
针对基础模型微调中探索性优化导致鲁棒性下降的问题，提出选择性投影衰减（SPD）权重衰减方法，有选择地正则化参数层，在多个视觉和语言基准上提升分布内泛化与分布外鲁棒性。
摘要：
现代优化器（如AdamW）配备动量和自适应学习率，旨在逃离局部最小值并探索广阔的参数空间。这种探索在从头训练时有利于找到良好的损失盆地，但在从强大的基础模型恢复训练时却未必理想，因为它可能导致与预训练初始化的较大偏差，进而损害鲁棒性和泛化能力。同时，对所有参数施加强正则化可能导致欠拟合。我们假设，有选择地正则化参数空间是拟合和保留预训练知识的关键。本文提出一种新的权重衰减技术——选择性投影衰减（Selective Projection Decay, SPD），它选择性地对某些层施加强惩罚，同时允许其他层自由变化。直观上，SPD分别对损失减少一致和不一致的层扩展和收缩参数搜索空间。实验表明，配备SPD后，Adam在多个流行的视觉和语言基准上持续提供更好的分布内泛化和分布外鲁棒性性能。代码可在\url{https://github.com/GT-RIPL/Selective-Projection-Decay.git}获取。
英文摘要：
Modern optimizers such as AdamW, equipped with momentum and adaptive learning rate, are designed to escape local minima and explore the vast parameter space. This exploration is beneficial for finding good loss basins when training from scratch. It is not necessarily ideal when resuming from a powerful foundation model because it can lead to large deviations from the pre-trained initialization and, consequently, worse robustness and generalization. At the same time, strong regularization on all parameters can lead to under-fitting. We hypothesize that selectively regularizing the parameter space is the key to fitting and retraining the pre-trained knowledge. This paper proposes a new weight decay technique, Selective Projection Decay (SPD), that selectively imposes a strong penalty on certain layers while allowing others to change freely. Intuitively, SPD expands and contracts the parameter search space for layers with consistent and inconsistent loss reduction, respectively. Experimentally, when equipped with SPD, Adam consistently provides better in-distribution generalization and out-of-distribution robustness performance on multiple popular vision and language benchmarks. Code available at~\url{https://github.com/GT-RIPL/Selective-Projection-Decay.git}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
