权重衰减提升语言模型可塑性
Weight Decay Improves Language Model Plasticity
AI总结：
本文通过系统实验表明，预训练中较大的权重衰减能提高模型的可塑性，使微调后下游性能更优，并揭示了其促进线性可分表示、正则化注意力矩阵和减少过拟合的机制。
摘要：
大型语言模型通常分两个主要阶段训练：预训练以产生基础模型，然后进一步训练以提高下游性能。然而，超参数优化和缩放定律主要从基础模型验证损失的角度研究，忽略了一个关键的模型属性：下游适应性。在这项工作中，我们从模型可塑性的角度研究预训练，即基础模型在额外训练后成功适应下游任务的能力。我们关注权重衰减的作用，这是预训练中的一个关键正则化参数，并通过系统实验表明，较大的权重衰减提高了预训练模型的可塑性，导致微调后下游性能提升更大。这种效应可能导致反直觉的权衡，即预训练后表现较差的基础模型在进一步训练后可能表现更好。对权重衰减对模型行为的机制影响的进一步研究表明，它鼓励线性可分的表示，正则化注意力矩阵，并减少对训练数据的过拟合。这些发现共同强调了预训练模型可塑性的重要性，使用交叉熵损失作为超参数优化的唯一指标的局限性，以及单个优化超参数在塑造模型行为中的多方面作用。
英文摘要：
Large language models are typically trained in two broad phases: pretraining to produce a base model, followed by further training to improve downstream performance. However, hyperparameter optimization and scaling laws are studied primarily from the perspective of the base model's validation loss, overlooking a crucial model property: downstream adaptability. In this work, we study pretraining from the perspective of model plasticity, that is, the ability of the base model to successfully adapt to downstream tasks upon additional training. We focus on the role of weight decay, a key regularization parameter during pretraining, and show through systematic experiments that larger weight decay increases the plasticity of the pretrained model, resulting in greater performance gains downstream after fine-tuning. This effect can lead to counterintuitive trade-offs where base models that perform worse after pretraining can perform better after further training. Further investigation of weight decay's mechanistic effects on model behavior reveals that it encourages linearly separable representations, regularizes attention matrices, and reduces overfitting on the training data. Together, these findings highlight the importance of pretrained model plasticity, the limits of using cross-entropy loss as the sole metric for hyperparameter optimization, and the multifaceted role that a single optimization hyperparameter plays in shaping model behavior.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
