AI 中文总结
研究探讨深度神经网络训练中被忽视的权重范数临界性，它由归一化与权重衰减相互作用引发，随权重衰减系数增加，尺度不变权重范数趋零致损失景观尖锐度剧增、损失尖峰出现，解释权重惩罚不能过强的原因并提供新理解及实证支持。
AI 中文摘要
大多数关于训练不稳定性的解释都集中在学习率临界性上，通常以稳定性边缘为特征，超过该边缘优化就会变得不稳定。我们认为，在实际的深度神经网络训练中，存在一种额外的且经常被忽视的权重范数临界性。这种临界性是由归一化（引入尺度不变分量）和权重衰减（持续缩小参数范数）之间的相互作用引起的。随着权重衰减系数增加，尺度不变权重的范数逐渐趋于零。同时，损失景观的尖锐度迅速增加，破坏了优化动态并导致突然的损失尖峰。这一观点解释了为什么权重惩罚可以提高泛化能力但不能任意增强：过度衰减会使尺度不变权重范数超过临界边界并破坏训练。我们的工作通过权重范数临界性对损失尖峰提供了新的机制理解。此外，权重范数临界性产生了可测试的预测，我们在具有尺度不变分量的网络中通过实验进行了验证，为所提出的机制提供了实证支持。
英文摘要
Most explanations of training instability focus on \emph{learning-rate criticality}, typically characterized by the Edge of Stability, beyond which optimization becomes unstable. We argue that, in practical deep neural network training, there is an additional and often overlooked \emph{weight-norm criticality}. This criticality is induced by the interaction between normalization (which introduces scale-invariant components) and weight decay (which persistently shrinks parameter norms). As the weight decay coefficient increases, the norms of scale-invariant weights are progressively driven toward zero. Meanwhile, the sharpness of the loss landscape increases rapidly, destabilizing the optimization dynamics and resulting in abrupt loss spikes. This perspective provides a rationale for why weight penalties can improve generalization yet cannot be made arbitrarily strong: excessive decay drives scale-invariant weight norms past a critical boundary and destabilizes training. Our work provides a new mechanistic understanding of loss spikes through the lens of \emph{weight-norm criticality}. Moreover, \emph{weight-norm criticality} yields testable predictions that we validate empirically in networks with scale-invariant components, providing empirical support for the proposed mechanism.
