arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.16642cs.LG
收藏

优化器选择对神经崩溃的出现至关重要

Optimizer choice matters for the emergence of Neural Collapse

University of Basel(巴塞尔大学);Warsaw University of Technology(华沙理工大学);IDEAS Research Institute(IDEAS研究院)

Jim Zhao, Tin Sum Cheng, Wojciech Masarczyk, Aurelien Lucchi

AI总结:

本文研究了优化器选择对神经崩溃(NC)出现的影响,提出NC0指标证明解耦权重衰减抑制NC,并展示了动量对NC的加速作用。

摘要:

神经崩溃(NC)指的是深度神经网络在训练后期表示中出现高度对称的几何结构。尽管其普遍存在,但对其理论理解仍然有限。现有分析大多忽略了优化器的作用,从而暗示NC在各种优化方法中普遍存在。在本文中,我们挑战这一假设,证明优化器的选择在NC的出现中起关键作用。该现象通常通过NC指标量化,但这些指标难以跟踪和理论分析。为克服这一限制,我们引入了新的诊断指标NC0,其收敛到零是NC出现的必要条件。使用NC0,我们提供了理论证据,证明在自适应优化器中使用解耦权重衰减(如AdamW实现)时,NC无法出现。具体而言,我们证明了SGD、带有耦合权重衰减的SignGD(Adam的特例)以及带有解耦权重衰减的SignGD(AdamW的特例)在NC0动态上存在本质不同。此外,我们还展示了在SGD训练中动量对NC的加速作用(超越训练损失收敛),这是在NC背景下关于动量的第一个结果。最后,我们进行了涵盖各种数据集、架构、优化器和超参数的大量实证实验,共进行了3900次训练运行,验证了我们的理论结果。本文为优化器依赖性NC的首次理论解释,并突显了权重衰减耦合在塑造优化器隐含偏置中的被忽视作用。

英文摘要:

Neural Collapse (NC) refers to the emergence of highly symmetric geometric structures in the representations of deep neural networks during the terminal phase of training. Despite its prevalence, the theoretical understanding of NC remains limited. Existing analyses largely ignore the role of the optimizer, thereby suggesting that NC is universal across optimization methods. In this work, we challenge this assumption and demonstrate that the choice of optimizer plays a critical role in the emergence of NC. The phenomenon is typically quantified through NC metrics, which, however, are difficult to track and analyze theoretically. To overcome this limitation, we introduce a novel diagnostic metric, NC0, whose convergence to zero is a necessary condition for NC. Using NC0, we provide theoretical evidence that NC cannot emerge under decoupled weight decay in adaptive optimizers, as implemented in AdamW. Concretely, we prove that SGD, SignGD with coupled weight decay (a special case of Adam), and SignGD with decoupled weight decay (a special case of AdamW) exhibit qualitatively different NC0 dynamics. Also, we show the accelerating effect of momentum on NC (beyond convergence of train loss) when trained with SGD, being the first result concerning momentum in the context of NC. Finally, we conduct extensive empirical experiments consisting of 3,900 training runs across various datasets, architectures, optimizers, and hyperparameters, confirming our theoretical results. This work provides the first theoretical explanation for optimizer-dependent emergence of NC and highlights the overlooked role of weight-decay coupling in shaping the implicit biases of optimizers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑