Neural Collapse (NC) refers to the emergence of highly symmetric geometric structures in the representations of deep neural networks during the terminal phase of training. Despite its prevalence, the theoretical understanding of NC remains limited. Existing analyses largely ignore the role of the optimizer, thereby suggesting that NC is universal across optimization methods. In this work, we challenge this assumption and demonstrate that the choice of optimizer plays a critical role in the emergence of NC. The phenomenon is typically quantified through NC metrics, which, however, are difficult to track and analyze theoretically. To overcome this limitation, we introduce a novel diagnostic metric, NC0, whose convergence to zero is a necessary condition for NC. Using NC0, we provide theoretical evidence that NC cannot emerge under decoupled weight decay in adaptive optimizers, as implemented in AdamW. Concretely, we prove that SGD, SignGD with coupled weight decay (a special case of Adam), and SignGD with decoupled weight decay (a special case of AdamW) exhibit qualitatively different NC0 dynamics. Also, we show the accelerating effect of momentum on NC (beyond convergence of train loss) when trained with SGD, being the first result concerning momentum in the context of NC. Finally, we conduct extensive empirical experiments consisting of 3,900 training runs across various datasets, architectures, optimizers, and hyperparameters, confirming our theoretical results. This work provides the first theoretical explanation for optimizer-dependent emergence of NC and highlights the overlooked role of weight-decay coupling in shaping the implicit biases of optimizers.
Common-Mode Collapse and Recovery in Direct Feedback Alignment
Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University(哈佛大学肯普纳自然与人工智能研究所);Howard Hughes Medical Institute(霍华德·休斯医学研究所);Harvard Medical School(哈佛医学院)
On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capability
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);RIKEN AIP(理化学研究所人工智能项目);School of Software, Shandong University(山东大学软件学院);Dept. of Comp. Sci. & Tech., BNRist Center, THU-Bosch ML Center, Tsinghua University(清华大学计算机科学与技术系、BNRist中心、THU-Bosch机器学习中心)
Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data
Institute of Natural Sciences & School of Mathematical Sciences, Shanghai Jiao Tong University(上海交通大学自然科学研究院与数学科学学院);Computer Science and Engineering, Ohio State University(俄亥俄州立大学计算机科学与工程系);Electrical and Systems Engineering & Department of Radiology, University of Pennsylvania(宾夕法尼亚大学电气与系统工程系及放射学系)