Modern practice for training classification deepnets involves a Terminal Phase of Training (TPT), which begins at the epoch where training error first vanishes; During TPT, the training error stays effectively zero while training loss is pushed towards zero. Direct measurements of TPT, for three prototypical deepnet architectures and across seven canonical classification datasets, expose a pervasive inductive bias we call Neural Collapse, involving four deeply interconnected phenomena: (NC1) Cross-example within-class variability of last-layer training activations collapses to zero, as the individual activations themselves collapse to their class-means; (NC2) The class-means collapse to the vertices of a Simplex Equiangular Tight Frame (ETF); (NC3) Up to rescaling, the last-layer classifiers collapse to the class-means, or in other words to the Simplex ETF, i.e. to a self-dual configuration; (NC4) For a given activation, the classifier's decision collapses to simply choosing whichever class has the closest train class-mean, i.e. the Nearest Class Center (NCC) decision rule. The symmetric and very simple geometry induced by the TPT confers important benefits, including better generalization performance, better robustness, and better interpretability.
The Prevalence of Neural Collapse in Neural Multivariate Regression
New York University Abu Dhabi（纽约大学阿布扎克分校）；SFSC of AI and DL, NYU Shanghai（人工智能与深度学习学院（SFSC），纽约大学上海）；New York University Shanghai（纽约大学上海）；New York University（纽约大学）
Progressive Feedforward Collapse of ResNet Training
Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）
Rethinking Loss Reweighting for Imbalance Learning as an Inverse Problem: A Neural Collapse Point of View
CSMT, Wenzhou-Kean University（温州肯恩大学计算机科学与技术学院）；International Frontier Interdisciplinary Research Institute, Wenzhou-Kean University（温州肯恩大学国际前沿交叉研究 institutes）
All-around Neural Collapse for Imbalanced Classification
MIIT Key Laboratory of Pattern Analysis and Machine Intelligence（MIIT模式分析与智能关键实验室）；College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics (NUAA)（南京航空航天大学计算机科学与技术学院）
Understanding and Improving Transfer Learning of Deep Models via Neural Collapse
University of Michigan（密歇根大学）；Stanford University（斯坦福大学）；New York University（纽约大学）；Courant Institute of Mathematical Sciences, New York University（纽约大学柯朗数学科学研究所）；Ohio State University（俄亥俄州立大学）；Carnegie Mellon University（卡内基梅隆大学）
Low-Rank Learning by Design: the Role of Network Architecture and Activation Linearity in Gradient Rank Collapse
Translational Research Center for Neuroimaging and Data Science (TReNDs)（神经成像与数据科学转化研究中心（TReNDs））；Georgia State University（佐治亚州立大学）；Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）；Maynooth University（梅努斯大学）
Rethinking Continual Learning with Progressive Neural Collapse
Department of Computer Science, University of Houston, USA（德克萨斯大学休斯顿分校计算机科学系）；Department of Computer Science, University of North Carolina at Charlotte, USA（北卡罗来纳大学夏洛特分校计算机科学系）
Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）；Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS)（上海数学与交叉学科研究院）；Key Laboratory of Systems Health Science of Zhejiang Province, School of Life Science, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（浙江省系统健康科学重点实验室，中国科学院大学杭州高等研究院生命科学学院）
Rooted Absorbed Prefix Trajectory Balance with Submodular Replay for GFlowNet Training
Courant Institute School of Mathematics, Computing, and Data Science, New York University（纽约大学Courant研究所数学、计算与数据科学学院）；Courant Institute School of Mathematics, Computing（纽约大学Courant研究所数学、计算与数据科学学院）；Data Science, New York University（纽约大学数据科学学院）
