Neural collapse is an emergent phenomenon in deep learning that was recently discovered by Papyan, Han and Donoho. We propose a simple "unconstrained features model" in which neural collapse also emerges empirically. By studying this model, we provide some explanation for the emergence of neural collapse in terms of the landscape of empirical risk.
The modern strategy for training deep neural networks for classification tasks includes optimizing the network's weights even after the training error vanishes to further push the training loss toward zero. Recently, a phenomenon termed "neural collapse" (NC) has been empirically observed in this training procedure. Specifically, it has been shown that the learned features (the output of the penultimate layer) of within-class samples converge to their mean, and the means of different classes exhibit a certain tight frame structure, which is also aligned with the last layer's weights. Recent pa…
Peking University（北京大学）；Stanford University（斯坦福大学）；University of Pennsylvania（宾夕法尼亚大学）；Harvard University（哈佛大学）
Neural collapse is a highly symmetric geometric pattern of neural networks that emerges during the terminal phase of training, with profound implications on the generalization performance and robustness of the trained networks. To understand how the last-layer features and classifiers exhibit this recently discovered implicit bias, in this paper, we introduce a surrogate model called the unconstrained layer-peeled model (ULPM). We prove that gradient flow on this model converges to critical points of a minimum-norm separation problem exhibiting neural collapse in its global minimizer. Moreover…
Microsoft Research（微软研究院）；Google Research（谷歌研究院）；University of Denver（丹佛大学）；University of Michigan（密歇根大学）
When training deep neural networks for classification tasks, an intriguing empirical phenomenon has been widely observed in the last-layer classifiers and features, where (i) the class means and the last-layer classifiers all collapse to the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling, and (ii) cross-example within-class variability of last-layer activations collapses to zero. This phenomenon is called Neural Collapse (NC), which seems to take place regardless of the choice of loss functions. In this work, we justify NC under the mean squared error (MSE) loss, where recen…
University of Denver（丹佛大学）；Johns Hopkins University（约翰霍普金斯大学）；New York University（纽约大学）；Google Research（谷歌研究院）；University of Michigan（密歇根大学）
We provide the first global optimization landscape analysis of $Neural\;Collapse$ -- an intriguing empirical phenomenon that arises in the last-layer classifiers and features of neural networks during the terminal phase of training. As recently reported by Papyan et al., this phenomenon implies that ($i$) the class means and the last-layer classifiers all collapse to the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling, and ($ii$) cross-example within-class variability of last-layer activations collapses to zero. We study the problem based on a simplified $unconstrained\;featu…
Monotonic neural networks have recently been proposed as a way to define invertible transformations. These transformations can be combined into powerful autoregressive flows that have been shown to be universal approximators of continuous probability distributions. Architectures that ensure monotonicity typically enforce constraints on weights and activation functions, which enables invertibility but leads to a cap on the expressiveness of the resulting transformations. In this work, we propose the Unconstrained Monotonic Neural Network (UMNN) architecture based on the insight that a function …
On the Robustness of Neural Collapse and the Neural Collapse of Robustness
Center for Data Science（数据科学中心）；New York University（纽约大学）；Statistical Laboratory（统计实验室）；University of Cambridge（剑桥大学）；Courant Institute of Mathematical Sciences（柯朗数学科学研究所）
Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data
Institute of Natural Sciences & School of Mathematical Sciences, Shanghai Jiao Tong University（上海交通大学自然科学研究院与数学科学学院）；Computer Science and Engineering, Ohio State University（俄亥俄州立大学计算机科学与工程系）；Electrical and Systems Engineering & Department of Radiology, University of Pennsylvania（宾夕法尼亚大学电气与系统工程系及放射学系）
University of Liège（列日大学）；Institut Polytechnique de Paris（巴黎理工学院）
The distributional reinforcement learning (RL) approach advocates for representing the complete probability distribution of the random return instead of only modelling its expectation. A distributional RL algorithm may be characterised by two main components, namely the representation of the distribution together with its parameterisation and the probability metric defining the loss. The present research work considers the unconstrained monotonic neural network (UMNN) architecture, a universal approximator of continuous monotonic functions which is particularly well suited for modelling differ…
We study the ability of foundation models to learn representations for classification that are transferable to new, unseen classes. Recent results in the literature show that representations learned by a single classifier over many classes are competitive on few-shot learning problems with representations learned by special-purpose algorithms designed for such problems. In this paper we provide an explanation for this behavior based on the recently observed phenomenon that the features learned by overparameterized classification networks show an interesting clustering property, called neural c…
Simon Fraser University（西蒙弗雷泽大学）；Digitalist Group Canada（加拿大Digitalist集团）；MetaOptima（MetaOptima公司）
We propose a simple modification to standard ResNet architectures--L2 normalization over feature space--that substantially improves out-of-distribution (OoD) performance on the previously proposed Deep Deterministic Uncertainty (DDU) benchmark. We show that this change also induces early Neural Collapse (NC), an effect linked to better OoD performance. Our method achieves comparable or superior OoD detection scores and classification accuracy in a small fraction of the training time of the benchmark. Additionally, it substantially improves worst case OoD performance over multiple, randomly ini…
Performance Analysis of Convolutional Neural Network By Applying Unconstrained Binary Quadratic Programming
Program in Computer Science Georg August University of Göttingen（计算机科学项目哥廷根大学）；Electronics and Computer Engineering Institute of Engineering (IOE)（电子与计算机工程学院（工程学院））；Georg August University of Göttingen（哥廷根大学）
Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Training
School of Computer and Communication Engineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院）；Tsinghua University（清华大学）；Shanghai AI Laboratory（上海人工智能实验室）
Reliable Neural Collapse Approximation for Open-World Test-Time Adaptation
Centre for Frontier AI Research, Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局前沿人工智能研究中心）；Sun Yat-sen University（中山大学）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；School of Mathematics (Zhuhai), Sun Yat-sen University（中山大学（珠海校区）数学学院）
We present a learning-based method for synthesizing novel views of complex scenes using only unstructured collections of in-the-wild photographs. We build on Neural Radiance Fields (NeRF), which uses the weights of a multilayer perceptron to model the density and color of a scene as a function of 3D coordinates. While NeRF works well on images of static subjects captured under controlled settings, it is incapable of modeling many ubiquitous, real-world phenomena in uncontrolled images, such as variable illumination or transient occluders. We introduce a series of extensions to NeRF to address …
Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking
Ting Han Lamarr Institute, TU Dortmund, Germany（廷汉拉马尔研究所，杜伊斯堡-艾森大学，德国）；Institute for AI in Medicine, UK Essen（人工智能医学研究所，英国埃森）；Research Center Trustworthy Data Science and Security of the University Alliance Ruhr, TU Dortmund, Germany（trustworthy数据科学与安全研究所以及鲁尔大学联盟，杜伊斯堡-艾森大学，德国）；Ruhr University Bochum, Germany（波恩鲁尔大学，德国）；Department of Physics, TU Dortmund, Germany（物理系，杜伊斯堡-艾森大学，德国）；Lamarr Institute, TU Dortmund, Germany（廷汉拉马尔研究所，杜伊斯堡-艾森大学，德国）
The Hong Kong University of Science and Technology（香港科技大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Although deep neural networks achieve tremendous success on various classification tasks, the generalization ability drops sheer when training datasets exhibit long-tailed distributions. One of the reasons is that the learned representations (i.e. features) from the imbalanced datasets are less effective than those from balanced datasets. Specifically, the learned representation under class-balanced distribution will present the Neural Collapse (NC) phenomena. NC indicates the features from the same category are close to each other and from different categories are maximally distant, showing a…
AInnovation Technology Co., Ltd.（创新奇智科技有限公司）；Nanjing University of Posts and Telecommunications（南京邮电大学）
Benefiting from the rapid development of convolutional neural networks, the performance of car license plate detection and recognition has been largely improved. Nonetheless, most existing methods solve detection and recognition problems separately, and focus on specific scenarios, which hinders the deployment for real-world applications. To overcome these challenges, we present an efficient and accurate framework to solve the license plate detection and recognition tasks simultaneously. It is a lightweight and unified deep neural network, that can be optimized end-to-end and work in real-time…
The recent work of Papyan, Han, & Donoho (2020) presented an intriguing "Neural Collapse" phenomenon, showing a structural property of interpolating classifiers in the late stage of training. This opened a rich area of exploration studying this phenomenon. Our motivation is to study the upper limits of this research program: How far will understanding Neural Collapse take us in understanding deep learning? First, we investigate its role in generalization. We refine the Neural Collapse conjecture into two separate conjectures: collapse on the train set (an optimization property) and collapse on…
Duke University（杜克大学）；University of Washington（华盛顿大学）
We consider the variational problem of cross-entropy loss with $n$ feature vectors on a unit hypersphere in $\mathbb{R}^d$. We prove that when $d \geq n - 1$, the global minimum is given by the simplex equiangular tight frame, which justifies the neural collapse behavior. We also prove that as $n \rightarrow \infty$ with fixed $d$, the minimizing points will distribute uniformly on the hypersphere and show a connection with the frame potential of Benedetto & Fickus.
Recent theoretical work has demonstrated that deep neural networks have superior performance over shallow networks, but their training is more difficult, e.g., they suffer from the vanishing gradient problem. This problem can be typically resolved by the rectified linear unit (ReLU) activation. However, here we show that even for such activation, deep and narrow neural networks (NNs) will converge to erroneous mean or median states of the target function depending on the loss with high probability. Deep and narrow NNs are encountered in solving partial differential equations with high-order de…
