We consider the variational problem of cross-entropy loss with $n$ feature vectors on a unit hypersphere in $\mathbb{R}^d$. We prove that when $d \geq n - 1$, the global minimum is given by the simplex equiangular tight frame, which justifies the neural collapse behavior. We also prove that as $n \rightarrow \infty$ with fixed $d$, the minimizing points will distribute uniformly on the hypersphere and show a connection with the frame potential of Benedetto & Fickus.
HEALing Entropy Collapse: Enhancing Exploration in Few-Shot RLVR via Hybrid-Domain Entropy Dynamics Alignment
School of Informatics, Xiamen University（厦门大学信息学院）；Tsinghua University（清华大学）；Xiamen Unisound Intelligence Technology Co., Ltd（厦门Unisound智能科技有限公司）；Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China（福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室（厦门大学），中华人民共和国文化和旅游部，中国）
Building on the success of deep learning, Generative Adversarial Networks (GANs) provide a modern approach to learn a probability distribution from observed samples. GANs are often formulated as a zero-sum game between two sets of functions; the generator and the discriminator. Although GANs have shown great potentials in learning complex distributions such as images, they often suffer from the mode collapse issue where the generator fails to capture all existing modes of the input distribution. As a consequence, the diversity of generated samples is lower than that of the observed ones. To ta…
Force without transmission: a depth-induced rank collapse that no loss on the representation reopens
Technische Universität Ilmenau（伊尔默瑙工业大学）；German Centre for Integrative Biodiversity Research (iDiv) Halle–Jena–Leipzig（德国综合生物多样性研究中心（iDiv）哈勒-耶拿-莱比锡）；Friedrich Schiller University（弗里德里希·席勒大学）
Northwestern University（西北大学）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；Independent（独立研究者）；Imperial College London（伦敦帝国学院）；Oxford University（牛津大学）；University of Washington（华盛顿大学）；Microsoft（微软）；Stanford University（斯坦福大学）
Entropy Minimization without Model Collapse: Mitigating Prediction Bias in Medical Imaging
School of Computation, Information and Technology, Technical University of Munich, Germany（慕尼黑技术大学计算、信息与技术学院）；Institute of Machine Learning in Biomedical Imaging, Helmholtz Munich, Germany（生物医学成像中的机器学习研究所，海德堡慕尼黑德国）；School of Biomedical Engineering and Imaging Sciences, King’s College London, UK（伦敦国王学院生物医学工程与成像科学学院）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））；relAI – Konrad Zuse School of Excellence in Reliable AI（relAI——Konrad Zuse可靠性人工智能卓越学院）；TUM University Hospital Rechts der Isar（慕尼黑技术大学医院Rechts der Isar）
Rethinking Loss Reweighting for Imbalance Learning as an Inverse Problem: A Neural Collapse Point of View
CSMT, Wenzhou-Kean University（温州肯恩大学计算机科学与技术学院）；International Frontier Interdisciplinary Research Institute, Wenzhou-Kean University（温州肯恩大学国际前沿交叉研究 institutes）
SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training
College of Software, Nankai University（南开大学软件学院）；Zhongguancun Academy（中关村学院）；Beijing Institute of Technology（北京理工大学）；Zhejiang University（浙江大学）；Harbin Institute of Technology（哈尔滨工业大学）
Microsoft Research（微软研究院）；Google Research（谷歌研究院）；University of Denver（丹佛大学）；University of Michigan（密歇根大学）
When training deep neural networks for classification tasks, an intriguing empirical phenomenon has been widely observed in the last-layer classifiers and features, where (i) the class means and the last-layer classifiers all collapse to the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling, and (ii) cross-example within-class variability of last-layer activations collapses to zero. This phenomenon is called Neural Collapse (NC), which seems to take place regardless of the choice of loss functions. In this work, we justify NC under the mean squared error (MSE) loss, where recen…
Class imbalance distribution widely exists in real-world engineering. However, the mainstream optimization algorithms that seek to minimize error will trap the deep learning model in sub-optimums when facing extreme class imbalance. It seriously harms the classification precision, especially on the minor classes. The essential reason is that the gradients of the classifier weights are imbalanced among the components from different classes. In this paper, we propose Attraction-Repulsion-Balanced Loss (ARB-Loss) to balance the different components of the gradients. We perform experiments on the …
We present the Tamed Cross Entropy (TCE) loss function, a robust derivative of the standard Cross Entropy (CE) loss used in deep learning for classification tasks. However, unlike other robust losses, the TCE loss is designed to exhibit the same training properties than the CE loss in noiseless scenarios. Therefore, the TCE loss requires no modification on the training regime compared to the CE loss and, in consequence, can be applied in all applications where the CE loss is currently used. We evaluate the TCE loss using the ResNet architecture on four image datasets that we artificially conta…
Ohio State University（俄亥俄州立大学）；Google Research（谷歌研究院）；University of Michigan（密歇根大学）；New York University（纽约大学）
While cross entropy (CE) is the most commonly used loss to train deep neural networks for classification tasks, many alternative losses have been developed to obtain better empirical performance. Among them, which one is the best to use is still a mystery, because there seem to be multiple factors affecting the answer, such as properties of the dataset, the choice of network architecture, and so on. This paper studies the choice of loss function by examining the last-layer features of deep networks, drawing inspiration from a recent line work showing that the global optimal solution of CE and …
In this report, we review the calculation of entropy-regularised Wasserstein loss introduced by Cuturi and document a practical implementation in PyTorch. Code is available at https://github.com/t-vi/pytorch-tvmisc/blob/master/wasserstein-distance/Pytorch_Wasserstein.ipynb
Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Training
School of Computer and Communication Engineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院）；Tsinghua University（清华大学）；Shanghai AI Laboratory（上海人工智能实验室）
Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China（天津大学计算机科学与技术学院 TJUNLP 实验室，中国）；Independent Researcher（独立研究者）；College of Computer and Information Engineering, Tianjin Normal University, China（天津师范大学计算机与信息工程学院，中国）
The University of Tokyo（东京大学）；Physics & Informatics Laboratories, NTT Research, Inc.（NTT研究公司物理与信息学实验室）；Harvard University（哈佛大学）；University of Michigan（密歇根大学）；RIKEN Center for Emergent Matter Science (CEMS)（理化学研究所涌现物质科学中心）
Prevention of complete and dimensional collapse of representations has recently become a design principle for self-supervised learning (SSL). However, questions remain in our theoretical understanding: When do those collapses occur? What are the mechanisms and causes? We answer these questions by deriving and thoroughly analyzing an analytically tractable theory of SSL loss landscapes. In this theory, we identify the causes of the dimensional collapse and study the effect of normalization and bias. Finally, we leverage the interpretability afforded by the analytical theory to understand how di…
The cross entropy loss is widely used due to its effectiveness and solid theoretical grounding. However, as training progresses, the loss tends to focus on hard to classify samples, which may prevent the network from obtaining gains in performance. While most work in the field suggest ways to classify hard negatives, we suggest to strategically leave hard negatives behind, in order to focus on misclassified samples with higher probabilities. We show that adding to the optimization goal the expectation loss, which is a better approximation of the zero-one loss, helps the network to achieve bett…
Cornell University（康奈尔大学）；University of Toronto（多伦多大学）；Stanford University（斯坦福大学）
The recently discovered Neural Collapse (NC) phenomenon occurs pervasively in today's deep net training paradigm of driving cross-entropy (CE) loss towards zero. During NC, last-layer features collapse to their class-means, both classifiers and class-means collapse to the same Simplex Equiangular Tight Frame, and classifier behavior collapses to the nearest-class-mean decision rule. Recent works demonstrated that deep nets trained with mean squared error (MSE) loss perform comparably to those trained with CE. As a preliminary, we empirically establish that NC emerges in such MSE-trained deep n…
Cross-entropy loss is the standard metric used to train classification models in deep learning and gradient boosting. It is well-known that this loss function fails to account for similarities between the different values of the target. We propose a generalization of entropy called {\em structured entropy} which uses a random partition to incorporate the structure of the target variable in a manner which retains many theoretical properties of standard entropy. We show that a structured cross-entropy loss yields better results on several classification problems where the target variable has an …
Policy-based reinforcement learning methods suffer from the policy collapse problem. We find valued-based reinforcement learning methods with ε-greedy mechanism are capable of enjoying three characteristics, Closed-form Diversity, Objective-invariant Exploration and Adaptive Trade-off, which help value-based methods avoid the policy collapse problem. However, there does not exist a parallel mechanism for policy-based methods that achieves all three characteristics. In this paper, we propose an entropy regularization free mechanism that is designed for policy-based methods, which achieves Close…
Modern neural architectures for classification tasks are trained using the cross-entropy loss, which is widely believed to be empirically superior to the square loss. In this work we provide evidence indicating that this belief may not be well-founded. We explore several major neural architectures and a range of standard benchmark datasets for NLP, automatic speech recognition (ASR) and computer vision tasks to show that these architectures, with the same hyper-parameter settings as reported in the literature, perform comparably or better when trained with the square loss, even after equalizin…
University of Cambridge（剑桥大学）；University of Salerno（萨莱诺大学）
Automatic segmentation methods are an important advancement in medical image analysis. Machine learning techniques, and deep neural networks in particular, are the state-of-the-art for most medical image segmentation tasks. Issues with class imbalance pose a significant challenge in medical datasets, with lesions often occupying a considerably smaller volume relative to the background. Loss functions used in the training of deep learning algorithms differ in their robustness to class imbalance, with direct consequences for model convergence. The most commonly used loss functions for segmentati…
Swiss Federal Institute of Technology Lausanne (EPFL)（洛桑联邦理工学院）
There is no such thing as a perfect dataset. In some datasets, deep neural networks discover underlying heuristics that allow them to take shortcuts in the learning process, resulting in poor generalization capability. Instead of using standard cross-entropy, we explore whether a modulated version of cross-entropy called focal loss can constrain the model so as not to use heuristics and improve generalization performance. Our experiments in natural language inference show that focal loss has a regularizing impact on the learning process, increasing accuracy on out-of-distribution data, but sli…
Recently, substantial research efforts in Deep Metric Learning (DML) focused on designing complex pairwise-distance losses, which require convoluted schemes to ease optimization, such as sample mining or pair weighting. The standard cross-entropy loss for classification has been largely overlooked in DML. On the surface, the cross-entropy may seem unrelated and irrelevant to metric learning as it does not explicitly involve pairwise distances. However, we provide a theoretical analysis that links the cross-entropy to several well-known and recent pairwise losses. Our connections are drawn from…
Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective
State Key Laboratory of Blockchain and Data Security（区块链与数据安全国家重点实验室）；Zhejiang University（浙江大学）；Tencent（腾讯）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）
Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework
Bangladesh University of Engineering and Technology（孟加拉工程科技大学）；West Virginia University（西弗吉尼亚大学）；University of Aberdeen（阿伯丁大学）；Fogsphere (Redev.AI Ltd, UK)（Fogsphere（Redev.AI Ltd，英国））；University College London（伦敦大学学院）
We present several methods to improve the generalisation of language identification (LID) systems to new speakers and to new domains. These methods involve Spectral augmentation, where spectrograms are masked in the frequency or time bands during training and CNN architectures that are pre-trained on the Imagenet dataset. The paper also introduces the novel Triplet Entropy Loss training method, which involves training a network simultaneously using Cross Entropy and Triplet loss. It was found that all three methods improved the generalisation of the models, though not significantly. Even thoug…
