arXiv 2103.016492021-03cs.LGcs.CV

Learning with Hyperspherical Uniformity

University of Cambridge（剑桥大学）；MPI-IS Tübingen（图宾根马克斯·普朗克智能系统研究所）；Emory University（埃默里大学）；Université de Montréal（蒙特利尔大学）；Alan Turing Institute（艾伦·图灵研究所）

Weiyang Liu, Rongmei Lin, Zhen Liu, Li Xiong, Bernhard Schölkopf, Adrian Weller

备注：: AISTATS 2021 (v3: fixed typos)

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

Due to the over-parameterization nature, neural networks are a powerful tool for nonlinear function approximation. In order to achieve good generalization on unseen data, a suitable inductive bias is of great importance for neural networks. One of the most straightforward ways is to regularize the neural network with some additional objectives. L2 regularization serves as a standard regularization for neural networks. Despite its popularity, it essentially regularizes one dimension of the individual neuron, which is not strong enough to control the capacity of highly over-parameterized neural networks. Motivated by this, hyperspherical uniformity is proposed as a novel family of relational regularizations that impact the interaction among neurons. We consider several geometrically distinct ways to achieve hyperspherical uniformity. The effectiveness of hyperspherical uniformity is justified by theoretical insights and empirical evaluations.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

通过超球均匀性差距推广并解耦神经坍缩

Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap

Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Cambridge（剑桥大学）；Peking University（北京大学）；The Alan Turing Institute（艾伦·图灵研究所）

本文提出广义神经坍缩假设，并将其解耦为类内变异最小化与类间可分最大化，进而提出超球均匀性差距（HUG）作为替代损失函数，在泛化性和鲁棒性上表现良好。

2023-04-15cs.CVcs.LGstat.ML

Hyperspherically Regularized Networks for Self-Supervision

University of Aberdeen（阿伯丁大学）

Bootstrap Your Own Latent (BYOL) introduced an approach to self-supervised learning avoiding the contrastive paradigm and subsequently removing the computational burden of negative sampling associated with such methods. However, we empirically find that the image representations produced under the BYOL's self-distillation paradigm are poorly distributed in representation space compared to contrastive methods. This work empirically demonstrates that feature diversity enforced by contrastive losses is beneficial to image representation uniformity when employed in BYOL, and as such, provides grea…

2022-03-27cs.LG

Regularizing Neural Networks via Minimizing Hyperspherical Energy

Emory University（埃默里大学）；Georgia Institute of Technology（佐治亚理工学院）；Mila；Université de Montréal（蒙特利尔大学）；New York University（纽约大学）；NVIDIA（英伟达）

Inspired by the Thomson problem in physics where the distribution of multiple propelling electrons on a unit sphere can be modeled via minimizing some potential energy, hyperspherical energy minimization has demonstrated its potential in regularizing neural networks and improving their generalization power. In this paper, we first study the important role that hyperspherical energy plays in neural network training by analyzing its training dynamics. Then we show that naively minimizing hyperspherical energy suffers from some difficulties due to highly non-linear and non-convex optimization as …

2020-04-09cs.CVcs.LG

Eccentric Regularization: Minimizing Hyperspherical Energy without explicit projection

University of New South Wales（新南威尔士大学）

Several regularization methods have recently been introduced which force the latent activations of an autoencoder or deep neural network to conform to either a Gaussian or hyperspherical distribution, or to minimize the implicit rank of the distribution in latent space. In the present work, we introduce a novel regularizing loss function which simulates a pairwise repulsive force between items and an attractive force of each item toward the origin. We show that minimizing this loss function in isolation achieves a hyperspherical distribution. Moreover, when used as a regularizing term, the sca…

2022-07-01cs.AIcs.LG

Hyperspherical Consistency Regularization

Westlake University（西湖大学）；Westlake Institute for Advanced Study（西湖高等研究院）

Recent advances in contrastive learning have enlightened diverse applications across various semi-supervised fields. Jointly training supervised learning and unsupervised learning with a shared feature encoder becomes a common scheme. Though it benefits from taking advantage of both feature-dependent information from self-supervised learning and label-dependent information from supervised learning, this scheme remains suffering from bias of the classifier. In this work, we systematically explore the relationship between self-supervised learning and supervised learning, and study how self-super…

2022-06-02cs.AIcs.CVcs.LG

Learning towards Minimum Hyperspherical Energy

Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）；South China University of Technology（华南理工大学）；NVIDIA（英伟达）；Google Brain（谷歌大脑）；Ant Financial（蚂蚁金服）

Neural networks are a powerful class of nonlinear functions that can be trained end-to-end on various applications. While the over-parametrization nature in many neural networks renders the ability to fit complex functions and the strong representation power to handle challenging tasks, it also leads to highly correlated neurons that can hurt the generalization ability and incur unnecessary computation cost. As a result, how to regularize the network to avoid undesired representation redundancy becomes an important issue. To this end, we draw inspiration from a well-known problem in physics --…

2020-07-22cs.CVcs.LGstat.ML

更多相关论文收起更多论文

Improving singing voice separation with the Wave-U-Net using Minimum Hyperspherical Energy

City, University of London（伦敦城市大学）

In recent years, deep learning has surpassed traditional approaches to the problem of singing voice separation. The Wave-U-Net is a recent deep network architecture that operates directly on the time domain. The standard Wave-U-Net is trained with data augmentation and early stopping to prevent overfitting. Minimum hyperspherical energy (MHE) regularization has recently proven to increase generalization in image classification problems by encouraging a diversified filter configuration. In this work, we apply MHE regularization to the 1D filters of the Wave-U-Net. We evaluated this approach for…

2019-10-22cs.LGcs.SDeess.AS

UniVarFL：面向异构数据的均匀性与方差正则化联邦学习

UniVarFL: Uniformity and Variance Regularized Federated Learning for Heterogeneous Data

Koita Centre for Digital Health（Koita数字健康中心）；Indian Institute of Technology, Bombay（印度理工学院，孟买）；Department of Electrical Engineering（电气工程系）

针对 non-IID 数据导致的局部分类器偏差，本文提出无需全局模型依赖的 UniVarFL，通过分类器方差与超球面均匀性正则化提升泛化，并在多个基准上取得更高准确率。

2025-06-09cs.AIcs.CVcs.DC

Rethinking Prototypical Contrastive Learning through Alignment, Uniformity and Correlation

Carnegie Mellon University（卡内基梅隆大学）；Tohoku University（东北大学）；RIKEN（理化学研究所）

Contrastive self-supervised learning (CSL) with a prototypical regularization has been introduced in learning meaningful representations for downstream tasks that require strong semantic information. However, to optimize CSL with a loss that performs the prototypical regularization aggressively, e.g., the ProtoNCE loss, might cause the "coagulation" of examples in the embedding space. That is, the intra-prototype diversity of samples collapses to trivial solutions for their prototype being well-separated from others. Motivated by previous works, we propose to mitigate this phenomenon by learni…

2022-10-18cs.AIcs.CVcs.LG

T-REGS：基于最小生成树的自监督学习正则化

T-REGS: Minimum Spanning Tree Regularization for Self-Supervised Learning

Inria Saclay（Inria萨克雷实验室）；LIX, CNRS, École Polytechnique, IP Paris（LIX，CNRS，巴黎高等理工学院，IP巴黎）

T-REGS通过最小生成树长度正则化，有效缓解自监督学习中的维度崩溃和分布不均匀问题。

2026-02-06cs.CGcs.CVcs.LG

Uniform Priors for Data-Efficient Transfer

University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Heidelberg University（海德堡大学）；Mila（米拉研究所）；Google AI（谷歌人工智能）；Nvidia（英伟达）

Deep Neural Networks have shown great promise on a variety of downstream applications; but their ability to adapt and generalize to new data and tasks remains a challenge. However, the ability to perform few or zero-shot adaptation to novel tasks is important for the scalability and deployment of machine learning models. It is therefore crucial to understand what makes for good, transfer-able features in deep networks that best allow for such adaptation. In this paper, we shed light on this by showing that features that are most transferable have high uniformity in the embedding space and prop…

2020-10-13cs.CVcs.LGstat.ML

FedUV：面向异构联邦学习的均匀性与方差方法

FedUV: Uniformity and Variance for Heterogeneous Federated Learning

University of California, Davis（加利福尼亚大学戴维斯分校）；Hippo T&C（河马T&C公司）

针对联邦学习中数据异构导致的性能下降问题，提出基于分类器维度分布方差和编码器表征超球面均匀性的正则化方法FedUV，在高度非IID场景下性能提升显著且可扩展至更大模型与数据集。

2024-03-01cs.AIcs.DCcs.LG

PerNodeDrop：一种平衡专用子网络和正则化的深度神经网络方法

PerNodeDrop: A Method Balancing Specialized Subnets and Regularization in Deep Neural Networks

Department of Computer Science, Christ University（计算机科学系，基督大学）

PerNodeDrop通过在样本和节点层面引入随机扰动，平衡专用子网络与正则化，提升深度神经网络的泛化能力。

2025-12-14cs.AIcs.CVcs.LG

图神经网络中用于节点分类的分布信号

Distributional Signals for Node Classification in Graph Neural Networks

Nanyang Technological University（南洋理工大学）；Jiangnan University（江南大学）

该研究提出分布图信号概念及通用正则化方法，通过编码模型输出的分布平滑性和非均匀性，显著提升了GNN在半监督节点分类任务中的性能。

2023-04-07cs.LGeess.SP

Sparse Super-Regular Networks

Drexel University（德雷塞尔大学）

It has been argued by Thom and Palm that sparsely-connected neural networks (SCNs) show improved performance over fully-connected networks (FCNs). Super-regular networks (SRNs) are neural networks composed of a set of stacked sparse layers of (epsilon, delta)-super-regular pairs, and randomly permuted node order. Using the Blow-up Lemma, we prove that as a result of the individual super-regularity of each pair of layers, SRNs guarantee a number of properties that make them suitable replacements for FCNs for many tasks. These guarantees include edge uniformity across all large-enough subsets, m…

2022-01-04cs.CCcs.LGcs.NE