Understanding the properties of the parameter space in feed-forward ReLU networks is critical for effectively analyzing and guiding training dynamics. After initialization, training under gradient flow decisively restricts the parameter space to an algebraic variety that emerges from the homogeneous nature of the ReLU activation function. In this study, we examine two key challenges associated with feed-forward ReLU networks built on general directed acyclic graph (DAG) architectures: the (dis)connectedness of the parameter space and the existence of singularities within it. We extend previous results by providing a thorough characterization of connectedness, highlighting the roles of bottleneck nodes and balance conditions associated with specific subsets of the network. Our findings clearly demonstrate that singularities are intricately connected to the topology of the underlying DAG and its induced sub-networks. We discuss the reachability of these singularities and establish a principled connection with differentiable pruning. We validate our theory with simple numerical experiments.
Trier University of Applied Sciences（特里尔应用技术大学）；IEE S.A.（IEE股份有限公司）；University of Kaiserslautern（凯泽斯劳滕大学）
We show that for neural network functions that have width less or equal to the input dimension all connected components of decision regions are unbounded. The result holds for continuous and strictly monotonic activation functions as well as for the ReLU activation function. This complements recent results on approximation capabilities by [Hanin 2017 Approximating] and connectivity of decision regions by [Nguyen 2018 Neural] for such narrow neural networks. Our results are illustrated by means of numerical experiments.
Stanford University（斯坦福大学）；Purdue University（普渡大学）；Institute for Advanced Study（高等研究院）
We consider networks, trained via stochastic gradient descent to minimize $\ell_2$ loss, with the training labels perturbed by independent noise at each iteration. We characterize the behavior of the training dynamics near any parameter vector that achieves zero training error, in terms of an implicit regularization term corresponding to the sum over the data points, of the squared $\ell_2$ norm of the gradient of the model with respect to the parameter vector, evaluated at each data point. This holds for networks of any connectivity, width, depth, and choice of activation function. We interpr…
IBM Research AI（IBM研究院人工智能）；New York University（纽约大学）
A well-known line of work (Barron, 1993; Breiman, 1993; Klusowski & Barron, 2018) provides bounds on the width $n$ of a ReLU two-layer neural network needed to approximate a function $f$ over the ball $\mathcal{B}_R(\mathbb{R}^d)$ up to error $ε$, when the Fourier based quantity $C_f = \frac{1}{(2π)^{d/2}} \int_{\mathbb{R}^d} \|ξ\|^2 |\hat{f}(ξ)| \ dξ$ is finite. More recently Ongie et al. (2019) used the Radon transform as a tool for analysis of infinite-width ReLU two-layer networks. In particular, they introduce the concept of Radon-based $\mathcal{R}$-norms and show that a function defined…
Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity
Wharton School, University of Pennsylvania（宾夕法尼亚大学沃顿商学院）；Peking University（北京大学）；Stanford University（斯坦福大学）；School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）
Xiamen University（厦门大学）；Southern Illinois University Carbondale（南伊利诺伊大学卡本代尔分校）；Kneron Inc.（耐能智慧股份有限公司）；Yulin Normal University（玉林师范学院）
Since sparse neural networks usually contain many zero weights, these unnecessary network connections can potentially be eliminated without degrading network performance. Therefore, well-designed sparse neural networks have the potential to significantly reduce FLOPs and computational resources. In this work, we propose a new automatic pruning method - Sparse Connectivity Learning (SCL). Specifically, a weight is re-parameterized as an element-wise multiplication of a trainable weight variable and a binary mask. Thus, network connectivity is fully described by the binary mask, which is modulat…
Müntz-Szász Networks: Neural Architectures with Learnable Power-Law Bases
Axiom Research Group（Axiom研究组）；Department of Applied Mathematics and Computational Science, The Nelson Mandela African Institution of Science and Technology (NM-AIST)（应用数学与计算科学系，纳尔逊·曼德拉非洲科学技术研究所）；African Institute for Mathematical Sciences (AIMS), Research and Innovation Centre (RIC)（非洲数学科学研究所，研究与创新中心）
Rare Events, Real Signals: Functional Ensembles as Units of Computation in Deep Spiking Networks
University of Crete（希腊克里特大学）；Foundation of Research & Technology - Hellas（希腊研究与技术基金会）；Archimedes Research Unit（阿基米德研究单位）；Harvard Medical School（哈佛医学院）；Brigham and Women’s Hospital（布莱根妇女医院）
