The Role of the Time-Dependent Hessian in High-Dimensional Optimization
1 Laboratoire de Physique de l'Ecole Normale Sup\'erieure, ENS, Universit\'e PSL, CNRS, Sorbonne Universit\'e, Universit\'e Paris Cit\'e, F-75005 Paris, France.
Gradient descent is commonly used to find minima in rough landscapes, particularly in recent machine learning applications. However, a theoretical understanding of why good solutions are found remains elusive, especially in strongly non-convex and high-dimensional settings. Here, we focus on the phase retrieval problem as a typical example, which has received a lot of attention recently in theoretical machine learning. We analyze the Hessian during gradient descent, identify a dynamical transition in its spectral properties, and relate it to the ability of escaping rough regions in the loss landscape. When the signal-to-noise ratio (SNR) is large enough, an informative negative direction exists in the Hessian at the beginning of the descent, i.e in the initial condition. While descending, a BBP transition in the spectrum takes place in finite time: the direction is lost, and the dynamics is trapped in a rugged region filled with marginally stable bad minima. Surprisingly, for finite system sizes, this window of negative curvature allows the system to recover the signal well before the theoretical SNR found for infinite sizes, emphasizing the central role of initialization and early-time dynamics for efficiently navigating rough landscapes.
The Ohio State University（俄亥俄州立大学）；University of Michigan, Ann Arbor（密歇根大学安娜堡分校）
Bilevel optimization has arisen as a powerful tool in modern machine learning. However, due to the nested structure of bilevel optimization, even gradient-based methods require second-order derivative approximations via Jacobian- or/and Hessian-vector computations, which can be costly and unscalable in practice. Recently, Hessian-free bilevel schemes have been proposed to resolve this issue, where the general idea is to use zeroth- or first-order methods to approximate the full hypergradient of the bilevel problem. However, we empirically observe that such approximation can lead to large varia…
University of Oxford（牛津大学）；Google Research, Brain Team（谷歌研究院，大脑团队）；NYU（纽约大学）；FAIR（Facebook人工智能研究院）；BAIR（伯克利人工智能研究院）
Over the last decade, a single algorithm has changed many facets of our lives - Stochastic Gradient Descent (SGD). In the era of ever decreasing loss functions, SGD and its various offspring have become the go-to optimization tool in machine learning and are a key component of the success of deep neural networks (DNNs). While SGD is guaranteed to converge to a local optimum (under loose assumptions), in some cases it may matter which local optimum is found, and this is often context-dependent. Examples frequently arise in machine learning, from shape-versus-texture-features to ensemble methods…
AI Theory Lab（AI理论实验室）；Huawei（华为）；Machine Learning Research Group and Oxford-Man Institute for Quantitative Finance（机器学习研究组与牛津-曼量化金融研究所）；University of Oxford（牛津大学）
We study the effect of mini-batching on the loss landscape of deep neural networks using spiked, field-dependent random matrix theory. We demonstrate that the magnitude of the extremal values of the batch Hessian are larger than those of the empirical Hessian. We also derive similar results for the Generalised Gauss-Newton matrix approximation of the Hessian. As a consequence of our theorems we derive an analytical expressions for the maximal learning rates as a function of batch size, informing practical training regimens for both stochastic gradient descent (linear scaling) and adaptive algo…
Discretized Langevin diffusions are efficient Monte Carlo methods for sampling from high dimensional target densities that are log-Lipschitz-smooth and (strongly) log-concave. In particular, the Euclidean Langevin Monte Carlo sampling algorithm has received much attention lately, leading to a detailed understanding of its non-asymptotic convergence properties and of the role that smoothness and log-concavity play in the convergence rate. Distributions that do not possess these regularity properties can be addressed by considering a Riemannian Langevin diffusion with a metric capturing the loca…
Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localization
Financial Computing and Analytics Group, University College London（金融计算与分析组，伦敦大学学院）；Robot Perception and Learning Lab, University College London（机器人感知与学习实验室，伦敦大学学院）
On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvature
School of Physics, Peking University（北京大学物理系）；Peking University Chengdu Academy for Advanced Interdisciplinary Biotechnologies（北京大学成都先进交叉生物技术研究院）；Flatiron Institute（Flatiron研究所）
We propose a distributed, cubic-regularized Newton method for large-scale convex optimization over networks. The proposed method requires only local computations and communications and is suitable for federated learning applications over arbitrary network topologies. We show a $O(k^{{-}3})$ convergence rate when the cost function is convex with Lipschitz gradient and Hessian, with $k$ being the number of iterations. We further provide network-dependent bounds for the communication required in each step of the algorithm. We provide numerical experiments that validate our theoretical results.
Peking University（北京大学）；University of Pennsylvania（宾夕法尼亚大学）
The phenomenon that stochastic gradient descent (SGD) favors flat minima has played a critical role in understanding the implicit regularization of SGD. In this paper, we provide an explanation of this striking phenomenon by relating the particular noise structure of SGD to its \emph{linear stability} (Wu et al., 2018). Specifically, we consider training over-parameterized models with square loss. We prove that if a global minimum $θ^*$ is linearly stable for SGD, then it must satisfy $\|H(θ^*)\|_F\leq O(\sqrt{B}/η)$, where $\|H(θ^*)\|_F, B,η$ denote the Frobenius norm of Hessian at $θ^*$, bat…
Leiden Institute of Advanced Computer Science（莱顿高级计算机科学研究所）；Leiden University（莱顿大学）
The problem of approximating the Pareto front of a multiobjective optimization problem can be reformulated as the problem of finding a set that maximizes the hypervolume indicator. This paper establishes the analytical expression of the Hessian matrix of the mapping from a (fixed size) collection of $n$ points in the $d$-dimensional decision space (or $m$ dimensional objective space) to the scalar hypervolume indicator value. To define the Hessian matrix, the input set is vectorized, and the matrix is derived by analytical differentiation of the mapping from a vectorized set to the hypervolume…
Baylor College of Medicine（贝勒医学院）；Rice University（莱斯大学）
Understanding the learning dynamics and inductive bias of neural networks (NNs) is hindered by the opacity of the relationship between NN parameters and the function represented. We propose reparametrizing ReLU NNs as continuous piecewise linear splines. Using this spline lens, we study learning dynamics in shallow univariate ReLU NNs, finding unexpected insights and explanations for several perplexing phenomena. We develop a surprisingly simple and transparent view of the structure of the loss surface, including its critical and fixed points, Hessian, and Hessian spectrum. We also show that s…
Finite-Sample Approximation of Hessian-Guided Perturbed Wasserstein Gradient Flows
The University of Tokyo（东京大学）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；Nanyang Technological University（南洋理工大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能研究中心）
Improved Analysis for Hessian-free High-resolution Monte Carlo Sampling
School of Mathematics and Statistics, Donghua University（东华大学数学与统计学院）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；School of Mathematical Sciences, Fudan University（复旦大学数学科学学院）；Department of Mathematics, Florida State University（佛罗里达州立大学数学系）
Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin（德克萨斯大学奥斯汀分校计算工程与科学奥登研究所）
In this work we analyze the role nonlinear activation functions play at stationary points of dense neural network training problems. We consider a generic least squares loss function training formulation. We show that the nonlinear activation functions used in the network construction play a critical role in classifying stationary points of the loss landscape. We show that for shallow dense networks, the nonlinear activation function determines the Hessian nullspace in the vicinity of global minima (if they exist), and therefore determines the ill-posedness of the training problem. Furthermore…
This paper re-examines the concept of node equivalences like structural equivalence or automorphic equivalence, which have originally emerged in social network analysis to characterize the role an actor plays within a social system, but have since then been of independent interest for graph-based learning tasks. Traditionally, such exact node equivalences have been defined either in terms of the one hop neighborhood of a node, or in terms of the global graph structure. Here we formalize exact node roles with a scale-parameter, describing up to what distance the ego network of a node should be …
