A large body of theory and empirical work hypothesizes a connection between the flatness of a neural network's loss landscape during training and its performance. However, there have been conceptually opposite pieces of evidence regarding when SGD prefers flatter or sharper solutions during training. In this work, we partially but causally clarify the flatness-seeking behavior of SGD by identifying and exactly solving an analytically solvable model that exhibits both flattening and sharpening behavior during training. In this model, the SGD training has no \textit{a priori} preference for flatness, but only a preference for minimal gradient fluctuations. This leads to the insight that, at least within this model, it is data distribution that uniquely determines the sharpness at convergence, and that a flat minimum is preferred if and only if the noise in the labels is isotropic across all output dimensions. When the noise in the labels is anisotropic, the model instead prefers sharpness and can converge to an arbitrarily sharp solution, depending on the imbalance in the noise in the labels spectrum. We reproduce this key insight in controlled settings with different model architectures such as MLP, RNN, and transformers.
University of Maryland（马里兰大学）；New York University（纽约大学）
Sharpness-Aware Minimization (SAM) has recently emerged as a robust technique for improving the accuracy of deep neural networks. However, SAM incurs a high computational cost in practice, requiring up to twice as much computation as vanilla SGD. The computational challenge posed by SAM arises because each iteration requires both ascent and descent steps and thus double the gradient computations. To address this challenge, we propose to compute gradients in both stages of SAM on only the top-k samples with highest loss. K-SAM is simple and extremely easy-to-implement while providing significan…
University of Oxford（牛津大学）；Hertford college（赫特福德学院）
The intuition that local flatness of the loss landscape is correlated with better generalization for deep neural networks (DNNs) has been explored for decades, spawning many different flatness measures. Recently, this link with generalization has been called into question by a demonstration that many measures of flatness are vulnerable to parameter re-scaling which arbitrarily changes their value without changing neural network outputs. Here we show that, in addition, some popular variants of SGD such as Adam and Entropy-SGD, can also break the flatness-generalization correlation. As an altern…
It has been empirically observed that the flatness of minima obtained from training deep networks seems to correlate with better generalization. However, for deep networks with positively homogeneous activations, most measures of sharpness/flatness are not invariant to rescaling of the network parameters, corresponding to the same function. This means that the measure of flatness/sharpness can be made as small or as large as possible through rescaling, rendering the quantitative measures meaningless. In this paper we show that for deep networks with positively homogenous activations, these res…
IBM T. J. Watson Research Center（IBM托马斯·J·沃森研究中心）；Duke University（杜克大学）
Despite the tremendous success of Stochastic Gradient Descent (SGD) algorithm in deep learning, little is known about how SGD finds generalizable solutions in the high-dimensional weight space. By analyzing the learning dynamics and loss function landscape, we discover a robust inverse relation between the weight variance and the landscape flatness (inverse of curvature) for all SGD-based learning algorithms. To explain the inverse variance-flatness relation, we develop a random landscape theory, which shows that the SGD noise strength (effective temperature) depends inversely on the landscape…
In this paper, we study the sharpness of a deep learning (DL) loss landscape around local minima in order to reveal systematic mechanisms underlying the generalization abilities of DL models. Our analysis is performed across varying network and optimizer hyper-parameters, and involves a rich family of different sharpness measures. We compare these measures and show that the low-pass filter-based measure exhibits the highest correlation with the generalization abilities of DL models, has high robustness to both data and label noise, and furthermore can track the double descent behavior for neur…
Decentralized SGD and Average-direction SAM are Asymptotically Equivalent
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；JD Explore Academy, JD.com, Inc.（京东探索研究院，京东集团）；Artificial Intelligence and its Applications Institute, School of Informatics, University of Edinburgh（爱丁堡大学信息学院人工智能及其应用研究所）；The University of Sydney（悉尼大学）
University of California, Los Angeles（加州大学洛杉矶分校）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Washington（华盛顿大学）；Microsoft Research（微软研究院）
There is an increasing realization that algorithmic inductive biases are central in preventing overfitting; empirically, we often see a benign overfitting phenomenon in overparameterized settings for natural learning algorithms, such as stochastic gradient descent (SGD), where little to no explicit regularization has been employed. This work considers this issue in arguably the most basic setting: constant-stepsize SGD (with iterate averaging or tail averaging) for linear regression in the overparameterized regime. Our main result provides a sharp excess risk bound, stated in terms of the full…
AI Lab, Institute for Data Science and Analytics, Bocconi University（博科尼大学数据科学与分析研究所人工智能实验室）；Dept. Applied Science and Technology, Politecnico di Torino（都灵理工大学应用科学与技术系）
The properties of flat minima in the empirical risk landscape of neural networks have been debated for some time. Increasing evidence suggests they possess better generalization capabilities with respect to sharp ones. First, we discuss Gaussian mixture classification models and show analytically that there exist Bayes optimal pointwise estimators which correspond to minimizers belonging to wide flat regions. These estimators can be found by applying maximum flatness algorithms either directly on the classifier (which is norm independent) or on the differentiable loss function used in learning…
Jagiellonian University（雅盖隆大学）；Mila / University of Montreal（Mila / 蒙特利尔大学）；Facebook AI Research（Facebook人工智能研究院）；Ruhr University Bochum（波鸿鲁尔大学）；University of Edinburgh（爱丁堡大学）
Stochastic Gradient Descent (SGD) based training of neural networks with a large learning rate or a small batch-size typically ends in well-generalizing, flat regions of the weight space, as indicated by small eigenvalues of the Hessian of the training loss. However, the curvature along the SGD trajectory is poorly understood. An empirical investigation shows that initially SGD visits increasingly sharp regions, reaching a maximum sharpness determined by both the learning rate and the batch-size of SGD. When studying the SGD dynamics in relation to the sharpest directions in this initial phase…
Peking University（北京大学）；University of Pennsylvania（宾夕法尼亚大学）
The phenomenon that stochastic gradient descent (SGD) favors flat minima has played a critical role in understanding the implicit regularization of SGD. In this paper, we provide an explanation of this striking phenomenon by relating the particular noise structure of SGD to its \emph{linear stability} (Wu et al., 2018). Specifically, we consider training over-parameterized models with square loss. We prove that if a global minimum $θ^*$ is linearly stable for SGD, then it must satisfy $\|H(θ^*)\|_F\leq O(\sqrt{B}/η)$, where $\|H(θ^*)\|_F, B,η$ denote the Frobenius norm of Hessian at $θ^*$, bat…
Membership Privacy Risks of Sharpness Aware Minimization
Department of Computer Science（计算机科学系）；Purdue University（普渡大学）；Delft University of Technology（代尔夫特理工大学）；Department of Industrial and Systems Engineering（工业与系统工程系）；University of Southern California（南加州大学）
Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy
School of Mathematics and Statistics（数学与统计学院）；Xi’an Jiaotong University（西安交通大学）；Centrum Wiskunde & Informatica（国家数学和计算机科学研究院（ Centrum Wiskunde & Informatica））
针对 SAM 易陷入鞍点且有效学习率区间有限的问题，本文提出轻量重归一化方法 SSAM，扩展有效学习率区间并在多个数据集和任务上稳定提升性能。
The multiplicative structure of parameters and input data in the first layer of neural networks is explored to build connection between the landscape of the loss function with respect to parameters and the landscape of the model function with respect to input data. By this connection, it is shown that flat minima regularize the gradient of the model function, which explains the good generalization performance of flat minima. Then, we go beyond the flatness and consider high-order moments of the gradient noise, and show that Stochastic Gradient Descent (SGD) tends to impose constraints on these…
Hessian based measures of flatness, such as the trace, Frobenius and spectral norms, have been argued, used and shown to relate to generalisation. In this paper we demonstrate that for feed forward neural networks under the cross entropy loss, we would expect low loss solutions with large weights to have small Hessian based measures of flatness. This implies that solutions obtained using $L2$ regularisation should in principle be sharper than those without, despite generalising better. We show this to be true for logistic regression, multi-layer perceptrons, simple convolutional, pre-activated…
Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning
SGIT AI Lab, State Grid Corporation of China（国网SGIT人工智能实验室）；School of Management, Xi’an Jiaotong University（西安交通大学管理学院）；School of Software, Xi’an Jiaotong University（西安交通大学软件学院）
Tight Generalization Error Bounds for Stochastic Gradient Descent in Non-convex Learning
School of Mathematics and Statistics（数学与统计学学院）；Guangxi Normal University（广西师范大学）；School of Mathematics（数学学院）；Hohai University（河海大学）；College of Marine Life Sciences（海洋生命科学学院）；Ocean University of China（中国海洋大学）；State Key Laboratory of Mathematical Sciences（数学科学国家重点实验室）；Academy of Mathematics and Systems Science（数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）
Local SGD is a communication-efficient variant of SGD for large-scale training, where multiple GPUs perform SGD independently and average the model parameters periodically. It has been recently observed that Local SGD can not only achieve the design goal of reducing the communication overhead but also lead to higher test accuracy than the corresponding SGD baseline (Lin et al., 2020b), though the training regimes for this to happen are still in debate (Ortiz et al., 2021). This paper aims to understand why (and when) Local SGD generalizes better based on Stochastic Differential Equation (SDE) …
Peking University（北京大学）；Peking-Tsinghua Center for Life Science（北京大学生命科学联合中心）；Center for Quantitative Biology, Peking University（北京大学定量生物学中心）；School of Physics, Peking University（北京大学物理学院）；IBM T. J. Watson Research Center（IBM托马斯·J·沃森研究中心）
Understanding the implicit bias of Stochastic Gradient Descent (SGD) is one of the key challenges in deep learning, especially for overparametrized models, where the local minimizers of the loss function $L$ can form a manifold. Intuitively, with a sufficiently small learning rate $η$, SGD tracks Gradient Descent (GD) until it gets close to such manifold, where the gradient noise prevents further convergence. In such a regime, Blanc et al. (2020) proved that SGD with label noise locally decreases a regularizer-like term, the sharpness of loss, $\mathrm{tr}[\nabla^2 L]$. The current paper gives…
Sharpness-Aware Minimization Can Hallucinate Minimizers
Department of Electrical and Computer Engineering, Seoul National University（电子与计算机工程系，首尔国立大学）；Department of Mathematics, University of California, Los Angeles（数学系，加州大学洛杉矶分校）
UNIST InnoCORE AI-Space Solar Initiative, Ulsan National Institute of Science and Technology (UNIST), Ulsan, 44919, Republic of Korea（UNIST InnoCORE AI-Space Solar Initiative，乌山国立科学与技术研究所（UNIST），乌山，44919，韩国）；Artificial Intelligence Graduate School, Ulsan National Institute of Science and Technology (UNIST), Ulsan, 44919, Republic of Korea（人工智能研究生院，乌山国立科学与技术研究所（UNIST），乌山，44919，韩国）；Department of Industrial Engineering, Ulsan National Institute of Science and Technology (UNIST), Ulsan, 44919, Republic of Korea（工业工程系，乌山国立科学与技术研究所（UNIST），乌山，44919，韩国）；School of Mathematics, University of Edinburgh, Edinburgh, United Kingdom（爱丁堡大学数学学院，爱丁堡，英国）；Department of Mathematics, National Technical University of Athens, Athens, Greece（雅典国家技术大学数学系，雅典，希腊）；Archimedes, Athena Research and Innovation Centre, Marousi, Greece（Archimedes，雅典研究与创新中心，Marousi，希腊）
Centre for Frontier AI Research (CFAR), A*STAR（新加坡科技研究局前沿人工智能研究中心）；National University of Singapore（新加坡国立大学）；Institute of High Performance Computing (IHPC), A*STAR（新加坡科技研究局高性能计算研究所）
Overparametrized Deep Neural Networks (DNNs) often achieve astounding performances, but may potentially result in severe generalization error. Recently, the relation between the sharpness of the loss landscape and the generalization error has been established by Foret et al. (2020), in which the Sharpness Aware Minimizer (SAM) was proposed to mitigate the degradation of the generalization. Unfortunately, SAM s computational cost is roughly double that of base optimizers, such as Stochastic Gradient Descent (SGD). This paper thus proposes Efficient Sharpness Aware Minimizer (ESAM), which boosts…
Xiamen University（厦门大学）；JD Explore Academy（京东探索研究院）；The University of Sydney（悉尼大学）
Deep neural networks often suffer from poor generalization caused by complex and non-convex loss landscapes. One of the popular solutions is Sharpness-Aware Minimization (SAM), which smooths the loss landscape via minimizing the maximized change of training loss when adding a perturbation to the weight. However, we find the indiscriminate perturbation of SAM on all parameters is suboptimal, which also results in excessive computation, i.e., double the overhead of common optimizers like Stochastic Gradient Descent (SGD). In this paper, we propose an efficient and effective training scheme coine…
