Modern deep learning models are over-parameterized, where different optima can result in widely varying generalization performance. The Sharpness-Aware Minimization (SAM) technique modifies the fundamental loss function that steers gradient descent methods toward flatter minima, which are believed to exhibit enhanced generalization prowess. Our study delves into a specific variant of SAM known as micro-batch SAM (mSAM). This variation involves aggregating updates derived from adversarial perturbations across multiple shards (micro-batches) of a mini-batch during training. We extend a recently developed and well-studied general framework for flatness analysis to theoretically show that SAM achieves flatter minima than SGD, and mSAM achieves even flatter minima than SAM. We provide a thorough empirical evaluation of various image classification and natural language processing tasks to substantiate this theoretical advancement. We also show that contrary to previous work, mSAM can be implemented in a flexible and parallelizable manner without significantly increasing computational costs. Our implementation of mSAM yields superior generalization performance across a wide range of tasks compared to SAM, further supporting our theoretical framework.
It has been empirically observed that the flatness of minima obtained from training deep networks seems to correlate with better generalization. However, for deep networks with positively homogeneous activations, most measures of sharpness/flatness are not invariant to rescaling of the network parameters, corresponding to the same function. This means that the measure of flatness/sharpness can be made as small or as large as possible through rescaling, rendering the quantitative measures meaningless. In this paper we show that for deep networks with positively homogenous activations, these res…
National University of Singapore（新加坡国立大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Recently, Sharpness-Aware Minimization (SAM), which connects the geometry of the loss landscape and generalization, has demonstrated significant performance boosts on training large-scale models such as vision transformers. However, the update rule of SAM requires two sequential (non-parallelizable) gradient computations at each step, which can double the computational overhead. In this paper, we propose a novel algorithm LookSAM - that only periodically calculates the inner gradient ascent, to significantly reduce the additional training cost of SAM. The empirical results illustrate that Look…
Recent findings (e.g., arXiv:2103.00065) demonstrate that modern neural networks trained by full-batch gradient descent typically enter a regime called Edge of Stability (EOS). In this regime, the sharpness, i.e., the maximum Hessian eigenvalue, first increases to the value 2/(step size) (the progressive sharpening phase) and then oscillates around this value (the EOS phase). This paper aims to analyze the GD dynamics and the sharpness along the optimization trajectory. Our analysis naturally divides the GD trajectory into four phases depending on the change of the sharpness. We empirically id…
The University of Tokyo（东京大学）；RIKEN Center for AIP（理化学研究所人工智能中心）
Stochastic Gradient Descent (SGD) and its variants are mainstream methods for training deep networks in practice. SGD is known to find a flat minimum that often generalizes well. However, it is mathematically unclear how deep learning can select a flat minimum among so many minima. To answer the question quantitatively, we develop a density diffusion theory (DDT) to reveal how minima selection quantitatively depends on the minima sharpness and the hyperparameters. To the best of our knowledge, we are the first to theoretically and empirically prove that, benefited from the Hessian-dependent co…
University of Pittsburgh（匹兹堡大学）；University of Nevada – Reno（内华达大学雷诺分校）；HP Labs（惠普实验室）
In Deep Learning, Stochastic Gradient Descent (SGD) is usually selected as a training method because of its efficiency; however, recently, a problem in SGD gains research interest: sharp minima in Deep Neural Networks (DNNs) have poor generalization; especially, large-batch SGD tends to converge to sharp minima. It becomes an open question whether escaping sharp minima can improve the generalization. To answer this question, we propose SmoothOut framework to smooth out sharp minima in DNNs and thereby improve generalization. In a nutshell, SmoothOut perturbs multiple copies of the DNN by noise…
Jagiellonian University（雅盖隆大学）；Mila / University of Montreal（Mila / 蒙特利尔大学）；Facebook AI Research（Facebook人工智能研究院）；Ruhr University Bochum（波鸿鲁尔大学）；University of Edinburgh（爱丁堡大学）
Stochastic Gradient Descent (SGD) based training of neural networks with a large learning rate or a small batch-size typically ends in well-generalizing, flat regions of the weight space, as indicated by small eigenvalues of the Hessian of the training loss. However, the curvature along the SGD trajectory is poorly understood. An empirical investigation shows that initially SGD visits increasingly sharp regions, reaching a maximum sharpness determined by both the learning rate and the batch-size of SGD. When studying the SGD dynamics in relation to the sharpest directions in this initial phase…
Stability Analysis of Sharpness-Aware Minimization
Chung-Ang University, South Korea（Chung-Ang 大学，韩国）；Korea Institute for Advanced Study, South Korea（韩国高级研究院）；Ulsan National Institute of Science（乌山国家科学研究院）；Nanyang Technological University (NTU), Singapore（南洋理工大学（NTU），新加坡）；Seoul National University, South Korea（首尔国立大学，韩国）
University of Science and Technology of China（中国科学技术大学）；Wuhan University（武汉大学）；JD Explore Academy（京东探索研究院）
Sharpness aware minimization (SAM) optimizer has been extensively explored as it can generalize better for training deep neural networks via introducing extra perturbation steps to flatten the landscape of deep learning models. Integrating SAM with adaptive learning rate and momentum acceleration, dubbed AdaSAM, has already been explored empirically to train large-scale deep neural networks without theoretical guarantee due to the triple difficulties in analyzing the coupled perturbation step, adaptive learning rate and momentum step. In this paper, we try to analyze the convergence rate of Ad…
Decentralized SGD and Average-direction SAM are Asymptotically Equivalent
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；JD Explore Academy, JD.com, Inc.（京东探索研究院，京东集团）；Artificial Intelligence and its Applications Institute, School of Informatics, University of Edinburgh（爱丁堡大学信息学院人工智能及其应用研究所）；The University of Sydney（悉尼大学）
Normalization methods play an important role in enhancing the performance of deep learning while their theoretical understandings have been limited. To theoretically elucidate the effectiveness of normalization, we quantify the geometry of the parameter space determined by the Fisher information matrix (FIM), which also corresponds to the local shape of the loss landscape under certain conditions. We analyze deep neural networks with random initialization, which is known to suffer from a pathologically sharp shape of the landscape when the network becomes sufficiently wide. We reveal that batc…
Peking University（北京大学）；University of Pennsylvania（宾夕法尼亚大学）
The phenomenon that stochastic gradient descent (SGD) favors flat minima has played a critical role in understanding the implicit regularization of SGD. In this paper, we provide an explanation of this striking phenomenon by relating the particular noise structure of SGD to its \emph{linear stability} (Wu et al., 2018). Specifically, we consider training over-parameterized models with square loss. We prove that if a global minimum $θ^*$ is linearly stable for SGD, then it must satisfy $\|H(θ^*)\|_F\leq O(\sqrt{B}/η)$, where $\|H(θ^*)\|_F, B,η$ denote the Frobenius norm of Hessian at $θ^*$, bat…
University of Basel（巴塞尔大学）；ETH Zürich（苏黎世联邦理工学院）；University of Oslo（奥斯陆大学）；Inria（法国国家信息与自动化研究所）；Ecole Normale Supérieure PSL Research University（巴黎高等师范学院-PSL研究大学）
本文为 SAM 及其两种变体在全批量与小批量设置下建立 SDE 连续时间模型，解释其偏好平坦极小值的机制，并证明其在现实条件下会被鞍点吸引。
Sharpness-Aware Minimization (SAM) is a highly effective regularization technique for improving the generalization of deep neural networks for various settings. However, the underlying working of SAM remains elusive because of various intriguing approximations in the theoretical characterizations. SAM intends to penalize a notion of sharpness of the model but implements a computationally efficient variant; moreover, a third notion of sharpness was used for proving generalization guarantees. The subtle differences in these notions of sharpness can indeed lead to significantly different empirica…
Bias-Aware Minimisation: Understanding and Mitigating Estimator Bias in Private SGD
Institute for AI in Medicine, Technical University of Munich（慕尼黑工业大学医学人工智能研究所）；Konrad Zuse School for Excelllence in Reliable AI（康拉德·楚泽卓越可靠人工智能学院）；V7 Labs（V7实验室）；Imperial College London（帝国理工学院）
