Sharpness-Aware Minimization (SAM) is a recent training method that relies on worst-case weight perturbations which significantly improves generalization in various settings. We argue that the existing justifications for the success of SAM which are based on a PAC-Bayes generalization bound and the idea of convergence to flat minima are incomplete. Moreover, there are no explanations for the success of using $m$-sharpness in SAM which has been shown as essential for generalization. To better understand this aspect of SAM, we theoretically analyze its implicit bias for diagonal linear networks. We prove that SAM always chooses a solution that enjoys better generalization properties than standard gradient descent for a certain class of problems, and this effect is amplified by using $m$-sharpness. We further study the properties of the implicit bias on non-linear networks empirically, where we show that fine-tuning a standard model with SAM can lead to significant generalization improvements. Finally, we provide convergence results of SAM for non-convex objectives when used with stochastic gradients. We illustrate these results empirically for deep networks and discuss their relation to the generalization behavior of SAM. The code of our experiments is available at https://github.com/tml-epfl/understanding-sam.
Recently, learning algorithms motivated from sharpness of loss surface as an effective measure of generalization gap have shown state-of-the-art performances. Nevertheless, sharpness defined in a rigid region with a fixed radius, has a drawback in sensitivity to parameter re-scaling which leaves the loss unaffected, leading to weakening of the connection between sharpness and generalization gap. In this paper, we introduce the concept of adaptive sharpness which is scale-invariant and propose the corresponding generalization bound. We suggest a novel learning method, adaptive sharpness-aware m…
Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
Institute of Information Engineering, CAS（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Tech., CAS（中国科学院计算技术研究所）；China Electronics Technology Group Corporation（中国电子科技集团公司）；Sun Yat-sen University（中山大学）
University of Maryland（马里兰大学）；New York University（纽约大学）
Sharpness-Aware Minimization (SAM) has recently emerged as a robust technique for improving the accuracy of deep neural networks. However, SAM incurs a high computational cost in practice, requiring up to twice as much computation as vanilla SGD. The computational challenge posed by SAM arises because each iteration requires both ascent and descent steps and thus double the gradient computations. To address this challenge, we propose to compute gradients in both stages of SAM on only the top-k samples with highest loss. K-SAM is simple and extremely easy-to-implement while providing significan…
School of Cyber Science and Technology, Sun Yat-sen University, Shenzhen Campus（网络安全科学与技术学院，中山大学深圳校区）；School of Computer Science, University of Science and Technology of China（计算机科学学院，中国科学技术大学）；Hong Kong University of Science and Technology（香港科技大学）
Google Research（谷歌研究院）；Blueshift, Alphabet（Alphabet 旗下 Blueshift）
In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability. Indeed, optimizing only the training loss value, as is commonly done, can easily lead to suboptimal model quality. Motivated by prior work connecting the geometry of the loss landscape and generalization, we introduce a novel, effective procedure for instead simultaneously minimizing loss value and loss sharpness. In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks parameters that lie in neighborhoods having uniformly low loss; this formulatio…
Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning
SGIT AI Lab, State Grid Corporation of China（国网SGIT人工智能实验室）；School of Management, Xi’an Jiaotong University（西安交通大学管理学院）；School of Software, Xi’an Jiaotong University（西安交通大学软件学院）
Centre for Frontier AI Research (CFAR), A*STAR（新加坡科技研究局前沿人工智能研究中心）；National University of Singapore（新加坡国立大学）；Institute of High Performance Computing (IHPC), A*STAR（新加坡科技研究局高性能计算研究所）
Overparametrized Deep Neural Networks (DNNs) often achieve astounding performances, but may potentially result in severe generalization error. Recently, the relation between the sharpness of the loss landscape and the generalization error has been established by Foret et al. (2020), in which the Sharpness Aware Minimizer (SAM) was proposed to mitigate the degradation of the generalization. Unfortunately, SAM s computational cost is roughly double that of base optimizers, such as Stochastic Gradient Descent (SGD). This paper thus proposes Efficient Sharpness Aware Minimizer (ESAM), which boosts…
Membership Privacy Risks of Sharpness Aware Minimization
Department of Computer Science（计算机科学系）；Purdue University（普渡大学）；Delft University of Technology（代尔夫特理工大学）；Department of Industrial and Systems Engineering（工业与系统工程系）；University of Southern California（南加州大学）
Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy
School of Mathematics and Statistics（数学与统计学院）；Xi’an Jiaotong University（西安交通大学）；Centrum Wiskunde & Informatica（国家数学和计算机科学研究院（ Centrum Wiskunde & Informatica））
针对 SAM 易陷入鞍点且有效学习率区间有限的问题，本文提出轻量重归一化方法 SSAM，扩展有效学习率区间并在多个数据集和任务上稳定提升性能。
Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization
Northwestern Polytechnical University（西北工业大学）；SGIT AI Lab, State Grid Corporation of China（国网SGIT人工智能实验室）；Xi’an Jiaotong University（西安交通大学）；ETH Zürich（苏黎世联邦理工学院）；Chinese University of Hong Kong（香港中文大学）
The recently proposed Sharpness-Aware Minimization (SAM) improves generalization by minimizing a \textit{perturbed loss} defined as the maximum loss within a neighborhood in the parameter space. However, we show that both sharp and flat minima can have a low perturbed loss, implying that SAM does not always prefer flat minima. Instead, we define a \textit{surrogate gap}, a measure equivalent to the dominant eigenvalue of Hessian at a local minimum when the radius of the neighborhood (to derive the perturbed loss) is small. The surrogate gap is easy to compute and feasible for direct minimizati…
Sharpness-Aware Minimization Can Hallucinate Minimizers
Department of Electrical and Computer Engineering, Seoul National University（电子与计算机工程系，首尔国立大学）；Department of Mathematics, University of California, Los Angeles（数学系，加州大学洛杉矶分校）
National University of Singapore（新加坡国立大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Recently, Sharpness-Aware Minimization (SAM), which connects the geometry of the loss landscape and generalization, has demonstrated significant performance boosts on training large-scale models such as vision transformers. However, the update rule of SAM requires two sequential (non-parallelizable) gradient computations at each step, which can double the computational overhead. In this paper, we propose a novel algorithm LookSAM - that only periodically calculates the inner gradient ascent, to significantly reduce the additional training cost of SAM. The empirical results illustrate that Look…
Xiamen University（厦门大学）；JD Explore Academy（京东探索研究院）；The University of Sydney（悉尼大学）
Deep neural networks often suffer from poor generalization caused by complex and non-convex loss landscapes. One of the popular solutions is Sharpness-Aware Minimization (SAM), which smooths the loss landscape via minimizing the maximized change of training loss when adding a perturbation to the weight. However, we find the indiscriminate perturbation of SAM on all parameters is suboptimal, which also results in excessive computation, i.e., double the overhead of common optimizers like Stochastic Gradient Descent (SGD). In this paper, we propose an efficient and effective training scheme coine…
Gradient Compression May Hurt Generalization: A Remedy by Synthetic Data Guided Sharpness Aware Minimization
Department of Information and Communication Engineering, Zhejiang University（浙江大学信息与通信工程学院）；Zhejiang Provincial Key Laboratory of Multi-Modal Communication Networks and Intelligent Information Processing（浙江省多模态通信网络与智能信息处理重点实验室）；Department of Electrical and Computer Engineering, North Carolina State University（北卡罗来纳州立大学电子与计算机工程系）
Stability Analysis of Sharpness-Aware Minimization
Chung-Ang University, South Korea（Chung-Ang 大学，韩国）；Korea Institute for Advanced Study, South Korea（韩国高级研究院）；Ulsan National Institute of Science（乌山国家科学研究院）；Nanyang Technological University (NTU), Singapore（南洋理工大学（NTU），新加坡）；Seoul National University, South Korea（首尔国立大学，韩国）
Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimization
School of Information Science and Technology, Beijing University of Technology（北京工业大学信息科学与技术学院）；Hangzhou Research Institute, School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院杭州研究院）；Nanchang Institute of Technology（南昌理工学院）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工大学宁波数字孪生研究院）
University of Science and Technology of China（中国科学技术大学）；Wuhan University（武汉大学）；JD Explore Academy（京东探索研究院）
Sharpness aware minimization (SAM) optimizer has been extensively explored as it can generalize better for training deep neural networks via introducing extra perturbation steps to flatten the landscape of deep learning models. Integrating SAM with adaptive learning rate and momentum acceleration, dubbed AdaSAM, has already been explored empirically to train large-scale deep neural networks without theoretical guarantee due to the triple difficulties in analyzing the coupled perturbation step, adaptive learning rate and momentum step. In this paper, we try to analyze the convergence rate of Ad…
Centre for Frontier AI Research (CFAR), A*STAR（新加坡科技研究局前沿人工智能研究中心）；National University of Singapore（新加坡国立大学）；ByteDance（字节跳动）
Modern deep neural networks (DNNs) have achieved state-of-the-art performances but are typically over-parameterized. The over-parameterization may result in undesirably large generalization error in the absence of other customized training strategies. Recently, a line of research under the name of Sharpness-Aware Minimization (SAM) has shown that minimizing a sharpness measure, which reflects the geometry of the loss landscape, can significantly reduce the generalization error. However, SAM-like methods incur a two-fold computational overhead of the given base optimizer (e.g. SGD) for approxim…
Southern University of Science and Technology（南方科技大学）；University of Technology Sydney（悉尼科技大学）；RIKEN Center for Advanced Intelligence Project (AIP)（理化学研究所先进智能项目中心）；Center for Frontier AI Research, Agency for Science Technology（新加坡科技研究局前沿人工智能研究中心）；Institute of High Performance Computing, Agency for Science Technology（新加坡科技研究局高性能计算研究所）；Nanyang Technological University（南洋理工大学）
Network quantization is a dominant paradigm of model compression. However, the abrupt changes in quantized weights during training often lead to severe loss fluctuations and result in a sharp loss landscape, making the gradients unstable and thus degrading the performance. Recently, Sharpness-Aware Minimization (SAM) has been proposed to smooth the loss landscape and improve the generalization performance of the models. Nevertheless, directly applying SAM to the quantized models can lead to perturbation mismatch or diminishment issues, resulting in suboptimal performance. In this paper, we pro…
Conflicting Biases at the Edge of Stability: Norm versus Sharpness Regularization
Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Prusa Research（普拉萨研究公司）；Institute for Robotics and Mechatronics, DLR-German Aerospace Center（德国航空航天中心机器人与机电研究所）
Methods for improving deep neural network training times and model generalizability consist of various data augmentation, regularization, and optimization approaches, which tend to be sensitive to hyperparameter settings and make reproducibility more challenging. This work jointly considers two recent training strategies that address model generalizability: sharpness-aware minimization, and self-distillation, and proposes the novel training strategy of Sharpness-Aware Distilled Teachers (SADT). The experimental section of this work shows that SADT consistently outperforms previously published …
Recently, Sharpness-Aware Minimization (SAM) algorithm has shown state-of-the-art generalization abilities in vision tasks. It demonstrates that flat minima tend to imply better generalization abilities. However, it has some difficulty implying SAM to some natural language tasks, especially to models with drastic gradient changes, such as RNNs. In this work, we analyze the relation between the flatness of the local minimum and its generalization ability from a novel and straightforward theoretical perspective. We propose that the shift of the training and test distributions can be equivalently…
VASSO: Variance Suppression for Sharpness-Aware Minimization
Department of Computer Science at ETH Zürich（苏黎世联邦理工学院计算机科学系）；Department of Electrical and Computer Engineering, University of Minnesota（明尼苏达大学电气与计算机工程系）
Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization
CSMT, Wenzhou-Kean University（温州肯恩大学计算机科学与技术学院）；International Frontier Interdisciplinary Research Institute, Wenzhou-Kean University（温州肯恩大学国际前沿交叉学科研究院）
University of South Florida（南佛罗里达大学）；Mississippi State University（密西西比州立大学）
Federated Learning (FL) is a promising framework for performing privacy-preserving, distributed learning with a set of clients. However, the data distribution among clients often exhibits non-IID, i.e., distribution shift, which makes efficient optimization difficult. To tackle this problem, many FL algorithms focus on mitigating the effects of data heterogeneity across clients by increasing the performance of the global model. However, almost all algorithms leverage Empirical Risk Minimization (ERM) to be the local optimizer, which is easy to make the global model fall into a sharp valley and…
