The recently proposed Sharpness-Aware Minimization (SAM) improves generalization by minimizing a \textit{perturbed loss} defined as the maximum loss within a neighborhood in the parameter space. However, we show that both sharp and flat minima can have a low perturbed loss, implying that SAM does not always prefer flat minima. Instead, we define a \textit{surrogate gap}, a measure equivalent to the dominant eigenvalue of Hessian at a local minimum when the radius of the neighborhood (to derive the perturbed loss) is small. The surrogate gap is easy to compute and feasible for direct minimization during training. Based on the above observations, we propose Surrogate \textbf{G}ap Guided \textbf{S}harpness-\textbf{A}ware \textbf{M}inimization (GSAM), a novel improvement over SAM with negligible computation overhead. Conceptually, GSAM consists of two steps: 1) a gradient descent like SAM to minimize the perturbed loss, and 2) an \textit{ascent} step in the \textit{orthogonal} direction (after gradient decomposition) to minimize the surrogate gap and yet not affect the perturbed loss. GSAM seeks a region with both small loss (by step 1) and low sharpness (by step 2), giving rise to a model with high generalization capabilities. Theoretically, we show the convergence of GSAM and provably better generalization than SAM. Empirically, GSAM consistently improves generalization (e.g., +3.2\% over SAM and +5.4\% over AdamW on ImageNet top-1 accuracy for ViT-B/32). Code is released at \url{ https://sites.google.com/view/gsam-iclr22/home}.
Membership Privacy Risks of Sharpness Aware Minimization
Department of Computer Science（计算机科学系）；Purdue University（普渡大学）；Delft University of Technology（代尔夫特理工大学）；Department of Industrial and Systems Engineering（工业与系统工程系）；University of Southern California（南加州大学）
Recently, learning algorithms motivated from sharpness of loss surface as an effective measure of generalization gap have shown state-of-the-art performances. Nevertheless, sharpness defined in a rigid region with a fixed radius, has a drawback in sensitivity to parameter re-scaling which leaves the loss unaffected, leading to weakening of the connection between sharpness and generalization gap. In this paper, we introduce the concept of adaptive sharpness which is scale-invariant and propose the corresponding generalization bound. We suggest a novel learning method, adaptive sharpness-aware m…
Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy
School of Mathematics and Statistics（数学与统计学院）；Xi’an Jiaotong University（西安交通大学）；Centrum Wiskunde & Informatica（国家数学和计算机科学研究院（ Centrum Wiskunde & Informatica））
针对 SAM 易陷入鞍点且有效学习率区间有限的问题，本文提出轻量重归一化方法 SSAM，扩展有效学习率区间并在多个数据集和任务上稳定提升性能。
Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization
Northwestern Polytechnical University（西北工业大学）；SGIT AI Lab, State Grid Corporation of China（国网SGIT人工智能实验室）；Xi’an Jiaotong University（西安交通大学）；ETH Zürich（苏黎世联邦理工学院）；Chinese University of Hong Kong（香港中文大学）
Gradient Compression May Hurt Generalization: A Remedy by Synthetic Data Guided Sharpness Aware Minimization
Department of Information and Communication Engineering, Zhejiang University（浙江大学信息与通信工程学院）；Zhejiang Provincial Key Laboratory of Multi-Modal Communication Networks and Intelligent Information Processing（浙江省多模态通信网络与智能信息处理重点实验室）；Department of Electrical and Computer Engineering, North Carolina State University（北卡罗来纳州立大学电子与计算机工程系）
Centre for Frontier AI Research (CFAR), A*STAR（新加坡科技研究局前沿人工智能研究中心）；National University of Singapore（新加坡国立大学）；Institute of High Performance Computing (IHPC), A*STAR（新加坡科技研究局高性能计算研究所）
Overparametrized Deep Neural Networks (DNNs) often achieve astounding performances, but may potentially result in severe generalization error. Recently, the relation between the sharpness of the loss landscape and the generalization error has been established by Foret et al. (2020), in which the Sharpness Aware Minimizer (SAM) was proposed to mitigate the degradation of the generalization. Unfortunately, SAM s computational cost is roughly double that of base optimizers, such as Stochastic Gradient Descent (SGD). This paper thus proposes Efficient Sharpness Aware Minimizer (ESAM), which boosts…
Stability Analysis of Sharpness-Aware Minimization
Chung-Ang University, South Korea（Chung-Ang 大学，韩国）；Korea Institute for Advanced Study, South Korea（韩国高级研究院）；Ulsan National Institute of Science（乌山国家科学研究院）；Nanyang Technological University (NTU), Singapore（南洋理工大学（NTU），新加坡）；Seoul National University, South Korea（首尔国立大学，韩国）
National University of Singapore（新加坡国立大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Recently, Sharpness-Aware Minimization (SAM), which connects the geometry of the loss landscape and generalization, has demonstrated significant performance boosts on training large-scale models such as vision transformers. However, the update rule of SAM requires two sequential (non-parallelizable) gradient computations at each step, which can double the computational overhead. In this paper, we propose a novel algorithm LookSAM - that only periodically calculates the inner gradient ascent, to significantly reduce the additional training cost of SAM. The empirical results illustrate that Look…
Sharpness-Aware Minimization (SAM) is a recent training method that relies on worst-case weight perturbations which significantly improves generalization in various settings. We argue that the existing justifications for the success of SAM which are based on a PAC-Bayes generalization bound and the idea of convergence to flat minima are incomplete. Moreover, there are no explanations for the success of using $m$-sharpness in SAM which has been shown as essential for generalization. To better understand this aspect of SAM, we theoretically analyze its implicit bias for diagonal linear networks.…
University of Maryland（马里兰大学）；New York University（纽约大学）
Sharpness-Aware Minimization (SAM) has recently emerged as a robust technique for improving the accuracy of deep neural networks. However, SAM incurs a high computational cost in practice, requiring up to twice as much computation as vanilla SGD. The computational challenge posed by SAM arises because each iteration requires both ascent and descent steps and thus double the gradient computations. To address this challenge, we propose to compute gradients in both stages of SAM on only the top-k samples with highest loss. K-SAM is simple and extremely easy-to-implement while providing significan…
Xiamen University（厦门大学）；JD Explore Academy（京东探索研究院）；The University of Sydney（悉尼大学）
Deep neural networks often suffer from poor generalization caused by complex and non-convex loss landscapes. One of the popular solutions is Sharpness-Aware Minimization (SAM), which smooths the loss landscape via minimizing the maximized change of training loss when adding a perturbation to the weight. However, we find the indiscriminate perturbation of SAM on all parameters is suboptimal, which also results in excessive computation, i.e., double the overhead of common optimizers like Stochastic Gradient Descent (SGD). In this paper, we propose an efficient and effective training scheme coine…
Google Research（谷歌研究院）；Blueshift, Alphabet（Alphabet 旗下 Blueshift）
In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability. Indeed, optimizing only the training loss value, as is commonly done, can easily lead to suboptimal model quality. Motivated by prior work connecting the geometry of the loss landscape and generalization, we introduce a novel, effective procedure for instead simultaneously minimizing loss value and loss sharpness. In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks parameters that lie in neighborhoods having uniformly low loss; this formulatio…
Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
Institute of Information Engineering, CAS（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Tech., CAS（中国科学院计算技术研究所）；China Electronics Technology Group Corporation（中国电子科技集团公司）；Sun Yat-sen University（中山大学）
Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning
SGIT AI Lab, State Grid Corporation of China（国网SGIT人工智能实验室）；School of Management, Xi’an Jiaotong University（西安交通大学管理学院）；School of Software, Xi’an Jiaotong University（西安交通大学软件学院）
School of Cyber Science and Technology, Sun Yat-sen University, Shenzhen Campus（网络安全科学与技术学院，中山大学深圳校区）；School of Computer Science, University of Science and Technology of China（计算机科学学院，中国科学技术大学）；Hong Kong University of Science and Technology（香港科技大学）
Sharpness-Aware Minimization Can Hallucinate Minimizers
Department of Electrical and Computer Engineering, Seoul National University（电子与计算机工程系，首尔国立大学）；Department of Mathematics, University of California, Los Angeles（数学系，加州大学洛杉矶分校）
Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimization
School of Information Science and Technology, Beijing University of Technology（北京工业大学信息科学与技术学院）；Hangzhou Research Institute, School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院杭州研究院）；Nanchang Institute of Technology（南昌理工学院）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工大学宁波数字孪生研究院）
University of Science and Technology of China（中国科学技术大学）；Wuhan University（武汉大学）；JD Explore Academy（京东探索研究院）
Sharpness aware minimization (SAM) optimizer has been extensively explored as it can generalize better for training deep neural networks via introducing extra perturbation steps to flatten the landscape of deep learning models. Integrating SAM with adaptive learning rate and momentum acceleration, dubbed AdaSAM, has already been explored empirically to train large-scale deep neural networks without theoretical guarantee due to the triple difficulties in analyzing the coupled perturbation step, adaptive learning rate and momentum step. In this paper, we try to analyze the convergence rate of Ad…
Southern University of Science and Technology（南方科技大学）；University of Technology Sydney（悉尼科技大学）；RIKEN Center for Advanced Intelligence Project (AIP)（理化学研究所先进智能项目中心）；Center for Frontier AI Research, Agency for Science Technology（新加坡科技研究局前沿人工智能研究中心）；Institute of High Performance Computing, Agency for Science Technology（新加坡科技研究局高性能计算研究所）；Nanyang Technological University（南洋理工大学）
Stochastic majorization-minimization (SMM) is a class of stochastic optimization algorithms that proceed by sampling new data points and minimizing a recursive average of surrogate functions of an objective function. The surrogates are required to be strongly convex and convergence rate analysis for the general non-convex setting was not available. In this paper, we propose an extension of SMM where surrogates are allowed to be only weakly convex or block multi-convex, and the averaged surrogates are approximately minimized with proximal regularization or block-minimized within diminishing rad…
Surrogate risk minimization is an ubiquitous paradigm in supervised machine learning, wherein a target problem is solved by minimizing a surrogate loss on a dataset. Surrogate regret bounds, also called excess risk bounds, are a common tool to prove generalization rates for surrogate risk minimization. While surrogate regret bounds have been developed for certain classes of loss functions, such as proper losses, general results are relatively sparse. We provide two general results. The first gives a linear surrogate regret bound for any polyhedral (piecewise-linear and convex) surrogate, meani…
Centre for Frontier AI Research (CFAR), A*STAR（新加坡科技研究局前沿人工智能研究中心）；National University of Singapore（新加坡国立大学）；ByteDance（字节跳动）
Modern deep neural networks (DNNs) have achieved state-of-the-art performances but are typically over-parameterized. The over-parameterization may result in undesirably large generalization error in the absence of other customized training strategies. Recently, a line of research under the name of Sharpness-Aware Minimization (SAM) has shown that minimizing a sharpness measure, which reflects the geometry of the loss landscape, can significantly reduce the generalization error. However, SAM-like methods incur a two-fold computational overhead of the given base optimizer (e.g. SGD) for approxim…
Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；Polytechnique Montreal（蒙特利尔综合理工学院）
Energy-efficient deep neural network (DNN) accelerators are prone to non-idealities that degrade DNN performance at inference time. To mitigate such degradation, existing methods typically add perturbations to the DNN weights during training to simulate inference on noisy hardware. However, this often requires knowledge about the target hardware and leads to a trade-off between DNN performance and robustness, decreasing the former to increase the latter. In this work, we show that applying sharpness-aware training, by optimizing for both the loss value and loss sharpness, significantly improve…
CMAP, École polytechnique, Insitut Polytechnique de Paris（CMAP，巴黎高等理工学院，巴黎理工 institute）；LAAS-CNRS, Université de Toulouse, CNRS, Toulouse, France（LAAS-CNRS，图卢兹大学，CNRS，法国图卢兹）；Department of SEEM, The Chinese University of Hong Kong Shatin, Hong Kong（信息工程系，香港中文大学（深圳））
Conflicting Biases at the Edge of Stability: Norm versus Sharpness Regularization
Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Prusa Research（普拉萨研究公司）；Institute for Robotics and Mechatronics, DLR-German Aerospace Center（德国航空航天中心机器人与机电研究所）
Sharpness-Aware Minimization (SAM) and adaptive sharpness-aware minimization (ASAM) aim to improve the model generalization. And in this project, we proposed three experiments to valid their generalization from the sharpness aware perspective. And our experiments show that sharpness aware-based optimization techniques could help to provide models with strong generalization ability. Our experiments also show that ASAM could improve the generalization performance on un-normalized data, but further research is needed to confirm this.
Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（图像处理与模式识别研究所，自动化与智能感知学院，上海交通大学）；MoE Key Laboratory of System Control and Information Processing (Shanghai)（系统控制与信息处理MOE重点实验室（上海））
