Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning
SGIT AI Lab, State Grid Corporation of China（国网SGIT人工智能实验室）；School of Management, Xi’an Jiaotong University（西安交通大学管理学院）；School of Software, Xi’an Jiaotong University（西安交通大学软件学院）
Yao Fu, Chunxia Zhang, Junmin Liu, Yihang Jin, Haishan Ye, Yuanao Yang
Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data. Building on Sharpness-Aware Minimization (SAM), for seeking flat minima associated with improved generalization, we propose the Extragradient-Inspired Sharpness-Aware Minimization (EISAM), a novel optimizer that enhances generalization via the extragradient technique. EISAM uses a two-step update process: a prediction step investigating the geometry of the loss landscape and a perturbation step that refines updates with a base optimizer. This approach achieves better generalization performance than SAM. Crucially, EISAM reduces sensitivity to the perturbation radius, enhancing robustness, and simplifying the tuning across diverse settings. Extensive experiments on benchmark datasets demonstrate that EISAM consistently outperforms SGD, Adaptive Moment Estimation (Adam), and SAM in test accuracy and training efficiency across various architectures. Theoretical analysis further confirms that EISAM tightens the generalization bound by steering parameters toward flatter minima with reduced curvature. Accompanied by a thorough hyperparameter analysis, EISAM offers practical tuning guidance, establishing it as a robust, scalable, and broadly applicable optimization solution that advances both the theory and practice in deep learning.
Stochastic variance reduced extragradient methods for solving hierarchical variational inequalities
Weierstrass Institute for Applied Analysis and Stochastics（魏尔斯特拉斯应用分析与随机学研究所）；Mannheim University（曼海姆大学）；Technion - Israel Institute of Technology（以色列理工学院）
Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
Institute of Information Engineering, CAS（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Tech., CAS（中国科学院计算技术研究所）；China Electronics Technology Group Corporation（中国电子科技集团公司）；Sun Yat-sen University（中山大学）
Efficient Sharpness-aware Minimization for Improved Training of Neural Networks
Centre for Frontier AI Research (CFAR), A*STAR（新加坡科技研究局前沿人工智能研究中心）；National University of Singapore（新加坡国立大学）；Institute of High Performance Computing (IHPC), A*STAR（新加坡科技研究局高性能计算研究所）
过度参数化的深度神经网络（DNN）通常可以实现惊人的性能，但可能会导致严重的泛化误差。最近，Foret et al.（2020）建立了损失景观的清晰度与泛化误差之间的关系，其中提出了SharpingAware Minimizer（SAM）来减轻泛化的退化。不幸的是，SAM的计算成本大约是基本优化器（如随机梯度下降（SGD））的两倍。因此，本文提出了高效的共享感知最小化器（ESAM），它提高了SAM的效率，而不牺牲其泛化性能。ESAM包括两种新颖有效的训练策略--随机权值扰动和锐度敏感数据选择。在前者中，通过在每次迭代中扰动随机选择的权重集来近似锐度度量;在后者中，仅使用对锐度敏感的明智选择的数据子集来优化SAM损失。我们提供了理论解释，为什么这些策略表现良好。我们还通过对CIFAR和ImageNet数据集的广泛实验表明，ESAM比SAM提高了效率，从需要100%的额外计算到相对于基本优化器的40%，同时保持甚至提高了测试准确度。
School of Cyber Science and Technology, Sun Yat-sen University, Shenzhen Campus（网络安全科学与技术学院，中山大学深圳校区）；School of Computer Science, University of Science and Technology of China（计算机科学学院，中国科学技术大学）；Hong Kong University of Science and Technology（香港科技大学）
Membership Privacy Risks of Sharpness Aware Minimization
Department of Computer Science（计算机科学系）；Purdue University（普渡大学）；Delft University of Technology（代尔夫特理工大学）；Department of Industrial and Systems Engineering（工业与系统工程系）；University of Southern California（南加州大学）
Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy
School of Mathematics and Statistics（数学与统计学院）；Xi’an Jiaotong University（西安交通大学）；Centrum Wiskunde & Informatica（国家数学和计算机科学研究院（ Centrum Wiskunde & Informatica））
针对 SAM 易陷入鞍点且有效学习率区间有限的问题，本文提出轻量重归一化方法 SSAM，扩展有效学习率区间并在多个数据集和任务上稳定提升性能。
Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization
Northwestern Polytechnical University（西北工业大学）；SGIT AI Lab, State Grid Corporation of China（国网SGIT人工智能实验室）；Xi’an Jiaotong University（西安交通大学）；ETH Zürich（苏黎世联邦理工学院）；Chinese University of Hong Kong（香港中文大学）
Sharpness-Aware Minimization Can Hallucinate Minimizers
Department of Electrical and Computer Engineering, Seoul National University（电子与计算机工程系，首尔国立大学）；Department of Mathematics, University of California, Los Angeles（数学系，加州大学洛杉矶分校）
Gradient Compression May Hurt Generalization: A Remedy by Synthetic Data Guided Sharpness Aware Minimization
Department of Information and Communication Engineering, Zhejiang University（浙江大学信息与通信工程学院）；Zhejiang Provincial Key Laboratory of Multi-Modal Communication Networks and Intelligent Information Processing（浙江省多模态通信网络与智能信息处理重点实验室）；Department of Electrical and Computer Engineering, North Carolina State University（北卡罗来纳州立大学电子与计算机工程系）
Stability Analysis of Sharpness-Aware Minimization
Chung-Ang University, South Korea（Chung-Ang 大学，韩国）；Korea Institute for Advanced Study, South Korea（韩国高级研究院）；Ulsan National Institute of Science（乌山国家科学研究院）；Nanyang Technological University (NTU), Singapore（南洋理工大学（NTU），新加坡）；Seoul National University, South Korea（首尔国立大学，韩国）
Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimization
School of Information Science and Technology, Beijing University of Technology（北京工业大学信息科学与技术学院）；Hangzhou Research Institute, School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院杭州研究院）；Nanchang Institute of Technology（南昌理工学院）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工大学宁波数字孪生研究院）
Conflicting Biases at the Edge of Stability: Norm versus Sharpness Regularization
Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Prusa Research（普拉萨研究公司）；Institute for Robotics and Mechatronics, DLR-German Aerospace Center（德国航空航天中心机器人与机电研究所）
Southern University of Science and Technology（南方科技大学）；University of Technology Sydney（悉尼科技大学）；RIKEN Center for Advanced Intelligence Project (AIP)（理化学研究所先进智能项目中心）；Center for Frontier AI Research, Agency for Science Technology（新加坡科技研究局前沿人工智能研究中心）；Institute of High Performance Computing, Agency for Science Technology（新加坡科技研究局高性能计算研究所）；Nanyang Technological University（南洋理工大学）
