本文研究了锐度感知最小化(Sharpness-Aware Minimization,SAM)的基础收敛性质。SAM 是最近提出的一种基于梯度的优化方法 [Foret et al., 2021],能够显著提升深度神经网络的泛化能力。本文为该方法的收敛性质建立了理论保证,包括聚点的平稳性、梯度序列收敛到原点、函数值序列收敛到最优值,以及迭代序列收敛到最优解。所提供的收敛性分析基于不精确梯度下降框架 Khanh et al. [2023b],具有普适性,因此可以扩展到 SAM 的高效归一化版本,如 F-SAM [Li et al., 2024]、VaSSO [Li and Giannakis, 2023]、RSAM [Liu et al., 2022],以及 SAM 的非归一化版本,如 USAM [Andriushchenko and Flammarion, 2022]。本文在分类任务上使用深度学习模型进行了数值实验,以验证所提出分析的实际适用性。
英文摘要:
The paper investigates the fundamental convergence properties of Sharpness-Aware Minimization (SAM), a recently proposed gradient-based optimization method [Foret et al., 2021] that significantly improves the generalization of deep neural networks. The convergence properties, including the stationarity of accumulation points, the convergence of the sequence of gradients to the origin, the sequence of function values to the optimal value, and the sequence of iterates to the optimal solution, are established for the method. The universality of the provided convergence analysis, based on inexact gradient descent frameworks Khanh et al. [2023b], allows its extensions to efficient normalized versions of SAM such as F-SAM [Li et al., 2024], VaSSO [Li and Giannakis, 2023], RSAM [Liu et al., 2022], and to the unnormalized versions of SAM such as USAM [Andriushchenko and Flammarion, 2022]. Numerical experiments are conducted on classification tasks using deep learning models to confirm the practical aspects of our analysis.
Sharpness-Aware Minimization Can Hallucinate Minimizers
Department of Electrical and Computer Engineering, Seoul National University(电子与计算机工程系,首尔国立大学);Department of Mathematics, University of California, Los Angeles(数学系,加州大学洛杉矶分校)
PurSAMERE: Reliable Adversarial Purification via Sharpness-Aware Minimization of Expected Reconstruction Error
Department of Mathematics, RWTH-Aachen University(亚琛工业大学数学系);Institute for a sustainable Hydrogen Economy (IHE), Forschungszentrum Jülich, Germany(可持续氢能经济研究所(IHE)、茹里奇研究中心,德国);Scientific Computing Center and Institute for Applied and Numerical Mathematics, Karlsruhe Institute of Technology, Germany(科学计算中心和应用与数值数学研究所、卡尔斯鲁厄理工学院,德国);Department of Mathematics & Munich Center for Machine Learning, Ludwig Maximilian University of Munich, Germany(数学系及慕尼黑机器学习中心、慕尼黑路德维希-马克西米利安大学,德国);Computer, Electrical and Mathematical Sciences and Engineering, KAUST, Saudi Arabia(计算机、电气和数学科学与工程,卡塔尔大学,沙特阿拉伯)