arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.08060math.OC
收藏

锐度感知最小化的基础收敛性分析

Fundamental Convergence Analysis of Sharpness-Aware Minimization

Pham Duy Khanh, Hoang-Chau Luong, Boris S. Mordukhovich, Dat Ba Tran

AI总结:

本文基于不精确梯度下降框架建立了锐度感知最小化(SAM)的基础收敛性理论,证明其梯度、函数值和迭代序列均收敛到最优,并将分析推广到 F-SAM、VaSSO、RSAM、USAM 等变体,同时通过深度学习分类实验验证了理论的实际有效性。

摘要:

本文研究了锐度感知最小化(Sharpness-Aware Minimization,SAM)的基础收敛性质。SAM 是最近提出的一种基于梯度的优化方法 [Foret et al., 2021],能够显著提升深度神经网络的泛化能力。本文为该方法的收敛性质建立了理论保证,包括聚点的平稳性、梯度序列收敛到原点、函数值序列收敛到最优值,以及迭代序列收敛到最优解。所提供的收敛性分析基于不精确梯度下降框架 Khanh et al. [2023b],具有普适性,因此可以扩展到 SAM 的高效归一化版本,如 F-SAM [Li et al., 2024]、VaSSO [Li and Giannakis, 2023]、RSAM [Liu et al., 2022],以及 SAM 的非归一化版本,如 USAM [Andriushchenko and Flammarion, 2022]。本文在分类任务上使用深度学习模型进行了数值实验,以验证所提出分析的实际适用性。

英文摘要:

The paper investigates the fundamental convergence properties of Sharpness-Aware Minimization (SAM), a recently proposed gradient-based optimization method [Foret et al., 2021] that significantly improves the generalization of deep neural networks. The convergence properties, including the stationarity of accumulation points, the convergence of the sequence of gradients to the origin, the sequence of function values to the optimal value, and the sequence of iterates to the optimal solution, are established for the method. The universality of the provided convergence analysis, based on inexact gradient descent frameworks Khanh et al. [2023b], allows its extensions to efficient normalized versions of SAM such as F-SAM [Li et al., 2024], VaSSO [Li and Giannakis, 2023], RSAM [Liu et al., 2022], and to the unnormalized versions of SAM such as USAM [Andriushchenko and Flammarion, 2022]. Numerical experiments are conducted on classification tasks using deep learning models to confirm the practical aspects of our analysis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑