提出受指数倾斜启发的平滑 SAM 推广 TSAM，用倾斜超参数优先考虑高损失局部解，并通过 Hamiltonian 动力学离散化算法求解，在图像和文本任务上获得更平坦最小值与更优测试性能。
摘要：
锐度感知最小化（Sharpness-Aware Minimization，SAM）已被证明可以通过在损失景观中寻找平坦最小值来提升过参数化模型的泛化性能，其做法是优化邻域内导致最大损失的模型参数。然而，这种最小-最大形式在计算上具有挑战性，尤其是当问题高度非凸时。此外，在寻找平坦最小值时，只关注最坏情况下的局部解而忽略可能存在的许多其他局部解，可能并非最优。在本文中，我们提出 Tilted SAM（TSAM），这是受指数倾斜启发的 SAM 的平滑推广，能够有效地为导致较大损失的局部解赋予更高优先级。TSAM 由倾斜超参数 $t$ 参数化，并在 $t$ 趋近于无穷大时退化为 SAM。我们证明 TSAM 比 SAM 更平滑，因此更易于优化，并且它明确偏好更平坦的最小值。我们开发了受 Hamiltonian 动力学离散化启发的算法来求解 TSAM。实验上，在一系列图像和文本任务中，TSAM 得到了更平坦的局部最小值，并且相较 SAM 和 ERM 基线取得了更优的测试性能。
英文摘要：
Sharpness-Aware Minimization (SAM) has been demonstrated to improve the generalization performance of overparameterized models by seeking flat minima on the loss landscape through optimizing model parameters that incur the largest loss within a neighborhood. Nevertheless, such min-max formulations are computationally challenging especially when the problem is highly non-convex. Additionally, focusing only on the worst-case local solution while ignoring potentially many other local solutions may be suboptimal when searching for flat minima. In this work, we propose Tilted SAM (TSAM), a smoothed generalization of SAM inspired by exponential tilting that effectively assigns higher priority to local solutions that incur larger losses. TSAM is parameterized by a tilt hyperparameter $t$ and reduces to SAM as $t$ approaches infinity. We show that TSAM is smoother than SAM and thus easier to optimize, and it explicitly favors flatter minima. We develop algorithms motivated by the discretization of Hamiltonian dynamics to solve TSAM. Empirically, TSAM arrives at flatter local minima and results in superior test performance than the baselines of SAM and ERM across a range of image and text tasks.
VASSO: Variance Suppression for Sharpness-Aware Minimization
Department of Computer Science at ETH Zürich（苏黎世联邦理工学院计算机科学系）；Department of Electrical and Computer Engineering, University of Minnesota（明尼苏达大学电气与计算机工程系）
Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization
CSMT, Wenzhou-Kean University（温州肯恩大学计算机科学与技术学院）；International Frontier Interdisciplinary Research Institute, Wenzhou-Kean University（温州肯恩大学国际前沿交叉学科研究院）
Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
Institute of Information Engineering, CAS（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Tech., CAS（中国科学院计算技术研究所）；China Electronics Technology Group Corporation（中国电子科技集团公司）；Sun Yat-sen University（中山大学）
School of Cyber Science and Technology, Sun Yat-sen University, Shenzhen Campus（网络安全科学与技术学院，中山大学深圳校区）；School of Computer Science, University of Science and Technology of China（计算机科学学院，中国科学技术大学）；Hong Kong University of Science and Technology（香港科技大学）
Sharpness-Aware Minimization for Evolutionary Feature Construction in Regression
School of Engineering and Computer Science, Victoria University of Wellington（惠灵顿维多利亚大学工程与计算机科学学院）；Department of Computer Science and Engineering, Michigan State University（密歇根州立大学计算机科学与工程系）
Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Institute for Medical Engineering & Science, Massachusetts Institute of Technology（麻省理工学院医学工程与科学研究所）；School of Computing, University of Georgia（佐治亚大学计算学院）；College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）
Sparse Layer Sharpness-Aware Minimization for Efficient Fine-Tuning
School of Cyber Science and Technology, Sun Yat-sen University, Shenzhen Campus（网络安全科学与技术学院，中山大学深圳校区）；School of Computing, National University of Singapore（计算学院，新加坡国立大学）；Baidu Inc（百度公司）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东人工智能与数字经济实验室（深圳））
TRAM: Bridging Trust Regions and Sharpness Aware Minimization
University of Edinburgh（爱丁堡大学）；Kempner Institute, Harvard University（哈佛大学肯普纳研究所）；Allen Institute for AI（艾伦人工智能研究所）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（图像处理与模式识别研究所，自动化与智能感知学院，上海交通大学）；MoE Key Laboratory of System Control and Information Processing (Shanghai)（系统控制与信息处理MOE重点实验室（上海））
Bilateral Sharpness-Aware Minimization for Flatter Minima
Faculty of Information Technology, Beijing University of Technology（北京工业大学信息技术学院）；School of Artificial Intelligence, BeiHang University（北京航空航天大学人工智能学院）；University of Chinese Academy of Sciences（中国科学院大学）；Chinese Academy of Sciences (CAS)（中国科学院）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）
