Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Institute for Medical Engineering & Science, Massachusetts Institute of Technology(麻省理工学院医学工程与科学研究所);School of Computing, University of Georgia(佐治亚大学计算学院);College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院)
Yili Wang, Kaixiong Zhou, Ninghao Liu, Ying Wang, Xin Wang
Sharpness-aware minimization (SAM) has received increasing attention in computer vision since it can effectively eliminate the sharp local minima from the training trajectory and mitigate generalization degradation. However, SAM requires two sequential gradient computations during the optimization of each step: one to obtain the perturbation gradient and the other to obtain the updating gradient. Compared with the base optimizer (e.g., Adam), SAM doubles the time overhead due to the additional perturbation gradient. By dissecting the theory of SAM and observing the training gradient of the molecular graph transformer, we propose a new algorithm named GraphSAM, which reduces the training cost of SAM and improves the generalization performance of graph transformer models. There are two key factors that contribute to this result: (i) \textit{gradient approximation}: we use the updating gradient of the previous step to approximate the perturbation gradient at the intermediate steps smoothly (\textbf{increases efficiency}); (ii) \textit{loss landscape approximation}: we theoretically prove that the loss landscape of GraphSAM is limited to a small range centered on the expected loss of SAM (\textbf{guarantees generalization performance}). The extensive experiments on six datasets with different tasks demonstrate the superiority of GraphSAM, especially in optimizing the model update process. The code is in:https://github.com/YL-wang/GraphSAM/tree/graphsam
Fast Graph Sharpness-Aware Minimization for Enhancing and Accelerating Few-Shot Node Classification
The Hong Kong University of Science and Technology(香港科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of California, Merced(加州大学默塞德分校);University of California, Los Angeles(加州大学洛杉矶分校);Createlink Technology(创联科技);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区网络空间安全学院)
VASSO: Variance Suppression for Sharpness-Aware Minimization
Department of Computer Science at ETH Zürich(苏黎世联邦理工学院计算机科学系);Department of Electrical and Computer Engineering, University of Minnesota(明尼苏达大学电气与计算机工程系)
Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization
CSMT, Wenzhou-Kean University(温州肯恩大学计算机科学与技术学院);International Frontier Interdisciplinary Research Institute, Wenzhou-Kean University(温州肯恩大学国际前沿交叉学科研究院)
Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
Institute of Information Engineering, CAS(中国科学院信息工程研究所);University of Chinese Academy of Sciences(中国科学院大学);Institute of Computing Tech., CAS(中国科学院计算技术研究所);China Electronics Technology Group Corporation(中国电子科技集团公司);Sun Yat-sen University(中山大学)
School of Cyber Science and Technology, Sun Yat-sen University, Shenzhen Campus(网络安全科学与技术学院,中山大学深圳校区);School of Computer Science, University of Science and Technology of China(计算机科学学院,中国科学技术大学);Hong Kong University of Science and Technology(香港科技大学)
Sharpness-Aware Minimization for Evolutionary Feature Construction in Regression
School of Engineering and Computer Science, Victoria University of Wellington(惠灵顿维多利亚大学工程与计算机科学学院);Department of Computer Science and Engineering, Michigan State University(密歇根州立大学计算机科学与工程系)
Sparse Layer Sharpness-Aware Minimization for Efficient Fine-Tuning
School of Cyber Science and Technology, Sun Yat-sen University, Shenzhen Campus(网络安全科学与技术学院,中山大学深圳校区);School of Computing, National University of Singapore(计算学院,新加坡国立大学);Baidu Inc(百度公司);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室(深圳))
TRAM: Bridging Trust Regions and Sharpness Aware Minimization
University of Edinburgh(爱丁堡大学);Kempner Institute, Harvard University(哈佛大学肯普纳研究所);Allen Institute for AI(艾伦人工智能研究所);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University(图像处理与模式识别研究所,自动化与智能感知学院,上海交通大学);MoE Key Laboratory of System Control and Information Processing (Shanghai)(系统控制与信息处理MOE重点实验室(上海))