arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.13137cs.LG
收藏

面向分子图Transformer模型的高效锐度感知最小化方法

Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models

School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Institute for Medical Engineering & Science, Massachusetts Institute of Technology(麻省理工学院医学工程与科学研究所);School of Computing, University of Georgia(佐治亚大学计算学院);College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院)

Yili Wang, Kaixiong Zhou, Ninghao Liu, Ying Wang, Xin Wang

AI总结:

针对SAM训练时间开销翻倍的问题,提出GraphSAM算法,通过梯度近似与损失景观近似,在提升分子图Transformer训练效率的同时保障泛化性能,经多数据集实验验证其优越性。

摘要:

锐度感知最小化(SAM)因能有效消除训练轨迹中的尖锐局部极小值、缓解泛化性能下降问题,在计算机视觉领域受到越来越多的关注。但SAM在每一步优化过程中需要两次连续的梯度计算:一次用于获取扰动梯度,另一次用于获取更新梯度。与基础优化器(如Adam)相比,SAM因额外的扰动梯度计算使时间开销翻倍。通过剖析SAM的理论并观察分子图Transformer的训练梯度,我们提出了名为GraphSAM的新算法,该算法降低了SAM的训练成本,同时提升了图Transformer模型的泛化性能。这一效果得益于两个关键因素:(i)梯度近似:我们使用上一步的更新梯度,平滑地近似中间步骤的扰动梯度(提升效率);(ii)损失景观近似:我们从理论上证明,GraphSAM的损失景观被限制在以SAM的期望损失为中心的小范围内(保障泛化性能)。在6个不同任务的数据集上开展的大量实验证明了GraphSAM的优越性,尤其是在优化模型更新流程方面表现突出。代码位于:https://github.com/YL-wang/GraphSAM/tree/graphsam

英文摘要:

Sharpness-aware minimization (SAM) has received increasing attention in computer vision since it can effectively eliminate the sharp local minima from the training trajectory and mitigate generalization degradation. However, SAM requires two sequential gradient computations during the optimization of each step: one to obtain the perturbation gradient and the other to obtain the updating gradient. Compared with the base optimizer (e.g., Adam), SAM doubles the time overhead due to the additional perturbation gradient. By dissecting the theory of SAM and observing the training gradient of the molecular graph transformer, we propose a new algorithm named GraphSAM, which reduces the training cost of SAM and improves the generalization performance of graph transformer models. There are two key factors that contribute to this result: (i) \textit{gradient approximation}: we use the updating gradient of the previous step to approximate the perturbation gradient at the intermediate steps smoothly (\textbf{increases efficiency}); (ii) \textit{loss landscape approximation}: we theoretically prove that the loss landscape of GraphSAM is limited to a small range centered on the expected loss of SAM (\textbf{guarantees generalization performance}). The extensive experiments on six datasets with different tasks demonstrate the superiority of GraphSAM, especially in optimizing the model update process. The code is in:https://github.com/YL-wang/GraphSAM/tree/graphsam

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑