arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.16788cs.LGcs.AI
收藏

为什么Transformer需要Adam:一种Hessian视角

Why Transformers Need Adam: A Hessian Perspective

Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))

Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, Zhi-Quan Luo

AI总结:

本文从Hessian块异质性视角解释Transformer上SGD差于Adam的原因,并通过多种模型与二次问题验证:统一学习率难以处理块间异质性,而Adam的逐坐标学习率可缓解该问题。

摘要:

在Transformer上,SGD的表现显著差于Adam,但其原因仍不清楚。本文通过Hessian视角给出解释:(i)Transformer是“异质的”:不同参数块之间的Hessian谱差异巨大,我们将这一现象称为“块异质性”;(ii)异质性阻碍了SGD:在存在块异质性的问题上,SGD表现差于Adam。为验证(i)和(ii),我们检验了多种Transformer、CNN、MLP以及二次问题,发现SGD在没有块异质性的问题上可以达到与Adam相当的表现,但当异质性存在时则表现差于Adam。我们的初步理论分析表明,SGD表现较差是因为它对所有块使用同一个学习率,无法处理块之间的异质性。如果像Adam设计的那样使用逐坐标学习率,这一局限可以得到改善。

英文摘要:

SGD performs worse than Adam by a significant margin on Transformers, but the reason remains unclear. In this work, we provide an explanation through the lens of Hessian: (i) Transformers are "heterogeneous": the Hessian spectrum across parameter blocks vary dramatically, a phenomenon we call "block heterogeneity"; (ii) Heterogeneity hampers SGD: SGD performs worse than Adam on problems with block heterogeneity. To validate (i) and (ii), we check various Transformers, CNNs, MLPs, and quadratic problems, and find that SGD can perform on par with Adam on problems without block heterogeneity, but performs worse than Adam when the heterogeneity exists. Our initial theoretical analysis indicates that SGD performs worse because it applies one single learning rate to all blocks, which cannot handle the heterogeneity among blocks. This limitation could be ameliorated if we use coordinate-wise learning rates, as designed in Adam.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑