arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.02092cs.LGmath.OC
收藏

Adam在更新规则上无需任何修改即可收敛

Adam Converges Without Any Modification On Update Rules

The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院)

Yushun Zhang, Bingran Li, Congliang Chen, Zhi-Quan Luo, Ruoyu Sun

AI总结:

本研究证明Adam在适当的问题依赖超参数下可收敛,并揭示了Adam在不同超参数组合下的相变特性,为优化器调参提供了理论依据。

摘要:

Adam是训练神经网络的默认算法,包括大型语言模型(LLMs)。然而,\citet{reddi2019convergence}提供了一个例子,表明Adam会发散,这引发了在AI模型训练中使用它的担忧。我们识别了发散示例与实践之间的关键不匹配:\citet{reddi2019convergence}在选择Adam的超参数(β₁,β₂)之后才选择问题;而实际应用中通常先固定问题再调整(β₁,β₂)。在本工作中,我们证明了Adam在适当的依赖于问题的超参数下会收敛。首先,我们证明当β₂较大且β₁ < √β₂时,Adam会收敛。其次,当β₂较小时,我们指出了一组(β₁,β₂)组合区域,其中Adam可能发散到无穷大。我们的结果表明,当改变(β₁,β₂)组合时,Adam从发散到收敛存在相变。到目前为止,这是文献中首次在(β₁,β₂)二维平面上报告的相变,为Adam优化器提供了严格的理论保证。我们进一步指出,临界边界(β₁*,β₂*)是问题依赖的,并且特别依赖于批量大小。这为如何调整β₁和β₂提供了建议:当Adam表现不佳时,我们建议反向随批量大小调整增加β₂以超过阈值β₂*,然后尝试β₁ < √β₂。我们的建议得到了几项实证研究的报告支持,这些研究观察到在应用时改进了LLM训练性能。

英文摘要:

Adam is the default algorithm for training neural networks, including large language models (LLMs). However, \citet{reddi2019convergence} provided an example that Adam diverges, raising concerns for its deployment in AI model training. We identify a key mismatch between the divergence example and practice: \citet{reddi2019convergence} pick the problem after picking the hyperparameters of Adam, i.e., $(β_1,β_2)$; while practical applications often fix the problem first and then tune $(β_1,β_2)$. In this work, we prove that Adam converges with proper problem-dependent hyperparameters. First, we prove that Adam converges when $β_2$ is large and $β_1 < \sqrt{β_2}$. Second, when $β_2$ is small, we point out a region of $(β_1,β_2)$ combinations where Adam can diverge to infinity. Our results indicate a phase transition for Adam from divergence to convergence when changing the $(β_1, β_2)$ combination. To our knowledge, this is the first phase transition in $(β_1,β_2)$ 2D-plane reported in the literature, providing rigorous theoretical guarantees for Adam optimizer. We further point out that the critical boundary $(β_1^*, β_2^*)$ is problem-dependent, and particularly, dependent on batch size. This provides suggestions on how to tune $β_1$ and $β_2$: when Adam does not work well, we suggest tuning up $β_2$ inversely with batch size to surpass the threshold $β_2^*$, and then trying $β_1< \sqrt{β_2}$. Our suggestions are supported by reports from several empirical studies, which observe improved LLM training performance when applying them.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑