arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2304.09871cs.LGcs.AImath.OC
收藏

大规模机器学习中 Adam 不稳定性的理论

A Theory on Adam Instability in Large-Scale Machine Learning

Meta AI
Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, Binh Tang, Diana… 展开作者

Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, Binh Tang, Diana Liskovich, Puxin Xu, Yuchen Zhang, Melanie Kambadur, Stephen Roller, Susan Zhang

AI总结:

本文提出 Adam 优化器在大批量深度模型训练中可能因更新范数过大且与损失下降方向不相关而导致发散的理论,并通过 70亿至 5460亿参数模型的训练观察加以验证。

摘要:

我们提出一种理论,用于解释在大语言模型训练中观察到的、此前未被解释的发散行为。我们认为,该现象是训练所采用的主导优化算法(名为 Adam)产生的一种伪影。我们观察到,Adam 可能进入一种状态:参数更新向量具有相对较大的范数,并且与训练损失景观上的下降方向基本不相关,从而导致发散。在使用大批量的深度模型训练中,这种伪影更有可能被观察到,而这正是大规模语言模型训练的典型设置。为论证该理论,我们展示了来自不同规模语言模型训练运行的观察结果:这些模型分别具有 70亿、300亿、650亿和 5460亿个参数。

英文摘要:

We present a theory for the previously unexplained divergent behavior noticed in the training of large language models. We argue that the phenomenon is an artifact of the dominant optimization algorithm used for training, called Adam. We observe that Adam can enter a state in which the parameter update vector has a relatively large norm and is essentially uncorrelated with the direction of descent on the training loss landscape, leading to divergence. This artifact is more likely to be observed in the training of a deep model with a large batch size, which is the typical setting of large-scale language model training. To argue the theory, we present observations from the training runs of the language models of different scales: 7 billion, 30 billion, 65 billion, and 546 billion parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑