A Theory on Adam Instability in Large-Scale Machine Learning
Meta AI
Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, Binh Tang, Diana… 展开作者
Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, Binh Tang, Diana Liskovich, Puxin Xu, Yuchen Zhang, Melanie Kambadur, Stephen Roller, Susan Zhang
We present a theory for the previously unexplained divergent behavior noticed in the training of large language models. We argue that the phenomenon is an artifact of the dominant optimization algorithm used for training, called Adam. We observe that Adam can enter a state in which the parameter update vector has a relatively large norm and is essentially uncorrelated with the direction of descent on the training loss landscape, leading to divergence. This artifact is more likely to be observed in the training of a deep model with a large batch size, which is the typical setting of large-scale language model training. To argue the theory, we present observations from the training runs of the language models of different scales: 7 billion, 30 billion, 65 billion, and 546 billion parameters.
SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training
University of Exeter(埃克塞特大学);Eindhoven University of Technology(埃因霍温理工大学);University of Leicester(莱斯特大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Oxford(牛津大学)
A Control Theoretic Framework for Adaptive Gradient Optimizers in Machine Learning
Division of Data and Decision Sciences(数据与决策科学部);Tata Consultancy Services Research(塔塔咨询服务公司研究院);University of Maryland(马里兰大学);Department of Mechanical Engineering(机械工程系)
本文基于状态空间建模与控制理论传递函数提出自适应梯度优化通用框架及 Adam 新变体 AdamSSM,证明其收敛性,并在图像分类和语言建模任务上验证其泛化与收敛优势。
Improving the Adaptive Moment Estimation (ADAM) stochastic optimizer through an Implicit-Explicit (IMEX) time-stepping approach
Computational Science Laboratory(计算科学实验室);Virginia Tech(弗吉尼亚理工大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校);California State University, Long Beach(加州州立大学长滩分校)
Averaged Adam accelerates stochastic optimization in the training of deep neural network approximations for partial differential equation and optimal control problems
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院)
本文受 Polyak-Ruppert 平均化启发,将平均化 Adam 用于 PDE、最优控制及图像分类的 DNN 训练,并在 PINN、deep BSDE 等多个数值实验中证明其优于标准 Adam 与 SGD。
Machine-Learning Driven Load Shedding to Mitigate Instability Attacks in Power Grids
Information and Decision Algorithms Laboratories (IDeA Labs), Brigham Young University, Provo, Utah(信息与决策算法实验室(IDeA实验室), Brigham Young大学,犹他州普罗沃);University of Utah, Salt Lake City, Utah(犹他大学,盐湖城,犹他州);Achilles Heel Technologies (AHT)(Achilles Heel Technologies(AHT))
Quantum Cognition Machine Learning for Forecasting Chromosomal Instability
Qognitive, Inc.(Qognitive公司);Epic Sciences;Stony Brook University(石溪大学);Indiana University(印第安纳大学);Wayne State University(韦恩州立大学);University of Vienna(维也纳大学);Cornell University(康奈尔大学);New Jersey Institute of Technology(新泽西理工学院)
Optimizing Energy-based Neural Network Training with Coherent Ising Machine
School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院);Laboratory for Advanced Computing and Intelligence Engineering, Information Engineering University(信息工程大学先进计算与智能工程实验室);China Mobile (Suzhou) Software Technology Company Limited(中移(苏州)软件技术有限公司);School of Science, Beijing University of Posts and Telecommunications(北京邮电大学理学院)
Instability in Complex Oscillator Networks: Limitations and Potentials of Network Measures and Machine Learning
RD IV Complexity Science(RD IV 复杂性科学);Potsdam Institute for Climate Impact Research(波茨坦气候影响研究所);Technical University of Berlin(柏林技术大学);Department of Digital Transformation in Energy Systems, Institute of Energy Technology(能源系统数字化转型部门,能源技术研究所)
VibroML: an automated toolkit for high-throughput vibrational analysis and dynamic instability remediation of crystalline materials using machine-learned potentials