Training extremely large language models (LLMs) with billions of parameters is a computationally intensive task that pushes the limits of current data parallel training systems. While techniques like ZeRO++ have enabled efficient distributed training of such giant models on inexpensive low-bandwidth clusters, they can suffer from convergence issues due to potential race conditions in the hierarchical partitioning (hpZ) scheme employed to reduce cross-machine communication. In this work, we first show how these race conditions cause instability when training models with billions of parameters. We then propose a modification to the partitioning algorithm that addresses these convergence challenges while maintaining competitive training efficiency. Empirical evaluation on training the multi-billion parameters Falcon Models and Llama-2 models demonstrates the updated algorithm's ability to achieve reliable convergence on these massive models, where stock ZeRO++ hpZ fails to converge. The updated algorithm enables robust training of larger models with 98\% throughput and model training speed improvement without sacrificing the quality of convergence.
Taming Latency and Bandwidth: A Theoretical Framework and Adaptive Algorithm for Communication-Constrained Training
Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Tsinghua University(清华大学);Shenzhen Technology University(深圳科技大学);University of Science and Technology of China(中国科学技术大学)
Enhancing Lithological Mapping with Spatially Constrained Bayesian Network (SCB-Net): An Approach for Field Data-Constrained Predictions with Uncertainty Evaluation
Eau Terre Environnement (ETE)(水土地环境研究所);Institut National de la Recherche Scientifique (INRS)(国家科学研究院);Geological Survey of Canada (GSC)(加拿大地质调查局);Natural Resources Canada (NRCan)(加拿大自然资源部)
An Adaptable Budget Planner for Enhancing Budget-Constrained Auto-Bidding in Online Advertising
School of Computer Science, Peking University(北京大学计算机学院);Center on Frontiers of Computing Studies, Peking University(北京大学计算科学前沿中心);Alibaba Group(阿里巴巴集团)
Enhancing Materials Discovery with Valence Constrained Design in Generative Modeling
Quantum Measurement Group, MIT(麻省理工学院量子测量组);Department of Materials Science and Engineering, MIT(麻省理工学院材料科学与工程系);Department of Chemistry, MIT(麻省理工学院化学系);Neutron Scattering Division, Oak Ridge National Laboratory(橡树岭国家实验室中子散射部);Department of Chemistry, Michigan State University(密歇根州立大学化学系);Department of Nuclear Science and Engineering, MIT(麻省理工学院核科学与工程系)
Spectral Normalization for Lipschitz-Constrained Policies on Learning Humanoid Locomotion
Department of Intelligence and Information, Graduate School of Convergence Science and Technology, Seoul National University(智能信息系,融合科学与技术研究院,首尔国立大学);X Technologies(1X技术公司);Department of Intelligence and Information, Graduate School of Convergence Science and Technology, ASRI, AIIS, Seoul National University, Republic of Korea, and Advanced Institute of Convergence Technology (AICT), Suwon, Republic of Korea(智能信息系,融合科学与技术研究院,ASRI、AIIS、首尔国立大学,韩国,以及融合技术研究所(AICT),Suwon,韩国)