The training of large models demands substantial computational resources, typically available only in data centers with high-bandwidth interconnects. However, reducing the reliance on high-bandwidth interconnects between nodes enables the use of distributed compute resources as an alternative to centralized data center training. Building on recent advances in distributed model training, we propose an approach that further reduces communication by combining infrequent synchronizations across distributed model replicas with gradient momentum compression. In particular, we treat the optimizer momentum as a signal and decompose the Nesterov momentum into high- and low-frequency components via the discrete cosine transform (DCT). Only the high-frequency components are synchronized across model replicas every $H$ steps. Empirically, our method achieves up to a $16\times$ reduction in communication compared to the baseline DiLoCo, and it generalizes across architectures, including transformer-based language models and convolutional neural networks for images. Overall, this work advances the feasibility of training large models on distributed nodes with low-bandwidth interconnects.
Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent
Anhui Province Key Laboratory of Big Data Analysis and Application(安徽省大数据分析与应用重点实验室);School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);School of Data Science, University of Science and Technology of China(中国科学技术大学数据科学学院)
随着数据量的增加和模型规模的扩大,分布式并行训练成为解决优化挑战的重要和成功的技术。然而,尽管分布式随机梯度下降(SGD)算法可以实现线性迭代加速,但它们在实践中受到通信成本的显著限制,使得难以实现线性时间加速。在本文中,我们提出了一个计算和通信解耦的随机梯度下降(CoCoD-SGD)算法,并行运行的计算和通信,以减少通信成本。我们证明了CoCoD-SGD相对于硬件资源的总计算能力具有线性迭代加速比。与传统的分布式SGD算法相比,该算法具有较低的通信复杂性和较好的时间加速比。深度神经网络训练的实验证明了CoCoD-SGD的显著改进:在使用16个Geforce GTX 1080 Ti GPU训练ResNet 18和VGG 16时,CoCoD-SGD比传统同步SGD快2-3 $\times$。
Byzantine-Robust and Communication-Efficient Distributed Learning via Compressed Momentum Filtering
Digital Futures(数字未来机构);College of Computer Science, Sichuan University(四川大学计算机学院);Institute of Clinical Pathology, West China Hospital, Sichuan University(四川大学华西医院临床病理研究所)
Distributed Deep Learning using Stochastic Gradient Staleness
Information Technology Department, Posts and Telecommunications Institute of Technology (PTIT)(信息科技部,邮电技术研究所);Department of Mechanical and Robotics Engineering, Gwangju Institute of Science and Technology(机械与机器人工程系,全州科学技术院)
Distributed Stochastic Consensus Optimization with Momentum for Nonconvex Nonsmooth Problems
Sichuan University(四川大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳市大数据研究院);University of Florida(佛罗里达大学)
School of Computing, Montclair State University(蒙特克莱尔州立大学计算机学院);Department of Computer Science and Engineering, Southern University of Science and Technology(南方科技大学计算机科学与工程系)
Local adapt-then-combine algorithms for distributed nonsmooth optimization: Achieving provable communication acceleration
School of Computer Science and Engineering, Suzhou University of Technology(苏州科技大学计算机科学与工程学院);School of Mathematics, Southeast University(东南大学数学学院);School of Cyber Science and Engineering, Southeast University(东南大学网络科学与工程学院)
Parallel Momentum Methods Under Biased Gradient Estimations
National Institute of Standards and Technology(美国国家标准与技术研究院);Rice University(莱斯大学);Colorado State University(科罗拉多州立大学);University of Colorado(科罗拉多大学);National Research Institute for Metals(国立金属材料研究所)
Accelerating Wireless Federated Learning via Nesterov's Momentum and Distributed Principle Component Analysis
Shenzhen MSU-BIT University(深圳北理莫斯科大学);Shenzhen University(深圳大学);University of Science and Technology Beijing(北京科技大学);Carleton University(卡尔顿大学);The University of British Columbia(不列颠哥伦比亚大学)