arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.03371cs.LGcs.AIcs.DC
收藏

采用解耦动量优化的低通信量分布式训练

Distributed Low-Communication Training with Decoupled Momentum Optimization

Sasho Nedelkoski, Alexander Acker, Odej Kao, Soeren Becker, Dominik Scheinert

AI总结:

针对大模型分布式训练对高带宽互连的依赖问题,提出结合低频同步与DCT分解Nesterov动量的方法,仅同步高频分量,通信量较DiLoCo最多降16倍,适配多架构,提升低带宽分布式训练大模型的可行性。

摘要:

大模型训练需要大量计算资源,通常只有配备高带宽互连网络的数据中心才能提供。然而,降低对节点间高带宽互连的依赖,可让分布式计算资源成为集中式数据中心训练的替代方案。基于分布式模型训练的最新进展,我们提出一种进一步降低通信量的方法,将分布式模型副本间的低频同步与梯度动量压缩相结合。具体而言,我们将优化器动量视为一种信号,通过离散余弦变换(DCT)将Nesterov动量分解为高频和低频分量。每H步仅在各模型副本间同步高频分量。实验结果表明,与基线方法DiLoCo相比,我们的方法通信量最多可降低16倍,且能在不同架构间通用,包括基于Transformer的语言模型和用于图像任务的卷积神经网络。总体而言,这项工作提升了在低带宽互连的分布式节点上训练大模型的可行性。

英文摘要:

The training of large models demands substantial computational resources, typically available only in data centers with high-bandwidth interconnects. However, reducing the reliance on high-bandwidth interconnects between nodes enables the use of distributed compute resources as an alternative to centralized data center training. Building on recent advances in distributed model training, we propose an approach that further reduces communication by combining infrequent synchronizations across distributed model replicas with gradient momentum compression. In particular, we treat the optimizer momentum as a signal and decompose the Nesterov momentum into high- and low-frequency components via the discrete cosine transform (DCT). Only the high-frequency components are synchronized across model replicas every $H$ steps. Empirically, our method achieves up to a $16\times$ reduction in communication compared to the baseline DiLoCo, and it generalizes across architectures, including transformer-based language models and convolutional neural networks for images. Overall, this work advances the feasibility of training large models on distributed nodes with low-bandwidth interconnects.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑