arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.02478cs.ITcs.DCcs.LGeess.SPmath.IT
收藏

分布式学习中用于梯度压缩的时间预测编码

Temporal Predictive Coding for Gradient Compression in Distributed Learning

Université Paris-Saclay, CentraleSupelec, CNRS(巴黎-萨克雷大学、中央高等电力学院、法国国家科学研究中心);Linköping University(林雪平大学);KTH, Stockholm(斯德哥尔摩皇家理工学院)

Adrian Edin, Zheng Chen, Michel Kieffer, Mikael Johansson

AI总结:

本文提出一种基于时间预测编码的梯度压缩方法,通过最小二乘优化的线性预测器和事件触发残差传输机制,在分布式学习中显著降低通信成本并保持收敛性能。

摘要:

本文提出了一种基于预测的梯度压缩方法,用于具有事件触发通信的分布式学习。我们的目标是通过利用局部梯度中的时间相关性来减少从分布式智能体传输到参数服务器的信息量。我们使用一种线性预测器,该预测器结合过去的梯度来形成对当前梯度的预测,其系数通过求解最小二乘问题来优化。在每次迭代中,每个智能体将预测器系数传输到服务器,以便可以计算预测的局部梯度。真实局部梯度与预测梯度之间的差异,称为预测残差,仅当其范数高于某个阈值时才被传输。当省略这一额外通信步骤时,服务器使用预测值作为估计梯度。与文献中的现有方法相比,该设计显示出显著的性能提升,在降低通信成本的同时实现了收敛。

英文摘要:

This paper proposes a prediction-based gradient compression method for distributed learning with event-triggered communication. Our goal is to reduce the amount of information transmitted from the distributed agents to the parameter server by exploiting temporal correlation in the local gradients. We use a linear predictor that \textit{combines past gradients to form a prediction of the current gradient}, with coefficients that are optimized by solving a least-square problem. In each iteration, every agent transmits the predictor coefficients to the server such that the predicted local gradient can be computed. The difference between the true local gradient and the predicted one, termed the \textit{prediction residual, is only transmitted when its norm is above some threshold.} When this additional communication step is omitted, the server uses the prediction as the estimated gradient. This proposed design shows notable performance gains compared to existing methods in the literature, achieving convergence with reduced communication costs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑