用于深度学习训练加速的全局绘制分布式SGD算法
A Distributed SGD Algorithm with Global Sketching for Deep Learning Training Acceleration
摘要：
分布式训练是加速大规模深度学习模型训练过程的有效方法。然而，分布式随机梯度下降的参数交换和同步引入了大量的通信开销。梯度压缩是降低通信开销的有效方法。在同步SGD压缩方法中，已经提出了许多基于Top-k稀疏化的梯度压缩方法以减少通信。然而，基于参数服务器的集中式方法存在单点故障问题，可扩展性有限，而全局参数交换的分散式方法可能会降低训练的收敛速度。与基于$k$的方法相比，本文提出了一种基于全局梯度矢量草图的梯度压缩方法--全局草图SGD（global-sketching SGD，gs-SGD）。gs-SGD在深度学习模型上具有更好的收敛效率，通信复杂性为O（$\log d*\log P$），其中$d$是模型参数的数量，P是工人的数量。在GPU集群上进行了实验，验证了该方法比全局$k$和基于Sketching的方法具有更好的收敛效率。此外，与gTop- $k$相比，gs-SGD的吞吐量高出1.3- 3.1倍，与原始Sketched-SGD相比，吞吐量高出1.1- 1.2倍。
英文摘要：
Distributed training is an effective way to accelerate the training process of large-scale deep learning models. However, the parameter exchange and synchronization of distributed stochastic gradient descent introduce a large amount of communication overhead. Gradient compression is an effective method to reduce communication overhead. In synchronization SGD compression methods, many Top-k sparsification based gradient compression methods have been proposed to reduce the communication. However, the centralized method based on the parameter servers has the single point of failure problem and limited scalability, while the decentralized method with global parameter exchanging may reduce the convergence rate of training. In contrast with Top-$k$ based methods, we proposed a gradient compression method with globe gradient vector sketching, which uses the Count-Sketch structure to store the gradients to reduce the loss of the accuracy in the training process, named global-sketching SGD (gs-SGD). The gs-SGD has better convergence efficiency on deep learning models and a communication complexity of O($\log d*\log P$), where $d$ is the number of model parameters and P is the number of workers. We conducted experiments on GPU clusters to verify that our method has better convergence efficiency than global Top-$k$ and Sketching-based methods. In addition, gs-SGD achieves 1.3-3.1x higher throughput compared with gTop-$k$, and 1.1-1.2x higher throughput compared with original Sketched-SGD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
