Shuffle-Exchange带来更快：减少分散式神经网络训练通信期间的空闲时间
Shuffle-Exchange Brings Faster: Reduce the Idle Time During Communication for Decentralized Neural Network Training
摘要：
分布式随机梯度下降（DSGD）作为加速深度神经网络（DNN）训练的重要方法，在实际应用中得到了广泛的应用。在大多数分布式深度学习（DL）框架中，DSGD采用Ring-AllReduce架构（Ring-SGD）实现，并使用计算-通信重叠策略来解决DSGD所需的大量通信的开销。然而，我们观察到，尽管Ring-SGD中每个工作者都需要传递$O(1)$梯度，但Ring-SGD所需的$O(n)$握手限制了其在与许多工作者一起训练或在高延迟网络中的使用。本文提出了洗牌交换SGD（SESGD）来解决Ring-SGD的困境。在具有0.1ms以太网延迟的16个工作者的集群中，SESGD可以加速DNN训练到$1.7 \times$，而不会损失模型准确性。此外，在高延迟网络（5 ms）中，该过程可以加速到$5\times$。
英文摘要：
As a crucial scheme to accelerate the deep neural network (DNN) training, distributed stochastic gradient descent (DSGD) is widely adopted in many real-world applications. In most distributed deep learning (DL) frameworks, DSGD is implemented with Ring-AllReduce architecture (Ring-SGD) and uses a computation-communication overlap strategy to address the overhead of the massive communications required by DSGD. However, we observe that although $O(1)$ gradients are needed to be communicated per worker in Ring-SGD, the $O(n)$ handshakes required by Ring-SGD limits its usage when training with many workers or in high latency network. In this paper, we propose Shuffle-Exchange SGD (SESGD) to solve the dilemma of Ring-SGD. In the cluster of 16 workers with 0.1ms Ethernet latency, SESGD can accelerate the DNN training to $1.7 \times$ without losing model accuracy. Moreover, the process can be accelerated up to $5\times$ in high latency networks (5ms).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
