RedSync:减少分布式深度学习的同步流量
RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
摘要:
数据并行已成为跨多个节点扩展深度神经网络(DNN)训练的主要方法。由于局部模型的大量梯度同步可能是大规模分布式训练的瓶颈,因此压缩通信数据近年来得到了广泛的关注。在最近提出的几种压缩算法中,残差梯度压缩(RGC)是最成功的方法之一,它可以在保持正确精度和相同收敛速度的前提下,显著地压缩每个节点的发送消息大小(0.1%的梯度大小)。然而,关于压缩深度网络的文献几乎完全集中在实现良好的理论压缩率上,而RGC在实际分布式实现中的效率却很少被研究。在本文中,我们开发了一个基于RGC的系统,能够减少真实世界多GPU系统上的端到端训练时间。我们提出的设计称为RedSync,它引入了一组优化,以减少通信带宽的要求,同时引入有限的开销。我们在两个不同的多GPU平台上评估了RedSync的性能,包括128个GPU的超级计算机和8-GPU服务器。我们的测试用例包括Cifar 10和ImageNet上的图像分类任务,以及Penn Treebank和Wiki 2数据集上的语言建模任务。对于具有高通信计算比的DNN,长期以来被认为可扩展性差,RedSync带来了显着的性能改进。
英文摘要:
Data parallelism has become a dominant method to scale Deep Neural Network (DNN) training across multiple nodes. Since synchronizing a large number of gradients of the local model can be a bottleneck for large-scale distributed training, compressing communication data has gained widespread attention recently. Among several recent proposed compression algorithms, Residual Gradient Compression (RGC) is one of the most successful approaches---it can significantly compress the transmitting message size (0.1\% of the gradient size) of each node and still achieve correct accuracy and the same convergence speed. However, the literature on compressing deep networks focuses almost exclusively on achieving good theoretical compression rate, while the efficiency of RGC in real distributed implementation has been less investigated. In this paper, we develop an RGC-based system that is able to reduce the end-to-end training time on real-world multi-GPU systems. Our proposed design called RedSync, which introduces a set of optimizations to reduce communication bandwidth requirement while introducing limited overhead. We evaluate the performance of RedSync on two different multiple GPU platforms, including 128 GPUs of a supercomputer and an 8-GPU server. Our test cases include image classification tasks on Cifar10 and ImageNet, and language modeling tasks on Penn Treebank and Wiki2 datasets. For DNNs featured with high communication to computation ratio, which have long been considered with poor scalability, RedSync brings significant performance improvements.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。