arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2012.05396cs.DC
收藏

SSD-SSD:用于分布式深度学习训练的通信稀疏化

SSD-SSD: Communication sparsification for distributed deep learning training

Yemao Xu, Dezun Dong, Yawei Zhao, Weixia Xu, Xiangke Liao

摘要:

梯度和参数的密集通信和同步成本是分布式深度学习训练的众所周知的瓶颈。基于同步SGD(SSGD)获得良好的收敛精度而异步SGD(ASGD)提供更快的原始训练速度的观察,我们提出了几步延迟SGD(SSD-SGD),以结合它们的优点,旨在通过通信稀疏化来解决通信瓶颈。SSD-SGD在每个周期性迭代中探索参数服务器中的全局同步更新和工作器中的异步本地更新。周期性和灵活的同步使SSD-SGD具有良好的收敛精度和快速的训练速度。据我们所知,我们在同步质量和通信稀疏化之间取得了新的平衡,并提高了准确性和训练速度之间的权衡。具体来说,SSD-SGD的核心组件包括适当的预热阶段,步骤延迟阶段,和我们的新算法的全局梯度局部更新(GLU)。GLU对于局部更新操作是至关重要的,以有效地补偿延迟的局部权重。此外,我们在MXNet框架上实现了SSD-SGD,并使用CIFAR-10和ImageNet数据集对其性能进行了全面评估。实验结果表明,SSD-SGD在不同的实验配置下可以加快分布式训练速度,最高可达110%,同时实现良好的收敛精度。

英文摘要:

Intensive communication and synchronization cost for gradients and parameters is the well-known bottleneck of distributed deep learning training. Based on the observations that Synchronous SGD (SSGD) obtains good convergence accuracy while asynchronous SGD (ASGD) delivers a faster raw training speed, we propose Several Steps Delay SGD (SSD-SGD) to combine their merits, aiming at tackling the communication bottleneck via communication sparsification. SSD-SGD explores both global synchronous updates in the parameter servers and asynchronous local updates in the workers in each periodic iteration. The periodic and flexible synchronization makes SSD-SGD achieve good convergence accuracy and fast training speed. To the best of our knowledge, we strike the new balance between synchronization quality and communication sparsification, and improve the trade-off between accuracy and training speed. Specifically, the core components of SSD-SGD include proper warm-up stage, steps delay stage, and our novel algorithm of global gradient for local update (GLU). GLU is critical for local update operations to effectively compensate the delayed local weights. Furthermore, we implement SSD-SGD on MXNet framework and comprehensively evaluate its performance with CIFAR-10 and ImageNet datasets. Experimental results show that SSD-SGD can accelerate distributed training speed under different experimental configurations, by up to 110%, while achieving good convergence accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑