arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1806.08054cs.CVcs.DC
收藏

误差补偿量化SGD及其在大规模分布式优化中的应用

Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization

Tencent AI Lab(腾讯人工智能实验室)

Jiaxiang Wu, Weidong Huang, Junzhou Huang, Tong Zhang

摘要:

大规模分布式优化在各种应用中具有重要意义。对于基于数据并行的分布式学习,节点间的梯度通信往往成为性能瓶颈。在本文中,我们提出了误差补偿量化随机梯度下降算法,以提高训练效率。局部梯度被量化以减少通信开销,并且累积量化误差被利用以加速收敛。此外,我们提出了理论分析的收敛行为,并证明了它的优势,竞争对手。大量的实验表明,我们的算法可以压缩梯度的一个因素高达两个数量级,而不会降低性能。

英文摘要:

Large-scale distributed optimization is of great importance in various applications. For data-parallel based distributed learning, the inter-node gradient communication often becomes the performance bottleneck. In this paper, we propose the error compensated quantized stochastic gradient descent algorithm to improve the training efficiency. Local gradients are quantized to reduce the communication overhead, and accumulated quantization error is utilized to speed up the convergence. Furthermore, we present theoretical analysis on the convergence behaviour, and demonstrate its advantage over competitors. Extensive experiments indicate that our algorithm can compress gradients by a factor of up to two magnitudes without performance degradation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑