arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1806.06573math.OCstat.ML
收藏

基于压缩梯度的分布式学习

Distributed learning with compressed gradients

Sarit Khirirat, Hamid Reza Feyzmahdavian, Mikael Johansson

摘要:

异步计算和梯度压缩已经成为大规模机器学习分布式优化中实现可扩展性的两项关键技术。本文提出了一个统一的分析框架,分布式梯度法运行与静态和压缩梯度。对于几种优化算法,推导了收敛速度和信息交换的非渐近界。这些边界给出了步长的显式表达式,并描述了压缩量和压缩精度如何影响迭代和通信复杂性保证。数值结果突出了不同的梯度压缩算法的收敛特性,并确认有限的信息交换下的快速收敛确实是可能的。

英文摘要:

Asynchronous computation and gradient compression have emerged as two key techniques for achieving scalability in distributed optimization for large-scale machine learning. This paper presents a unified analysis framework for distributed gradient methods operating with staled and compressed gradients. Non-asymptotic bounds on convergence rates and information exchange are derived for several optimization algorithms. These bounds give explicit expressions for step-sizes and characterize how the amount of asynchrony and the compression accuracy affect iteration and communication complexity guarantees. Numerical results highlight convergence properties of different gradient compression algorithms and confirm that fast convergence under limited information exchange is indeed possible.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑