arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2108.02102cs.DC
收藏

ErrorCompensatedX:方差缩减算法的误差补偿

ErrorCompensatedX: error compensation for variance reduced algorithms

Hanlin Tang, Yao Li, Ji Liu, Ming Yan

摘要:

通信开销是分布式学习可扩展性的一个主要瓶颈。降低通信成本的一种方法是在通信期间压缩梯度。然而,直接压缩梯度会减慢收敛速度,并且所得到的算法可能因有偏压缩而发散。最近的工作解决了这个问题的随机梯度下降从前一步的压缩误差加回。这个想法被进一步扩展到一类方差减少算法,其中随机梯度的方差通过对所有历史梯度取移动平均来减少。然而,我们的分析表明,只是增加前一步的压缩误差,在现有的工作中所做的,并不能完全补偿压缩误差。因此,我们提出了ErrorCompensatedX,它使用前两步的压缩误差。我们证明了ErrorCompensatedX可以在没有压缩的情况下实现与训练相同的渐近收敛速度。此外,我们提供了一个统一的理论分析框架,这类方差减少算法,有或没有误差补偿。

英文摘要:

Communication cost is one major bottleneck for the scalability for distributed learning. One approach to reduce the communication cost is to compress the gradient during communication. However, directly compressing the gradient decelerates the convergence speed, and the resulting algorithm may diverge for biased compression. Recent work addressed this problem for stochastic gradient descent by adding back the compression error from the previous step. This idea was further extended to one class of variance reduced algorithms, where the variance of the stochastic gradient is reduced by taking a moving average over all history gradients. However, our analysis shows that just adding the previous step's compression error, as done in existing work, does not fully compensate the compression error. So, we propose ErrorCompensatedX, which uses the compression error from the previous two steps. We show that ErrorCompensatedX can achieve the same asymptotic convergence rate with the training without compression. Moreover, we provide a unified theoretical analysis framework for this class of variance reduced algorithms, with or without error compensation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑