错误补偿方法的催化剂加速导致更好的通信复杂性
Catalyst Acceleration of Error Compensated Methods Leads to Better Communication Complexity
摘要：
众所周知，通信开销是大规模分布式学习中的一个关键瓶颈，并且有助于克服这个瓶颈的一类特别成功的方法是基于通信压缩的思想。一些最实际有效的梯度压缩器（诸如TopK）是有偏的，这导致收敛问题，除非采用良好设计的{ \em误差补偿/反馈}机制。因此，误差补偿是分布式学习文献中的基本技术。在最近的发展中，Qian等人（NeurIPS 2021）表明，误差补偿机制可以与加速/动量相结合，这是另一种关键且非常成功的优化技术。特别是，他们开发了误差补偿无环卡秋莎（ECLK）方法，并证明了在强凸情况下的加速线性速率。然而，它们的速率对压缩机参数的依赖性不匹配在非加速误差补偿方法中可获得的最佳依赖性。我们的工作解决了这个问题。我们提出了几个新的加速误差补偿方法，使用{ \em催化剂加速}技术，并获得的结果，匹配最佳的依赖于压缩机参数的非加速误差补偿方法的对数项。
英文摘要：
Communication overhead is well known to be a key bottleneck in large scale distributed learning, and a particularly successful class of methods which help to overcome this bottleneck is based on the idea of communication compression. Some of the most practically effective gradient compressors, such as TopK, are biased, which causes convergence issues unless one employs a well designed {\em error compensation/feedback} mechanism. Error compensation is therefore a fundamental technique in the distributed learning literature. In a recent development, Qian et al (NeurIPS 2021) showed that the error-compensation mechanism can be combined with acceleration/momentum, which is another key and highly successful optimization technique. In particular, they developed the error-compensated loop-less Katyusha (ECLK) method, and proved an accelerated linear rate in the strongly convex case. However, the dependence of their rate on the compressor parameter does not match the best dependence obtainable in the non-accelerated error-compensated methods. Our work addresses this problem. We propose several new accelerated error-compensated methods using the {\em catalyst acceleration} technique, and obtain results that match the best dependence on the compressor parameter in non-accelerated error-compensated methods up to logarithmic terms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
