arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2110.03300cs.LGmath.OCstat.ML
收藏

可证快速分布非凸优化的置换压缩器

Permutation Compressors for Provably Faster Distributed Nonconvex Optimization

University of Warwick(华威大学);KAUST(阿卜杜拉国王科技大学)

Rafał Szlendak, Alexander Tyurin, Peter Richtárik

摘要:

我们研究了Gorbunov等人(2021)的MARINA方法--从理论通信复杂性的角度来看,这是当前最先进的分布式非凸优化方法。这种方法的理论优势可以在很大程度上归因于两个来源:使用一个精心设计的偏置随机梯度估计器,这导致了通信轮数的减少,以及依赖于{ \em独立}随机通信压缩算子,这导致了每个通信轮内的传输比特数的减少。在本文中,我们i)扩展的理论,以支持更广泛的一类潜在的{ \em相关}压缩机,扩展的方法超出了经典的独立压缩机设置的范围,ii)表明,一个新的数量,我们硬币的名称{ \em海森方差},使我们能够显着细化原始分析的MARINA没有任何额外的假设,以及iii)基于{ \em随机置换}的思想识别特殊类别的相关压缩器,为此我们创造了术语$K$,其使用导致$O(\sqrt{n})$(分别地,$O(1 + d/\sqrt{n})$)在低Hessian方差状态下MARINA的理论通信复杂性的改进,当$d\geq n$(分别为$d \leq n$),其中$n$是工作者的数量,$d$是描述我们正在学习的模型的参数的数量。我们证实了我们的理论结果与精心设计的合成实验,最大限度地减少非凸二次曲线的平均值,并与MNIST数据集的自动编码器训练。

英文摘要:

We study the MARINA method of Gorbunov et al (2021) -- the current state-of-the-art distributed non-convex optimization method in terms of theoretical communication complexity. Theoretical superiority of this method can be largely attributed to two sources: the use of a carefully engineered biased stochastic gradient estimator, which leads to a reduction in the number of communication rounds, and the reliance on {\em independent} stochastic communication compression operators, which leads to a reduction in the number of transmitted bits within each communication round. In this paper we i) extend the theory of MARINA to support a much wider class of potentially {\em correlated} compressors, extending the reach of the method beyond the classical independent compressors setting, ii) show that a new quantity, for which we coin the name {\em Hessian variance}, allows us to significantly refine the original analysis of MARINA without any additional assumptions, and iii) identify a special class of correlated compressors based on the idea of {\em random permutations}, for which we coin the term Perm$K$, the use of which leads to $O(\sqrt{n})$ (resp. $O(1 + d/\sqrt{n})$) improvement in the theoretical communication complexity of MARINA in the low Hessian variance regime when $d\geq n$ (resp. $d \leq n$), where $n$ is the number of workers and $d$ is the number of parameters describing the model we are learning. We corroborate our theoretical results with carefully engineered synthetic experiments with minimizing the average of nonconvex quadratics, and on autoencoder training with the MNIST dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑