ProxSkip：是的！局部梯度步骤可证明导致通信加速！终于来了！
ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!
摘要：
我们介绍ProxSkip -一个令人惊讶的简单和证明有效的方法，用于最小化一个光滑（$f$）和一个昂贵的非光滑可逼近（$ψ$）函数的总和。解决此类问题的规范方法是通过近端梯度下降（ProxGD）算法，该算法基于每次迭代中对$f$的梯度和$ψ$的prox算子的评估。在这项工作中，我们特别感兴趣的制度中，评估的prox是昂贵的相对于评估的梯度，这是在许多应用程序中的情况。ProxSkip允许在大多数迭代中跳过昂贵的prox运算符：虽然其迭代复杂度为$\mathcal{O}\left(κ\log \frac{1}{\varepsilon}\right)$，其中$κ$是$f$的条件数，但prox求值的数量仅为$\mathcal{O}\left(\sqrtκ \log \frac{1}{\varepsilon}\right)$。我们的主要动机来自于联邦学习，其中梯度算子的评估对应于在所有设备上独立地采取局部GD步骤，而prox的评估对应于梯度平均形式的（昂贵的）通信。在这种情况下，ProxSkip提供了一个有效的加速通信复杂性。不像其他本地梯度型方法，如FedAvg，SCAFFOLD，S-Local-GD和FedLin，其理论上的通信复杂性比差，或在最佳匹配，香草GD在异构数据制度，我们得到了一个可证明的和大的改进，没有任何异质性约束的假设。
英文摘要：
We introduce ProxSkip -- a surprisingly simple and provably efficient method for minimizing the sum of a smooth ($f$) and an expensive nonsmooth proximable ($ψ$) function. The canonical approach to solving such problems is via the proximal gradient descent (ProxGD) algorithm, which is based on the evaluation of the gradient of $f$ and the prox operator of $ψ$ in each iteration. In this work we are specifically interested in the regime in which the evaluation of prox is costly relative to the evaluation of the gradient, which is the case in many applications. ProxSkip allows for the expensive prox operator to be skipped in most iterations: while its iteration complexity is $\mathcal{O}\left(κ\log \frac{1}{\varepsilon}\right)$, where $κ$ is the condition number of $f$, the number of prox evaluations is $\mathcal{O}\left(\sqrtκ \log \frac{1}{\varepsilon}\right)$ only. Our main motivation comes from federated learning, where evaluation of the gradient operator corresponds to taking a local GD step independently on all devices, and evaluation of prox corresponds to (expensive) communication in the form of gradient averaging. In this context, ProxSkip offers an effective acceleration of communication complexity. Unlike other local gradient-type methods, such as FedAvg, SCAFFOLD, S-Local-GD and FedLin, whose theoretical communication complexity is worse than, or at best matching, that of vanilla GD in the heterogeneous data regime, we obtain a provable and large improvement without any heterogeneity-bounding assumptions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
