arXiv 2005.106752020-05-20math.OCcs.DC

分散有限和Optimization的一个优化算法

An Optimal Algorithm for Decentralized Finite Sum Optimization

Hadrien Hendrikx, Francis Bach, Laurent Massoulie

备注：: arXiv admin note: substantial text overlap with arXiv:1905.11394

摘要：

现代大规模有限和优化依赖于两个关键方面：分布和随机更新。对于光滑和强凸问题，现有的分散算法在单机上运行时比现代加速方差缩减随机算法慢，因此效率不高。集中式算法速度很快，但它们的扩展受到全局聚合步骤的限制，这会导致通信瓶颈。在这项工作中，我们提出了一个有效的\textbf{A}加速\textbf{D} ecentralized随机算法的\textbf{F}有限\textbf{S}称为ADFS，它使用本地随机邻近更新和节点之间的分散通信。在$n$计算机上，ADFS使用$nm$样本最小化目标函数，同时使用最优算法在一台计算机上从$m$样本进行优化。这种缩放保持直到达到临界网络大小，这取决于通信延迟、样本$m$的数量以及网络拓扑。我们给出了一个下界的复杂性，表明ADFS是分散算法中的最佳。为了获得ADFS，我们首先开发了一个扩展的加速邻近坐标梯度算法，以任意采样。然后，我们将这种坐标下降算法应用到一个基于增广图方法的精心选择的对偶问题，从而得到一般的ADFS算法。我们用实验说明了ADFS对最先进的分散式方法的改进。

英文摘要：

Modern large-scale finite-sum optimization relies on two key aspects: distribution and stochastic updates. For smooth and strongly convex problems, existing decentralized algorithms are slower than modern accelerated variance-reduced stochastic algorithms when run on a single machine, and are therefore not efficient. Centralized algorithms are fast, but their scaling is limited by global aggregation steps that result in communication bottlenecks. In this work, we propose an efficient \textbf{A}ccelerated \textbf{D}ecentralized stochastic algorithm for \textbf{F}inite \textbf{S}ums named ADFS, which uses local stochastic proximal updates and decentralized communications between nodes. On $n$ machines, ADFS minimizes the objective function with $nm$ samples in the same time it takes optimal algorithms to optimize from $m$ samples on one machine. This scaling holds until a critical network size is reached, which depends on communication delays, on the number of samples $m$, and on the network topology. We give a lower bound of complexity to show that ADFS is optimal among decentralized algorithms. To derive ADFS, we first develop an extension of the accelerated proximal coordinate gradient algorithm to arbitrary sampling. Then, we apply this coordinate descent algorithm to a well-chosen dual problem based on an augmented graph approach, leading to the general ADFS algorithm. We illustrate the improvement of ADFS over state-of-the-art decentralized approaches with experiments.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

摘要：

英文摘要：

相关论文