利用动量补偿加速凸优化的异步算法
Accelerating Asynchronous Algorithms for Convex Optimization by Momentum Compensation
摘要：
由于求解大规模最优化问题的迫切需求，异步算法近年来受到了广泛的关注。然而，异步算法的加速版本很少被研究。在本文中，我们提出了“动量补偿”技术来加速凸问题的异步算法。具体来说，我们首先加速普通的异步梯度下降，这实现了更快的$O(1/\sqrtε)$（与$O(1/ε)$）收敛速度，以及$O(\sqrtκ\log(1/ε))$（v.s. $O(κ\log(1/ε))$），使强凸函数达到条件数为$κ$的$ε$ -近似极小元.我们进一步应用该技术来加速现代随机异步算法，如异步随机坐标下降和异步随机梯度下降。这两种实用算法都比现有的算法快。据我们所知，我们是第一个考虑加速算法，允许更新延迟梯度，是第一个提出真正的加速异步算法。最后，在共享内存系统上的实验结果表明，加速可以导致显着的性能增益病态问题。
英文摘要：
Asynchronous algorithms have attracted much attention recently due to the crucial demands on solving large-scale optimization problems. However, the accelerated versions of asynchronous algorithms are rarely studied. In this paper, we propose the "momentum compensation" technique to accelerate asynchronous algorithms for convex problems. Specifically, we first accelerate the plain Asynchronous Gradient Descent, which achieves a faster $O(1/\sqrtε)$ (v.s. $O(1/ε)$) convergence rate for non-strongly convex functions, and $O(\sqrtκ\log(1/ε))$ (v.s. $O(κ\log(1/ε))$) for strongly convex functions to reach an $ε$- approximate minimizer with the condition number $κ$. We further apply the technique to accelerate modern stochastic asynchronous algorithms such as Asynchronous Stochastic Coordinate Descent and Asynchronous Stochastic Gradient Descent. Both of the resultant practical algorithms are faster than existing ones by order. To the best of our knowledge, we are the first to consider accelerated algorithms that allow updating by delayed gradients and are the first to propose truly accelerated asynchronous algorithms. Finally, the experimental results on a shared memory system show that acceleration can lead to significant performance gains on ill-conditioned problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
