原始和对偶块坐标下降方法中的通信避免
Avoiding communication in primal and dual block coordinate descent methods
摘要:
原始和对偶块坐标下降法是求解正则化和非正则化优化问题的迭代法。这些方法的分布式内存并行实现在分析大型机器学习数据集方面已经变得流行。然而,现有的实现在每次迭代时进行通信,在现代数据中心和超级计算架构上,这通常主导浮点计算的成本。最近的结果通信避免Krylov子空间方法表明,大的加速是可能的,通过重新组织迭代算法,以避免通信。我们展示了如何应用类似的算法变换可以导致原始和双块坐标下降方法,只通信每一个$s$迭代-其中$s$是一个调整参数-而不是每一个迭代的\textit{regularized least-squares problem}。我们发现,避免通信的变种减少了同步的数量的一个因素$s$分布式内存并行机不改变收敛速度,并达到强大的缩放加速高达$6.1\times$的克雷XC 30超级计算机。
英文摘要:
Primal and dual block coordinate descent methods are iterative methods for solving regularized and unregularized optimization problems. Distributed-memory parallel implementations of these methods have become popular in analyzing large machine learning datasets. However, existing implementations communicate at every iteration which, on modern data center and supercomputing architectures, often dominates the cost of floating-point computation. Recent results on communication-avoiding Krylov subspace methods suggest that large speedups are possible by re-organizing iterative algorithms to avoid communication. We show how applying similar algorithmic transformations can lead to primal and dual block coordinate descent methods that only communicate every $s$ iterations--where $s$ is a tuning parameter--instead of every iteration for the \textit{regularized least-squares problem}. We show that the communication-avoiding variants reduce the number of synchronizations by a factor of $s$ on distributed-memory parallel machines without altering the convergence rate and attains strong scaling speedups of up to $6.1\times$ on a Cray XC30 supercomputer.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。