arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1412.0159math.OCcs.GT
收藏

异步梯度下降的摊销分析

Amortized Analysis on Asynchronous Gradient Descent

Yun Kuen Cheung, Richard Cole

摘要:

梯度下降法是求解凸函数极小化问题的一类重要迭代算法。传统上,梯度下降是一个顺序和同步的过程。自20世纪80年代以来,人们一直在研究梯度下降的分布式和异步变体,并且由于在多核处理器上运行的大规模机器学习问题的需求,它们一直在经历复兴。我们提供了一个版本的异步梯度下降(AGD),其中核心之间的通信是最小的,并且几乎没有同步开销。我们还提出了一个新的定时模型进行分析。在此基础上,首次给出了凸函数上的AGD的摊销分析。摊销允许坏的更新(增加凸函数的值的更新);相反,大多数先前的工作都强烈假设每次更新都必须显着改善。通常,AGD中使用的步长小于其同步对应物中使用的步长。本文提出了一种基于凸函数的Hessian元素确定AGD步长的方法。在某些情况下,所得到的步长是在相应的同步算法中使用的步长的恒定分数,使得AGD的总体性能能够随着核的数量线性地提高。我们给出了我们的摊销分析的两个应用。

英文摘要:

Gradient descent is an important class of iterative algorithms for minimizing convex functions. Classically, gradient descent has been a sequential and synchronous process. Distributed and asynchronous variants of gradient descent have been studied since the 1980s, and they have been experiencing a resurgence due to demand from large-scale machine learning problems running on multi-core processors. We provide a version of asynchronous gradient descent (AGD) in which communication between cores is minimal and for which there is little synchronization overhead. We also propose a new timing model for its analysis. With this model, we give the first amortized analysis of AGD on convex functions. The amortization allows for bad updates (updates that increase the value of the convex function); in contrast, most prior work makes the strong assumption that every update must be significantly improving. Typically, the step sizes used in AGD are smaller than those used in its synchronous counterpart. We provide a method to determine the step sizes in AGD based on the Hessian entries for the convex function. In certain circumstances, the resulting step sizes are a constant fraction of those used in the corresponding synchronous algorithm, enabling the overall performance of AGD to improve linearly with the number of cores. We give two applications of our amortized analysis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑