arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1811.12403math.OCcs.LG
收藏

随机梯度算法的新收敛性

New Convergence Aspects of Stochastic Gradient Algorithms

IBM Research(IBM研究院);University of Connecticut(康涅狄格大学);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);Cornell University(康奈尔大学);Lehigh University(利哈伊大学)

Lam M. Nguyen, Phuong Ha Nguyen, Peter Richtárik, Katya Scheinberg, Martin Takáč, Marten van Dijk

摘要:

在随机梯度范数一致有界的假设下,对SGD算法进行了经典的收敛性分析。虽然这可能适用于某些损失函数,但对于目标函数是强凸的情况,这是违反的。在Bottou等人(2018)中,在随机梯度相对于真实梯度范数有界的假设下,对SGD的收敛性进行了新的分析。我们表明,在机器学习中出现的随机问题,这样的界始终保持不变,我们还提出了一种替代的收敛性分析SGD与递减的学习率制度。然后,我们继续进行异步并行设置,并证明Hogwild的收敛性!算法在相同的制度的情况下减少学习率。众所周知,如果学习率序列$\{η_t\}$满足$\sum_{t=0}^\infty η_t \rightarrow \infty$和$\sum_{t=0}^\infty η^2_t < \infty$,则SGD收敛。当$\{η_t\}$为递减序列且$\sum_{t=0}^\infty η_t \rightarrow \infty$为有界梯度假设时,证明了强凸目标函数SGD的收敛性。换句话说,我们扩展了当前最先进的一类学习率,满足SGD的收敛性。

英文摘要:

The classical convergence analysis of SGD is carried out under the assumption that the norm of the stochastic gradient is uniformly bounded. While this might hold for some loss functions, it is violated for cases where the objective function is strongly convex. In Bottou et al. (2018), a new analysis of convergence of SGD is performed under the assumption that stochastic gradients are bounded with respect to the true gradient norm. We show that for stochastic problems arising in machine learning such bound always holds; and we also propose an alternative convergence analysis of SGD with diminishing learning rate regime. We then move on to the asynchronous parallel setting, and prove convergence of Hogwild! algorithm in the same regime in the case of diminished learning rate. It is well-known that SGD converges if a sequence of learning rates $\{η_t\}$ satisfies $\sum_{t=0}^\infty η_t \rightarrow \infty$ and $\sum_{t=0}^\infty η^2_t < \infty$. We show the convergence of SGD for strongly convex objective function without using bounded gradient assumption when $\{η_t\}$ is a diminishing sequence and $\sum_{t=0}^\infty η_t \rightarrow \infty$. In other words, we extend the current state-of-the-art class of learning rates satisfying the convergence of SGD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑