arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1810.02565math.OCcs.LG
收藏

随机最优化算法的连续时间模型

Continuous-time Models for Stochastic Optimization Algorithms

ETH Zurich(苏黎世联邦理工学院)

Antonio Orvieto, Aurelien Lucchi

摘要:

我们提出了新的连续时间配方的一阶随机最优化算法,如小批量梯度下降和方差减少的方法。我们利用这些连续时间模型,连同简单的李雅普诺夫分析,以及从随机分析的工具,以获得各种类型的非凸函数的收敛范围。在这样的分析指导下,我们表明,相同的李雅普诺夫参数在离散时间,导致匹配率。此外,我们使用这些模型和伊藤演算来推断关于SGD动态的新见解,证明学习率的降低可以作为时间扭曲,或者等价地,作为景观拉伸。

英文摘要:

We propose new continuous-time formulations for first-order stochastic optimization algorithms such as mini-batch gradient descent and variance-reduced methods. We exploit these continuous-time models, together with simple Lyapunov analysis as well as tools from stochastic calculus, in order to derive convergence bounds for various types of non-convex functions. Guided by such analysis, we show that the same Lyapunov arguments hold in discrete-time, leading to matching rates. In addition, we use these models and Ito calculus to infer novel insights on the dynamics of SGD, proving that a decreasing learning rate acts as time warping or, equivalently, as landscape stretching.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑