arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1901.07445stat.MLcs.LGmath.OC
收藏

随机动量方法在瓦瑟斯坦距离上的加速线性收敛性

Accelerated Linear Convergence of Stochastic Momentum Methods in Wasserstein Distances

Rutgers Business School(罗格斯商学院);Florida State University(佛罗里达州立大学)

Bugra Can, Mert Gurbuzbalaban, Lingjiong Zhu

摘要:

在机器学习实践中常用的动量方法有Polyak的重球(HB)方法、Nesterov的加速梯度(AG)方法和加速投影梯度(APG)方法,但它们的性能对梯度中的噪声非常敏感。我们研究这些方法下的一阶随机预言模型的噪声估计的梯度。对于强凸问题,我们表明,分布的AG迭代收敛加速$O(\sqrtκ\log(1/\varepsilon))$线性速率的半径$\varepsilon$球集中在一个独特的不变分布在1-Wasserstein度量,其中$κ$是条件数,只要噪声方差小于一个明确的上限,我们可以提供。我们的分析还证明了线性收敛率作为步长,动量参数和噪声方差的函数;在无噪声的情况下恢复加速速率,并量化可以容忍的噪声水平,以实现给定的性能。在强凸二次目标的特殊情况下,我们可以通过对噪声结构的一些附加假设下的非渐近分析,显示出任何$p\geq 1$的$p$ -Wasserstein度量中的加速线性速率,并改善了AG和HB对噪声的敏感性。我们对HB和AG的分析也导致了确定性和随机性设置的次优性的非渐近收敛界的改进,这是独立的兴趣。据我们所知,这是随机预言模型下随机动量方法的第一个线性收敛结果。我们还将我们的结果扩展到APG方法和弱凸函数,当噪声幅度足够小时,显示加速率。

英文摘要:

Momentum methods such as Polyak's heavy ball (HB) method, Nesterov's accelerated gradient (AG) as well as accelerated projected gradient (APG) method have been commonly used in machine learning practice, but their performance is quite sensitive to noise in the gradients. We study these methods under a first-order stochastic oracle model where noisy estimates of the gradients are available. For strongly convex problems, we show that the distribution of the iterates of AG converges with the accelerated $O(\sqrtκ\log(1/\varepsilon))$ linear rate to a ball of radius $\varepsilon$ centered at a unique invariant distribution in the 1-Wasserstein metric where $κ$ is the condition number as long as the noise variance is smaller than an explicit upper bound we can provide. Our analysis also certifies linear convergence rates as a function of the stepsize, momentum parameter and the noise variance; recovering the accelerated rates in the noiseless case and quantifying the level of noise that can be tolerated to achieve a given performance. In the special case of strongly convex quadratic objectives, we can show accelerated linear rates in the $p$-Wasserstein metric for any $p\geq 1$ with improved sensitivity to noise for both AG and HB through a non-asymptotic analysis under some additional assumptions on the noise structure. Our analysis for HB and AG also leads to improved non-asymptotic convergence bounds in suboptimality for both deterministic and stochastic settings which is of independent interest. To the best of our knowledge, these are the first linear convergence results for stochastic momentum methods under the stochastic oracle model. We also extend our results to the APG method and weakly convex functions showing accelerated rates when the noise magnitude is sufficiently small.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑