arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2008.12690math.OCcs.LGstat.ML
收藏

ROOT-SGD: 单一算法中的精确非渐近性与近最优渐近性

ROOT-SGD: Sharp Nonasymptotics and Near-Optimal Asymptotics in a Single Algorithm

UC Berkeley(加州大学伯克利分校)

Chris Junchi Li, Wenlong Mou, Martin J. Wainwright, Michael I. Jordan

AI总结:

本研究提出基于递归平均随机梯度的ROOT-SGD算法,可同时在非渐近和渐近层面达到强凸光滑无约束优化的最优性能,兼具精确有限样本风险界与最优渐近协方差特性。

摘要:

我们研究了使用随机一阶算法求解强凸且光滑的无约束优化问题的任务。我们设计了一种名为Recursive One-Over-T SGD(递归1/T型随机梯度下降,简称ROOT-SGD)的全新算法,其核心是对过往随机梯度进行易于实现的递归平均操作。我们证明该算法能同时在有限样本的非渐近层面和渐近层面达到当前最优性能。在非渐近方面,我们证明了ROOT-SGD最后一次迭代的风险界:其主阶项与最优统计风险完全匹配(前置因子为1),同时高阶项在海森矩阵满足利普希茨条件下,以$O(n^{-3/2})$的精确速率收敛。在渐近方面,我们表明当施加温和的单点海森连续性条件时,(多轮)ROOT-SGD的缩放后最后一次迭代会渐近收敛到具有Cramér-Rao最优渐近协方差的高斯极限,且该结论适用于广泛的步长选择范围。

英文摘要:

We study the problem of solving strongly convex and smooth unconstrained optimization problems using stochastic first-order algorithms. We devise a novel algorithm, referred to as Recursive One-Over-T SGD (ROOT-SGD), based on an easily implementable, recursive averaging of past stochastic gradients. We prove that it simultaneously achieves state-of-the-art performance in both a finite-sample, nonasymptotic sense and an asymptotic sense. On the non-asymptotic side, we prove risk bounds on the last iterate of ROOT-SGD with leading-order terms that match the optimal statistical risk with a unity pre-factor, along with a higher-order term that scales at the sharp rate of $O(n^{-3/2})$ under the Lipschitz condition on the Hessian matrix. On the asymptotic side, we show that when a mild, one-point Hessian continuity condition is imposed, the rescaled last iterate of (multi-epoch) ROOT-SGD converges asymptotically to a Gaussian limit with the Cramér-Rao optimal asymptotic covariance, for a broad range of step-size choices.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑