arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1607.01027math.OCcs.LGcs.NAmath.NAstat.ML
收藏

通过利用局部增长条件加速随机子梯度方法

Accelerate Stochastic Subgradient Method by Leveraging Local Growth Condition

Department of Computer Science(计算机科学系);Department of Management Sciences(管理科学系);The University of Iowa(爱荷华大学)

Yi Xu, Qihang Lin, Tianbao Yang

AI总结:

本文提出了一种新的理论,表明在最优解邻域内目标函数的局部增长率足以量化一阶随机凸优化的全局收敛率,通过局部区域逐步缩小的方法改进了加速随机子梯度方法的收敛性,并在实践中提出了无需知道乘法增长常数和增长率的实用变体。

摘要:

在本文中,我们为一阶随机凸优化开发了一种新理论,表明全局收敛率足以由最优解邻域内目标函数的局部增长率量化。具体而言,如果目标函数F(w)在ε子水平集内以速度‖w - w*‖_2^{1/θ}增长,其中w*是w最近的最优解,θ∈(0,1]表示局部增长率,则达到ε最优解的一阶随机优化迭代复杂度可以为~O(1/ε^{2(1-θ)}),这在至多对数因子范围内是最佳的。为了实现更快的全局收敛,我们通过在历史解的局部区域中迭代求解原始问题,开发了两种不同的加速随机子梯度方法,该局部区域的大小随着解接近最优集而逐渐减小。除了理论改进外,这项工作还包含了使所提算法实用的新贡献:(i) 我们提出了可以运行而无需知道乘法增长常数和增长率θ的加速随机子梯度方法的实用变体;(ii) 我们考虑了机器学习中的广泛问题集,以证明所提算法比传统随机子梯度方法具有更快的收敛速度。我们还表征了所提算法的复杂性,以确保在不假设光滑性的情况下梯度较小。

英文摘要:

In this paper, a new theory is developed for first-order stochastic convex optimization, showing that the global convergence rate is sufficiently quantified by a local growth rate of the objective function in a neighborhood of the optimal solutions. In particular, if the objective function $F(\mathbf w)$ in the $ε$-sublevel set grows as fast as $\|\mathbf w - \mathbf w_*\|_2^{1/θ}$, where $\mathbf w_*$ represents the closest optimal solution to $\mathbf w$ and $θ\in(0,1]$ quantifies the local growth rate, the iteration complexity of first-order stochastic optimization for achieving an $ε$-optimal solution can be $\widetilde O(1/ε^{2(1-θ)})$, which is optimal at most up to a logarithmic factor. To achieve the faster global convergence, we develop two different accelerated stochastic subgradient methods by iteratively solving the original problem approximately in a local region around a historical solution with the size of the local region gradually decreasing as the solution approaches the optimal set. Besides the theoretical improvements, this work also includes new contributions towards making the proposed algorithms practical: (i) we present practical variants of accelerated stochastic subgradient methods that can run without the knowledge of multiplicative growth constant and even the growth rate $θ$; (ii) we consider a broad family of problems in machine learning to demonstrate that the proposed algorithms enjoy faster convergence than traditional stochastic subgradient method. We also characterize the complexity of the proposed algorithms for ensuring the gradient is small without the smoothness assumption.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑