arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.06570stat.MLcs.LGmath.OC
收藏

超越均匀光滑:自适应SGD的停止分析

Beyond Uniform Smoothness: A Stopped Analysis of Adaptive SGD

University of Texas at Austin(德克萨斯大学奥斯汀分校)

Matthew Faw, Litu Rout, Constantine Caramanis, Sanjay Shakkottai

摘要:

本文研究了利用随机梯度预言机寻找光滑常数潜在无界的非凸函数的一阶平稳点的问题。本文研究了Zhang等人(ICLR'20)提出的一类$(L_0,L_1)$ -光滑函数。经验证据表明,与普遍的$L_0$ -平滑相比,这些函数更接近地捕获实际的机器学习问题。这个类足够丰富,可以包含高度非平滑的函数,例如$\exp(L_1 x)$,它是$(0,\mathcal{O}(L_1))$ -平滑的。尽管丰富,一个新兴的线的作品实现了$\widetilde{\mathcal{O}}(\frac{1}{\sqrt{T}})$的收敛速度时,随机梯度的噪声是确定性和一致有界的。在$L_0$平滑设置中不需要这种噪声限制,并且在许多实际设置中,这种噪声限制要么不被满足,要么导致相对于收敛速率的噪声缩放的较弱收敛速率。我们开发了一种技术,使我们能够证明$\mathcal{O}(\frac{\mathrm{poly}\log(T)}{\sqrt{T}})$收敛速度$(L_0,L_1)$ -光滑的功能,而无需假设统一的边界上的噪声支持。我们的结果背后的关键创新是一个精心构造的停止时间$τ$,这是同时“大”的平均值,但也允许我们处理$τ$作为(大致)独立的梯度之前的自适应步长。对于一般的$(L_0,L_1)$ -光滑函数,我们的分析需要对乘性噪声参数$σ_1 < 1$进行适当的限制.对于$(L_0,L_1)$光滑函数的一个广泛的子类,当$σ_1 \geq 1$。通过比较,我们证明了许多算法分析了以前的作品$(L_0,L_1)$ -光滑优化发散的恒定概率,即使是光滑和强凸函数时,$σ_1 > 1$。

英文摘要:

This work considers the problem of finding a first-order stationary point of a non-convex function with potentially unbounded smoothness constant using a stochastic gradient oracle. We focus on the class of $(L_0,L_1)$-smooth functions proposed by Zhang et al. (ICLR'20). Empirical evidence suggests that these functions more closely captures practical machine learning problems as compared to the pervasive $L_0$-smoothness. This class is rich enough to include highly non-smooth functions, such as $\exp(L_1 x)$ which is $(0,\mathcal{O}(L_1))$-smooth. Despite the richness, an emerging line of works achieves the $\widetilde{\mathcal{O}}(\frac{1}{\sqrt{T}})$ rate of convergence when the noise of the stochastic gradients is deterministically and uniformly bounded. This noise restriction is not required in the $L_0$-smooth setting, and in many practical settings is either not satisfied, or results in weaker convergence rates with respect to the noise scaling of the convergence rate. We develop a technique that allows us to prove $\mathcal{O}(\frac{\mathrm{poly}\log(T)}{\sqrt{T}})$ convergence rates for $(L_0,L_1)$-smooth functions without assuming uniform bounds on the noise support. The key innovation behind our results is a carefully constructed stopping time $τ$ which is simultaneously "large" on average, yet also allows us to treat the adaptive step sizes before $τ$ as (roughly) independent of the gradients. For general $(L_0,L_1)$-smooth functions, our analysis requires the mild restriction that the multiplicative noise parameter $σ_1 < 1$. For a broad subclass of $(L_0,L_1)$-smooth functions, our convergence rate continues to hold when $σ_1 \geq 1$. By contrast, we prove that many algorithms analyzed by prior works on $(L_0,L_1)$-smooth optimization diverge with constant probability even for smooth and strongly-convex functions when $σ_1 > 1$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑