arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.18784cs.LGmath.OCmath.STstat.MLstat.TH
收藏

重尾噪声下非线性随机梯度下降的高概率收敛界

High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise

Carnegie Mellon University(卡内基梅隆大学);Faculty of Technical Sciences, University of Novi Sad(诺维萨德大学技术科学学院);Faculty of Sciences, University of Novi Sad(诺维萨德大学理学院)

Aleksandar Armacki, Pranay Sharma, Gauri Joshi, Dragana Bajovic, Dusan Jakovetic, Soummya Kar

AI总结:

本文针对重尾噪声下的流式在线学习,提出非线性SGD一般框架,在非凸和强凸情形下分别建立指数与噪声无关的高概率收敛界,并证明裁剪并非最优非线性选择。

摘要:

我们研究在存在重尾噪声的情况下流式数据学习的高概率收敛保证。在所提出的场景中,模型以在线方式更新,当观察到新信息时即进行更新,而不存储任何额外数据。为了对抗重尾噪声,我们考虑一个非线性随机梯度下降(SGD)的一般框架,并给出若干强结果。首先,对于非凸代价和逐分量非线性,我们建立了任意接近 $\mathcal{O}\left(t^{-\frac{1}{4}}\right)$ 的收敛速率,其指数与噪声和问题参数无关。其次,对于强凸代价和逐分量非线性,我们对迭代的加权平均建立了任意接近 $\mathcal{O}\left(t^{-\frac{1}{2}}\right)$ 的速率,其指数同样与噪声和问题参数无关。最后,对于强凸代价和更广泛的非线性类,我们建立了最后迭代的收敛性,速率为 $\mathcal{O}\left(t^{-ζ} \right)$,其中 $ζ\in (0,1)$ 依赖于问题参数、噪声和非线性。正如我们通过分析和数值所展示的,$ζ$ 可用于为给定问题设置提供优选非线性的依据。与仅考虑裁剪、要求阶数 $η\in (1,2]$ 的有界噪声矩、并且其收敛速率指数在 $η\rightarrow 1$ 时趋于零的最新方法相比,我们为更广泛的非线性类和对称密度噪声提供了高概率保证,其收敛速率指数远离零,即使噪声仅具有有限一阶矩。此外,在强凸函数情形下,我们通过分析和数值证明裁剪并不总是最优的非线性,进一步凸显了我们一般框架的价值。

英文摘要:

We study high-probability convergence guarantees of learning on streaming data in the presence of heavy-tailed noise. In the proposed scenario, the model is updated in an online fashion, as new information is observed, without storing any additional data. To combat the heavy-tailed noise, we consider a general framework of nonlinear stochastic gradient descent (SGD), providing several strong results. First, for non-convex costs and component-wise nonlinearities, we establish a convergence rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{4}}\right)$, whose exponent is independent of noise and problem parameters. Second, for strongly convex costs and component-wise nonlinearities, we establish a rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{2}}\right)$ for the weighted average of iterates, with exponent again independent of noise and problem parameters. Finally, for strongly convex costs and a broader class of nonlinearities, we establish convergence of the last iterate, with a rate $\mathcal{O}\left(t^{-ζ} \right)$, where $ζ\in (0,1)$ depends on problem parameters, noise and nonlinearity. As we show analytically and numerically, $ζ$ can be used to inform the preferred choice of nonlinearity for given problem settings. Compared to state-of-the-art, who only consider clipping, require bounded noise moments of order $η\in (1,2]$, and establish convergence rates whose exponents go to zero as $η\rightarrow 1$, we provide high-probability guarantees for a much broader class of nonlinearities and symmetric density noise, with convergence rates whose exponents are bounded away from zero, even when the noise has finite first moment only. Moreover, in the case of strongly convex functions, we demonstrate analytically and numerically that clipping is not always the optimal nonlinearity, further underlining the value of our general framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑