arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.14567stat.MLcs.LG
收藏

常学习率SGD的强高斯近似:偏差与轨迹推断

Strong Gaussian Approximation of Constant Learning Rate SGD: Bias and Trajectory Inference

University of Chicago(芝加哥大学)

Joseph Keslin, Wei Biao Wu

AI总结:

本文针对重尾噪声下的常学习率SGD,构建KMT型强高斯近似,推导相关界与定理,提出Burn in方案和Richardson-Romberg外推法,将其应用于风险控制与同步推断并通过模拟验证。

摘要:

本文为常学习率随机梯度下降(SGD)构建了强高斯近似(KMT型)。我们针对重尾噪声开展研究,假设迭代次数增长且学习率同步衰减。我们在同一空间上构造了简单高斯过程,并概率性地界定了SGD误差与其各自极限之间的最大差值。尽管已有针对以最优解为中心的常学习率SGD的中心极限定理,但这些定理仅能提供单个迭代的推断。相比之下,我们的淬灭KMT近似允许有限样本、同步、全轨迹级别的推断,并隐含随机过程收敛(泛函中心极限定理(FCLTs))。对于以真实最优解为中心的累积误差,当学习率按γₙ = c/√n + o(n⁻¹/²)缩放(其中c为正实数,n为迭代次数)时,我们首先得到了KMT型近似界。该结果揭示了在控制SGD偏差与初始化效应之间存在权衡,通常不存在学习率缩放能使累积误差极限中的这两种贡献均消失。为消除这些贡献,我们考虑了Burn in方案与Richardson-Romberg外推法,并分别证明了KMT与FCLT定理。我们还针对重缩放迭代轨迹至平稳Ornstein-Uhlenbeck(OU)过程,证明了有限时域KMT型近似与FCLT。我们将其应用于风险控制与同步推断,并开展了模拟以说明我们的结果。

英文摘要:

This paper develops strong Gaussian approximations (KMT type) for constant learning rate stochastic gradient descent (SGD). We work with heavy tailed noise and assume that the number of iterates grows and the learning rate shrinks simultaneously. Our work constructs simple Gaussian processes on the same space and bounds, in probability, the maximum difference between the SGD errors and their respective limits. While there exist central limit theorems for constant rate SGD centered at the optimum, these only provide individual iterate inference. By contrast, our quenched KMT approximations allow for finite sample, simultaneous, whole-trajectory-level inference and imply stochastic process convergence (Functional Central Limit Theorems (FCLTs)). For accumulated errors centered at the true optimum, we first obtain a KMT-type approximation bound when the learning rate scales as $γ_n=\frac{c}{\sqrt n}+o(n^{-1/2})$ for a constant $c \in \mathbb{R}^+$ and $n$ iterations. The result reveals a trade off between controlling the bias of SGD and the initialization effect, with no learning rate scaling generally making both contributions vanish in the accumulated error limit. To eliminate these contributions, we consider a Burn in scheme and Richardson-Romberg extrapolation and prove respective KMT and FCLT theorems. We also prove a finite-horizon KMT-type approximation and FCLT for the rescaled iterate trajectory to a stationary Ornstein--Uhlenbeck (OU) process. We consider applications in risk control and simultaneous inference and perform simulations to illustrate our results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑