arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.02258stat.MLcs.LGmath.PRmath.STstat.TH
收藏

高维单层网络中随机梯度下降的极限定理

Limit Theorems for Stochastic Gradient Descent in High-Dimensional Single-Layer Networks

Department of Statistics and Actuarial Science, University of Waterloo(统计与精算科学系,滑铁卢大学)

Parsa Rangriz

AI总结:

本文研究了在线随机梯度下降在高维下的极限行为,揭示了步长临界尺度下有效动力学的确定性极限与新修正项对相图的影响,展示了信息指数对样本复杂度的控制作用。

摘要:

本文研究了在线随机梯度下降(SGD)在高维下的极限行为。基于Ben Arous、Gheissari和Jagannath关于SGD有效动力学的最新工作,我们研究了单层网络中步长的临界尺度。在该临界尺度以下,有效动力学由确定性(球形)极限支配,而在临界尺度上,新的修正项出现,改变了相图。在此尺度下,近固定点处的有效动力学扩散(SDE)极限在某些条件下退化为奥本海姆-乌尔申贝格过程。这些结果展示了信息指数如何控制样本复杂度,并说明了确定性极限在捕捉高维学习动态中的随机波动时的局限性。

英文摘要:

This paper studies the high-dimensional scaling limits of online stochastic gradient descent (SGD). Building on the work of Ben Arous, Gheissari, and Jagannath on the effective dynamics of SGD, we study the critical scaling regime of the step size for single-layer networks. Below this regime the effective dynamics are governed by deterministic (ballistic) limits, whereas at the critical scale a correction term emerges that changes the phase diagram. Near the fixed points of these dynamics, we show that the diffusive (SDE) limit of the rescaled correlation is an Ornstein-Uhlenbeck process. More precisely, it is mean-reverting whenever the information exponent is at least three. At information exponent two the drift has no universal sign, and the fixed point may become repelling; we show this explicitly for phase retrieval, where the sign is determined by the step size and the noise level. These results illustrate the limitations of deterministic scaling limits in capturing stochastic fluctuations in high-dimensional learning dynamics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑