arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.01853math.OC
收藏

自适应梯度方法在光滑非凸优化中的渐近收敛性与稳定性

Long-Run Dynamics of AdaGrad-Norm: Trajectory Stability and Asymptotic Stationarity

The Ohio State University(俄亥俄州立大学);University of Chinese Academy of Sciences(中国科学院大学)

Ruinan Jin, Xiaoyu Wang

AI总结:

本文首次对AdaGrad-Norm在非凸优化中的渐近收敛性进行严谨分析,并证明其稳定性及收敛性,同时扩展至RMSProp。

摘要:

自适应梯度方法,如AdaGrad,已成为深度学习中的基本工具。尽管其被广泛应用,但在非凸场景中AdaGrad的渐近收敛性仍缺乏深入理解。在本文中,我们首次对AdaGrad-Norm在光滑非凸优化中的渐近收敛性进行了严谨的分析。通过一种新颖的终止时间分拆技术,我们建立了关键的稳定性结果:目标函数值在期望下保持有界,并在弱协调性假设下,迭代点几乎 surely 保持有界。基于这些稳定性结果,我们证明了AdaGrad-Norm实现了几乎 surely 和均方收敛。进一步,我们将分析扩展到RMSProp,并证明在适当超参数选择下,它也享有稳定性和渐近收敛性。本文开发的技术可能对分析其他自适应随机优化算法具有独立价值。

英文摘要:

AdaGrad-Norm is widely studied, yet its long-run behavior in smooth nonconvex optimization remains delicate. For the standard current-denominator recursion, the same stochastic gradient determines both the update and its normalizer, rendering the effective stepsize nonpredictable. At the square-root normalization, the quadratic smoothness budget admits only logarithmic control rather than a uniform summability bound. We establish trajectory stability and full-sequence asymptotic stationarity directly for this unmodified recursion. Under global smoothness, non-flatness at infinity, conditional unbiasedness, and an affine conditional second-moment bound, we prove \(\E[\sup_{n\geq1} g(θ_n)]<\infty\) and \(\E[\sup_{n\geq1}\|θ_n\|]<\infty\), without assuming bounded iterates. For stationarity, we use a local conditional relative-moment condition that permits unbounded oracle values and covers fixed-size finite-sum mini-batches and locally nondegenerate additive noise. Together with a weak Sard condition, it yields \(\|\nabla g(θ_n)\|\to0\) almost surely and \(\E\|\nabla g(θ_n)\|^2\to0\). The proof combines a current-sample Lyapunov correction, a stopped-excursion stability argument, and finite-window control for the actual nonpredictable stepsize. We also analyze power normalization with \(q>1/2\), where the quadratic normalization budget is summable, identifying \(q=1/2\) as the boundary of the proof mechanism developed here rather than a universal algorithmic phase transition.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑