arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.07862cs.LGmath.OC
收藏

无有界梯度假设下随机梯度法的几乎处处鞍点回避

Almost Sure Saddle Avoidance of Stochastic Gradient Methods without the Bounded Gradient Assumption

University of Waterloo(滑铁卢大学);Huazhong University of Science and Technology(华中科技大学)

Jun Liu, Ye Yuan

摘要:

我们证明了各种随机梯度下降方法,包括随机梯度下降(SGD),随机重球(SHB),和随机Nesterov的加速梯度(SNAG)方法,几乎肯定避免任何严格的鞍流形。据我们所知,这是第一次获得这样的结果SHB和SNAG方法。此外,我们的分析扩展了以前的研究SGD通过消除需要有界梯度的目标函数和均匀有界噪声。相反,我们引入了一个更实际的局部有界性假设的噪声梯度,这是自然满足经验风险最小化问题,通常在神经网络的训练。

英文摘要:

We prove that various stochastic gradient descent methods, including the stochastic gradient descent (SGD), stochastic heavy-ball (SHB), and stochastic Nesterov's accelerated gradient (SNAG) methods, almost surely avoid any strict saddle manifold. To the best of our knowledge, this is the first time such results are obtained for SHB and SNAG methods. Moreover, our analysis expands upon previous studies on SGD by removing the need for bounded gradients of the objective function and uniformly bounded noise. Instead, we introduce a more practical local boundedness assumption for the noisy gradient, which is naturally satisfied in empirical risk minimization problems typically seen in training of neural networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑