arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.06591math-phcs.ITcs.LGmath.ITmath.MPstat.ML
收藏

严格的动力学平均场理论用于随机梯度下降方法

Rigorous dynamical mean field theory for stochastic gradient descent methods

Cedric Gerbelot, Emanuele Troiani, Francesca Mignacco, Florent Krzakala, Lenka Zdeborova

AI总结:

本文通过严格动力学平均场理论推导了随机梯度下降方法的闭式方程,揭示了记忆核的形成机制并实现了非分离更新函数的处理。

摘要:

我们证明了对于一组一阶梯度基方法的精确高维渐进行为的闭式方程,这些方法通过经验风险最小化从高斯数据观测中学习估计器(例如M估计量、浅层神经网络等)。这包括广泛使用的算法,如随机梯度下降(SGD)或Nesterov加速。所得到的方程与应用于梯度流的动力学平均场理论(DMFT)方程的离散化结果相匹配。我们的证明方法使我们能够明确描述记忆核在有效动力学中如何形成,并允许包含非分离的更新函数,从而允许具有非单位协方差矩阵的数据集。最后,我们为具有通用扩展批大小和恒定学习率的SGD提供了方程的数值实现。

英文摘要:

We prove closed-form equations for the exact high-dimensional asymptotics of a family of first order gradient-based methods, learning an estimator (e.g. M-estimator, shallow neural network, ...) from observations on Gaussian data with empirical risk minimization. This includes widely used algorithms such as stochastic gradient descent (SGD) or Nesterov acceleration. The obtained equations match those resulting from the discretization of dynamical mean-field theory (DMFT) equations from statistical physics when applied to gradient flow. Our proof method allows us to give an explicit description of how memory kernels build up in the effective dynamics, and to include non-separable update functions, allowing datasets with non-identity covariance matrices. Finally, we provide numerical implementations of the equations for SGD with generic extensive batch-size and with constant learning rates.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑