arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2105.06569cs.LG
收藏

过参数化神经网络的梯度下降动力学

The Dynamics of Gradient Descent for Overparametrized Neural Networks

University of Illinois at Urbana Champaign(伊利诺伊大学厄巴纳-香槟分校)

Siddhartha Satpathi, R Srikant

摘要:

研究了具有平方损失函数的过参数单隐层神经网络的梯度下降动力学问题。最近,它已被证明,在某些条件下,使用GD获得的参数值实现零训练误差和推广良好,如果初始条件选择适当。在这里,通过李雅普诺夫分析,我们表明,动力学的神经网络的权重下GD收敛到一个点,这是接近的最小范数解的条件下,没有训练误差时,使用线性逼近的神经网络。为了说明这一结果的应用,我们证明了GD收敛于一个泛化良好的预测函数,从而为Arora等人的泛化结果提供了另一种证明。

英文摘要:

We consider the dynamics of gradient descent (GD) in overparameterized single hidden layer neural networks with a squared loss function. Recently, it has been shown that, under some conditions, the parameter values obtained using GD achieve zero training error and generalize well if the initial conditions are chosen appropriately. Here, through a Lyapunov analysis, we show that the dynamics of neural network weights under GD converge to a point which is close to the minimum norm solution subject to the condition that there is no training error when using the linear approximation to the neural network. To illustrate the application of this result, we show that the GD converges to a prediction function that generalizes well, thereby providing an alternative proof of the generalization results in Arora et al. (2019).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑