arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.15898cs.LGstat.ML
收藏

深度神经网络的运动方程:连续时间梯度下降和离散化误差分析

Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis

NEC Corporation(日本电气股份有限公司)

Taiki Miyagawa

摘要:

我们推导并求解了深度神经网络(DNN)的“运动方程”(EoM),这是一个精确描述DNN离散学习动态的微分方程。微分方程是连续的,但甚至在离散优化(梯度下降(GD)算法)的研究中也发挥了重要作用。然而,由于离散化误差,微分方程和DNN的实际学习动态之间仍然存在差距。在本文中,我们从梯度流(GF),并推导出一个计数器项,消除了GF和GD之间的离散误差。其结果是,我们得到EoM,一个连续的微分方程,精确地描述了GD的离散学习动力学。我们还推导出离散误差,以显示在何种程度上EoM是精确的。此外,我们将EoM应用于两种特定情况:尺度不变层和消隐不变层。EoM突出了连续时间和离散时间GD之间的差异,表明计数器项对于更好地描述GD的离散学习动态的重要性。我们的实验结果支持我们的理论研究结果。

英文摘要:

We derive and solve an ``Equation of Motion'' (EoM) for deep neural networks (DNNs), a differential equation that precisely describes the discrete learning dynamics of DNNs. Differential equations are continuous but have played a prominent role even in the study of discrete optimization (gradient descent (GD) algorithms). However, there still exist gaps between differential equations and the actual learning dynamics of DNNs due to discretization error. In this paper, we start from gradient flow (GF) and derive a counter term that cancels the discretization error between GF and GD. As a result, we obtain EoM, a continuous differential equation that precisely describes the discrete learning dynamics of GD. We also derive discretization error to show to what extent EoM is precise. In addition, we apply EoM to two specific cases: scale- and translation-invariant layers. EoM highlights differences between continuous-time and discrete-time GD, indicating the importance of the counter term for a better description of the discrete learning dynamics of GD. Our experimental results support our theoretical findings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑