深度学习中的大时间渐近性
Large-time asymptotics in deep learning
摘要：
我们考虑了监督学习的神经ODE观点，并研究了训练中最后一次$T$（这可能表明相应ResNet的深度）的影响。对于经典的$L^2$ -正则化经验风险最小化问题，只要神经ODE动力学是齐次的参数，我们表明，训练误差是在最多的顺序$\mathcal{O}\left(\frac{1}{T}\right)$。此外，如果导致经验风险的损失达到其最小值，最优参数收敛到最小的$L^2$ -范数参数插值的数据集。当$λ\searrow0$和$T$固定时，通过$T$和正则化超参数$λ$之间的自然缩放，我们得到了相同的结果。这使我们能够在过度参数化的制度，现在看到的大深度，神经ODE的角度来规定泛化性能。为了提高多项式衰减，启发最优控制中的收费公路理论，我们提出了一个学习问题，额外的积分正则化项的神经ODE轨迹在$[0,T]$。在$\ell^p$ -距离损失的设置中，我们证明了在任何$t\in[0,T]$中，训练误差和最优参数都至多是$\mathcal{O}\left(e^{-μt}\right)$的阶。上述的稳定性估计也显示为连续的空间-时间神经网络，采取非线性积分微分方程的形式。通过使用时间相关的移动网格来离散空间变量，我们证明了这些方程为解决具有可变宽度的ResNet提供了一个框架。
英文摘要：
We consider the neural ODE perspective of supervised learning and study the impact of the final time $T$ (which may indicate the depth of a corresponding ResNet) in training. For the classical $L^2$--regularized empirical risk minimization problem, whenever the neural ODE dynamics are homogeneous with respect to the parameters, we show that the training error is at most of the order $\mathcal{O}\left(\frac{1}{T}\right)$. Furthermore, if the loss inducing the empirical risk attains its minimum, the optimal parameters converge to minimal $L^2$--norm parameters which interpolate the dataset. By a natural scaling between $T$ and the regularization hyperparameter $λ$ we obtain the same results when $λ\searrow0$ and $T$ is fixed. This allows us to stipulate generalization properties in the overparametrized regime, now seen from the large depth, neural ODE perspective. To enhance the polynomial decay, inspired by turnpike theory in optimal control, we propose a learning problem with an additional integral regularization term of the neural ODE trajectory over $[0,T]$. In the setting of $\ell^p$--distance losses, we prove that both the training error and the optimal parameters are at most of the order $\mathcal{O}\left(e^{-μt}\right)$ in any $t\in[0,T]$. The aforementioned stability estimates are also shown for continuous space-time neural networks, taking the form of nonlinear integro-differential equations. By using a time-dependent moving grid for discretizing the spatial variable, we demonstrate that these equations provide a framework for addressing ResNets with variable widths.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
