通过均匀加倍条件训练深度神经网络的精度稳定性
Stability of Accuracy for the Training of DNNs Via the Uniform Doubling Condition
AI总结:
本文提出均匀加倍条件,确保深度神经网络训练中精度稳定性,并扩展至Leaky ReLU等分段线性激活函数。
摘要:
我们研究了深度神经网络(DNNs)训练过程中精度的稳定性。在此背景下,DNN的训练通过最小化交叉熵损失函数进行,性能指标是精度(正确分类对象的比例)。虽然训练导致损失下降,但精度在过程中不一定增加,有时甚至可能下降。实现精度稳定性的目标是确保如果精度在某个初始时间很高,那么在整个训练过程中它都保持很高。Berlyand、Jabin和Safsten最近的一个结果引入了训练数据上的加倍条件,该条件确保了使用绝对值激活函数的DNN在训练期间精度的稳定性。对于$\nmathbb{R}^n$中的训练数据,该加倍条件使用$\nmathbb{R}^n$中的板(slabs)来表述,并且依赖于板的选择。本文的目标有两个。首先,使加倍条件均匀化,即与板的选择无关。这导致了仅基于训练数据的稳定性充分条件。换句话说,对于满足均匀加倍条件的训练集$T$,存在一族DNN,使得来自该族的一个DNN在某个训练时间$t_0$在训练集上具有高精度,则对于所有时间$t>t_0$都将具有高精度。此外,建立均匀性对于加倍条件的数值实现是必要的。第二个目标是将原始的稳定性结果从绝对值激活函数扩展到更广泛的具有有限个临界点的分段线性激活函数类,例如流行的Leaky ReLU。
英文摘要:
We study the stability of accuracy during the training of deep neural networks (DNNs). In this context, the training of a DNN is performed via the minimization of a cross-entropy loss function, and the performance metric is accuracy (the proportion of objects that are classified correctly). While training results in a decrease of loss, the accuracy does not necessarily increase during the process and may sometimes even decrease. The goal of achieving stability of accuracy is to ensure that if accuracy is high at some initial time, it remains high throughout training. A recent result by Berlyand, Jabin, and Safsten introduces a doubling condition on the training data, which ensures the stability of accuracy during training for DNNs using the absolute value activation function. For training data in $\mathbb{R}^n$, this doubling condition is formulated using slabs in $\mathbb{R}^n$ and depends on the choice of the slabs. The goal of this paper is twofold. First, to make the doubling condition uniform, that is, independent of the choice of slabs. This leads to sufficient conditions for stability in terms of training data only. In other words, for a training set $T$ that satisfies the uniform doubling condition, there exists a family of DNNs such that a DNN from this family with high accuracy on the training set at some training time $t_0$ will have high accuracy for all time $t>t_0$. Moreover, establishing uniformity is necessary for the numerical implementation of the doubling condition. The second goal is to extend the original stability results from the absolute value activation function to a broader class of piecewise linear activation functions with finitely many critical points, such as the popular Leaky ReLU.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。