ReLU网络梯度下降训练中的高原现象：解释、量化与避免
Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidance
摘要：
神经网络在广泛的应用中提供“同类最佳”近似的能力是有据可查的。然而，如果不能有效地训练（选择）定义网络的参数，神经网络的强大表现力将化为乌有。通常，神经网络通过梯度下降型优化方法或其随机变体来训练。在实践中，这种方法导致损失函数在训练开始时迅速减小，但随后在相对少量的步骤之后显著减慢。这种损失甚至可能在大量的时期内表现为停滞，然后突然又开始快速减少，没有明显的原因。这种所谓的高原现象在许多学习任务中表现出来。目前的工作旨在确定和量化高原现象的根本原因。没有假设神经元的数量相对于训练数据的数量，我们的结果适用于懒惰和自适应制度。主要调查结果如下：平稳期对应于激活模式保持恒定的时期，其中激活模式是指激活给定神经元的数据点的数量;梯度流动动力学的收敛的量化;以及根据训练数据的子集上的局部最小二乘回归线的解来表征静止点。基于这些结论，我们提出了一种新的迭代训练方法，主动神经元最小二乘法（ANLS），其特征在于在每一步的激活模式的显式调整，其目的是使快速退出高原。说明性的数值例子包括在整个。
英文摘要：
The ability of neural networks to provide `best in class' approximation across a wide range of applications is well-documented. Nevertheless, the powerful expressivity of neural networks comes to naught if one is unable to effectively train (choose) the parameters defining the network. In general, neural networks are trained by gradient descent type optimization methods, or a stochastic variant thereof. In practice, such methods result in the loss function decreases rapidly at the beginning of training but then, after a relatively small number of steps, significantly slow down. The loss may even appear to stagnate over the period of a large number of epochs, only to then suddenly start to decrease fast again for no apparent reason. This so-called plateau phenomenon manifests itself in many learning tasks. The present work aims to identify and quantify the root causes of plateau phenomenon. No assumptions are made on the number of neurons relative to the number of training data, and our results hold for both the lazy and adaptive regimes. The main findings are: plateaux correspond to periods during which activation patterns remain constant, where activation pattern refers to the number of data points that activate a given neuron; quantification of convergence of the gradient flow dynamics; and, characterization of stationary points in terms solutions of local least squares regression lines over subsets of the training data. Based on these conclusions, we propose a new iterative training method, the Active Neuron Least Squares (ANLS), characterised by the explicit adjustment of the activation pattern at each step, which is designed to enable a quick exit from a plateau. Illustrative numerical examples are included throughout.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
