混沌边缘作为现代神经网络训练的指导原则
Edge of chaos as a guiding principle for modern neural network training
摘要：
深度神经网络在现实问题中的成功促使人们尝试解释其训练动态和泛化性能，但仍然需要更多的神经网络训练指导原则。受神经网络最优性能背后的混沌边缘原理的启发，从有序-混沌相图的角度研究了各种超参数在现代神经网络训练算法中的作用。特别是，我们研究了在广泛采用的Fashion-MNIST数据集上训练的完全分析的前馈神经网络，并研究了在训练过程中与反向传播中的超参数相关的动态。我们发现，对于基本算法的随机梯度下降的动量，在常用的超参数值的范围内，在相图中的有序阶段，相对于训练时间存在明显的标度关系，并且在混沌边缘的模型的最佳泛化能力是相似的不同的训练参数组合。在混沌阶段，相同的标度不再存在。缩放允许我们选择训练参数，以实现更快的训练，而不会牺牲性能。此外，我们发现常用的模型正则化方法--权重衰减--有效地将模型推向有序阶段，以获得更好的性能。利用这一事实和其他超参数中的标度关系，我们导出了超参数确定的原则性指导方针，使得模型可以通过在混沌边缘饱和来实现最佳性能。在这个简单的神经网络模型和训练算法上，我们的工作提高了对神经网络训练动态的理解，并且可以潜在地扩展到更复杂的模型架构和算法的指导原则。
英文摘要：
The success of deep neural networks in real-world problems has prompted many attempts to explain their training dynamics and generalization performance, but more guiding principles for the training of neural networks are still needed. Motivated by the edge of chaos principle behind the optimal performance of neural networks, we study the role of various hyperparameters in modern neural network training algorithms in terms of the order-chaos phase diagram. In particular, we study a fully analytical feedforward neural network trained on the widely adopted Fashion-MNIST dataset, and study the dynamics associated with the hyperparameters in back-propagation during the training process. We find that for the basic algorithm of stochastic gradient descent with momentum, in the range around the commonly used hyperparameter values, clear scaling relations are present with respect to the training time during the ordered phase in the phase diagram, and the model's optimal generalization power at the edge of chaos is similar across different training parameter combinations. In the chaotic phase, the same scaling no longer exists. The scaling allows us to choose the training parameters to achieve faster training without sacrificing performance. In addition, we find that the commonly used model regularization method - weight decay - effectively pushes the model towards the ordered phase to achieve better performance. Leveraging on this fact and the scaling relations in the other hyperparameters, we derived a principled guideline for hyperparameter determination, such that the model can achieve optimal performance by saturating it at the edge of chaos. Demonstrated on this simple neural network model and training algorithm, our work improves the understanding of neural network training dynamics, and can potentially be extended to guiding principles of more complex model architectures and algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
