非线性神经网络训练的不适定性与Optimization几何
Ill-Posedness and Optimization Geometry for Nonlinear Neural Network Training
摘要：
在这项工作中，我们分析的作用，非线性激活函数在密集的神经网络训练问题的稳定点。我们考虑一个通用的最小二乘损失函数训练公式。我们发现，在网络建设中使用的非线性激活函数在分类损失景观的稳定点中发挥着至关重要的作用。我们表明，对于浅稠密网络，非线性激活函数确定了全局极小值附近的Hessian零空间（如果存在），因此确定了训练问题的不适定性。此外，对于浅非线性网络，我们表明，零点的激活函数及其衍生物可以导致虚假的局部极小值，并讨论了严格的鞍点的条件。我们将这些结果扩展到深度密集神经网络，表明最后一个激活函数在分类固定点时起着重要作用，这是由于它如何出现在链式法则的梯度中。
英文摘要：
In this work we analyze the role nonlinear activation functions play at stationary points of dense neural network training problems. We consider a generic least squares loss function training formulation. We show that the nonlinear activation functions used in the network construction play a critical role in classifying stationary points of the loss landscape. We show that for shallow dense networks, the nonlinear activation function determines the Hessian nullspace in the vicinity of global minima (if they exist), and therefore determines the ill-posedness of the training problem. Furthermore, for shallow nonlinear networks we show that the zeros of the activation function and its derivatives can lead to spurious local minima, and discuss conditions for strict saddle points. We extend these results to deep dense neural networks, showing that the last activation function plays an important role in classifying stationary points, due to how it shows up in the gradient from the chain rule.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
