神经网络的梯度优势特征和正则条件
Characterization of Gradient Dominance and Regularity Conditions for Neural Networks
摘要：
在过去的十年中，深度学习成功地应用于解决机器学习和人工智能中的许多挑战性问题。然而，深度神经网络（特别是非线性网络）的损失函数从理论上讲还远未得到很好的理解。在本文中，我们丰富了目前的理解景观的平方损失函数的三种类型的神经网络。具体来说，当参数矩阵是平方，我们提供了一个明确的特征的全局极小的线性网络，线性残差网络，和非线性网络的一个隐藏层。然后，我们建立了这些神经网络的平方损失的两个二次型景观性质，即，在满秩全局极小点邻域内的梯度优势条件，以及在全局极小点邻域内沿某些方向的正则性条件。这两个景观属性是围绕这些神经网络的损失函数的全局最小值进行优化所期望的。
英文摘要：
The past decade has witnessed a successful application of deep learning to solving many challenging problems in machine learning and artificial intelligence. However, the loss functions of deep neural networks (especially nonlinear networks) are still far from being well understood from a theoretical aspect. In this paper, we enrich the current understanding of the landscape of the square loss functions for three types of neural networks. Specifically, when the parameter matrices are square, we provide an explicit characterization of the global minimizers for linear networks, linear residual networks, and nonlinear networks with one hidden layer. Then, we establish two quadratic types of landscape properties for the square loss of these neural networks, i.e., the gradient dominance condition within the neighborhood of their full rank global minimizers, and the regularity condition along certain directions and within the neighborhood of their global minimizers. These two landscape properties are desirable for the optimization around the global minimizers of the loss function for these neural networks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
