跳跃连接可证明能改变神经网络的损失格局吗？
Is the Skip Connection Provable to Reform the Neural Network Loss Landscape?
摘要：
残差网络现在是深度学习中最有效的结构之一，它利用跳过连接来"保证”性能不会变差。然而，神经网络的非凸性使得不清楚跳跃连接是否确实可证明地提高学习能力，因为非线性可能产生许多局部最小值。在以前的一些工作\cite{freeman2016topology}中，尽管非凸性，但当隐藏节点的数量$m$非常大时，双层ReLU网络的损失景观具有良好的特性。在本文中，我们沿着这条路线来研究拓扑（子水平集）的损失景观的深度ReLU神经网络与跳跃连接，并从理论上证明了跳跃连接网络继承了两层网络的良好性质和跳跃连接可以帮助控制子水平集的连通性，使得任何比一些双层ReLU网络的全局最小值更差的局部最小值将非常“浅”。这些局部最小值的"深度”至多是$O(m^{(η-1)/n})$，其中$n$是输入维数，$η<1$。这为深度学习中跳跃连接的有效性提供了理论解释。
英文摘要：
The residual network is now one of the most effective structures in deep learning, which utilizes the skip connections to ``guarantee" the performance will not get worse. However, the non-convexity of the neural network makes it unclear whether the skip connections do provably improve the learning ability since the nonlinearity may create many local minima. In some previous works \cite{freeman2016topology}, it is shown that despite the non-convexity, the loss landscape of the two-layer ReLU network has good properties when the number $m$ of hidden nodes is very large. In this paper, we follow this line to study the topology (sub-level sets) of the loss landscape of deep ReLU neural networks with a skip connection and theoretically prove that the skip connection network inherits the good properties of the two-layer network and skip connections can help to control the connectedness of the sub-level sets, such that any local minima worse than the global minima of some two-layer ReLU network will be very ``shallow". The ``depth" of these local minima are at most $O(m^{(η-1)/n})$, where $n$ is the input dimension, $η<1$. This provides a theoretical explanation for the effectiveness of the skip connection in deep learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
