深度线性网络的精确解
Exact Solutions of a Deep Linear Network
AI总结：
本研究推导带权重衰减与随机神经元的深度线性网络全局极小值解析表达式，揭示原点处的非线性现象与权重衰减的架构依赖效应，指出现有初始化方法不足以优化深度网络。
摘要：
本工作推导了带有权重衰减和随机神经元的深度线性网络全局极小值的解析表达式，该网络是理解神经网络损失景观的基础模型。研究结果表明，原点是深度神经网络损失景观中的特殊点，此处会出现高度非线性现象。我们证明权重衰减与模型架构存在强相互作用，在隐藏层数大于1的网络中，权重衰减可在零点处产生不良极小值，这与仅含1个隐藏层的网络有定性差异。实践层面，该结果表明常用的深度学习初始化方法通常不足以缓解神经网络的优化问题。
英文摘要：
This work finds the analytical expression of the global minima of a deep linear network with weight decay and stochastic neurons, a fundamental model for understanding the landscape of neural networks. Our result implies that the origin is a special point in deep neural network loss landscape where highly nonlinear phenomenon emerges. We show that weight decay strongly interacts with the model architecture and can create bad minima at zero in a network with more than $1$ hidden layer, qualitatively different from a network with only $1$ hidden layer. Practically, our result implies that common deep learning initialization methods are insufficient to ease the optimization of neural networks in general.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
