arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1808.00408cond-mat.dis-nncs.LGstat.ML
收藏

能量景观的几何和深度神经网络的可优化性

Geometry of energy landscapes and the optimizability of deep neural networks

University of Cambridge(剑桥大学)

Simon Becker, Yao Zhang, Alpha A. Lee

摘要:

深度神经网络是机器学习中的主力模型,具有串联组成的多层非线性函数。它们的损失函数是高度非凸的,但根据经验,即使梯度下降最小化也足以达到准确和预测的模型。迄今为止,人们还不知道为什么深度神经网络很容易优化。我们使用随机矩阵理论和代数几何分析了深度神经网络的自旋玻璃模型的能量景观。我们分析表明,多层结构的关键是优化:固定参数的数量和增加网络深度,在损失函数中的固定点的数量减少,极小值变得更加聚集在参数空间中,和最小值的深度和宽度之间的权衡变得不那么严重。我们的分析结果通过与在一组经典基准数据集上训练的神经网络进行比较进行了数值验证。我们的模型揭示了机器学习模型的通用设计原则。

英文摘要:

Deep neural networks are workhorse models in machine learning with multiple layers of non-linear functions composed in series. Their loss function is highly non-convex, yet empirically even gradient descent minimisation is sufficient to arrive at accurate and predictive models. It is hitherto unknown why are deep neural networks easily optimizable. We analyze the energy landscape of a spin glass model of deep neural networks using random matrix theory and algebraic geometry. We analytically show that the multilayered structure holds the key to optimizability: Fixing the number of parameters and increasing network depth, the number of stationary points in the loss function decreases, minima become more clustered in parameter space, and the tradeoff between the depth and width of minima becomes less severe. Our analytical results are numerically verified through comparison with neural networks trained on a set of classical benchmark datasets. Our model uncovers generic design principles of machine learning models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑