深度神经网络中的自由度
Degrees of Freedom in Deep Neural Networks
摘要:
在本文中,我们探讨了深度S形神经网络的自由度。我们证明了这些模型中的自由度与预期乐观度有关,预期乐观度是测试误差和训练误差之间的预期差。我们提供了一个有效的Monte-Carlo方法来估计多类分类方法的自由度。我们表明,自由度低于一个简单的XOR网络中的参数计数。我们将这些结果扩展到在合成和真实数据上训练的神经网络,并研究网络架构和不同正则化选择的影响。深度网络中的自由度比参数的数量小得多,在一些真实的数据集中是几个数量级。此外,我们观察到,对于固定数量的参数,更深的网络具有更少的自由度,表现出深度正则化。
英文摘要:
In this paper, we explore degrees of freedom in deep sigmoidal neural networks. We show that the degrees of freedom in these models is related to the expected optimism, which is the expected difference between test error and training error. We provide an efficient Monte-Carlo method to estimate the degrees of freedom for multi-class classification methods. We show degrees of freedom are lower than the parameter count in a simple XOR network. We extend these results to neural nets trained on synthetic and real data, and investigate impact of network's architecture and different regularization choices. The degrees of freedom in deep networks are dramatically smaller than the number of parameters, in some real datasets several orders of magnitude. Further, we observe that for fixed number of parameters, deeper networks have less degrees of freedom exhibiting a regularization-by-depth.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。