深度神经网络的丢弃Rademacher复杂性
Dropout Rademacher Complexity of Deep Neural Networks
摘要:
深度神经网络在各种实际应用中取得了巨大的成功。许多算法和实现技术已经被开发出来,然而,对深度神经网络的许多方面的理论理解还远未清楚。一个特别有趣的问题是dropout的有用性,这是出于防止特征检测器复杂的协同适应的直觉。在本文中,我们研究了不同类型的dropout的Rademacher复杂度,我们的理论结果表明,对于浅层神经网络(有一个或没有隐藏层)dropout能够降低Rademacher复杂度的多项式,而对于深层神经网络,它可以惊人地导致Rademacher复杂度的指数下降。
英文摘要:
Great successes of deep neural networks have been witnessed in various real applications. Many algorithmic and implementation techniques have been developed, however, theoretical understanding of many aspects of deep neural networks is far from clear. A particular interesting issue is the usefulness of dropout, which was motivated from the intuition of preventing complex co-adaptation of feature detectors. In this paper, we study the Rademacher complexity of different types of dropout, and our theoretical results disclose that for shallow neural networks (with one or none hidden layer) dropout is able to reduce the Rademacher complexity in polynomial, whereas for deep neural networks it can amazingly lead to an exponential reduction of the Rademacher complexity.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。