arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1506.05232cs.LG
收藏

论深度神经网络的深度:一种理论观点

On the Depth of Deep Neural Networks: A Theoretical View

Nankai University(南开大学);Microsoft Research(微软研究院);Peking University(北京大学)

Shizhao Sun, Wei Chen, Liwei Wang, Xiaoguang Liu, Tie-Yan Liu

摘要:

人们认为深度在深度神经网络(DNN)的成功中起着重要作用。然而,据我们所知,这种信念缺乏坚实的理论依据。我们从边际约束的角度研究了深度的作用。在裕度界中,期望误差由经验裕度误差加上基于Rademacher平均(RA)的容量项的上界。首先,我们推导出DNN的RA的上界,并表明它随着深度的增加而增加。这表明深度对测试性能有负面影响。其次,我们发现,在多类环境中,较深的网络往往比较浅的网络具有更大的表示能力(通过基于Betti数的复杂性来衡量),因此可以导致较小的经验边际误差。这意味着深度的积极影响。这两个结果的结合表明,对于隐藏单元数量有限的DNN,增加深度并不总是好的,因为在积极和消极影响之间存在权衡。这些结果激励我们寻求替代方法来实现深度的积极影响,例如,对交叉熵损失施加基于边缘的惩罚项,从而在不增加深度的情况下减小经验边缘误差。我们的实验表明,以这种方式,我们实现了显着更好的测试性能。

英文摘要:

People believe that depth plays an important role in success of deep neural networks (DNN). However, this belief lacks solid theoretical justifications as far as we know. We investigate role of depth from perspective of margin bound. In margin bound, expected error is upper bounded by empirical margin error plus Rademacher Average (RA) based capacity term. First, we derive an upper bound for RA of DNN, and show that it increases with increasing depth. This indicates negative impact of depth on test performance. Second, we show that deeper networks tend to have larger representation power (measured by Betti numbers based complexity) than shallower networks in multi-class setting, and thus can lead to smaller empirical margin error. This implies positive impact of depth. The combination of these two results shows that for DNN with restricted number of hidden units, increasing depth is not always good since there is a tradeoff between positive and negative impacts. These results inspire us to seek alternative ways to achieve positive impact of depth, e.g., imposing margin-based penalty terms to cross entropy loss so as to reduce empirical margin error without increasing depth. Our experiments show that in this way, we achieve significantly better test performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑