arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1802.04473cs.LGcs.ITmath.IT
收藏

深层神经网络的信息缩放定律

Information Scaling Law of Deep Neural Networks

Xiao-Yang Liu

摘要:

随着深度神经网络(DNN)的快速发展,各种显示出强大计算能力和令人印象深刻的表达能力的网络模型被提出。然而,目前还没有从信息论的角度对DNN进行全面的信息解释。由于DNN中使用的非线性函数和不确定的层数和神经元数量,网络结构表现出非线性和复杂性。通过典型的DNN卷积算术电路(ConvACs),可以将复杂的DNN转换为数学公式。因此,我们可以使用严格的数学理论,特别是信息论来分析复杂的DNN。在本文中,我们提出了一个新的信息缩放定律方案,可以解释网络的内部组织的信息论。首先,我们展示了激活函数的信息解释。其次,我们证明了当信息通过ConvAC传输时,信息熵增加。最后,通过合理的假设,我们给出了对流层的信息缩放定律。

英文摘要:

With the rapid development of Deep Neural Networks (DNNs), various network models that show strong computing power and impressive expressive power are proposed. However, there is no comprehensive informational interpretation of DNNs from the perspective of information theory. Due to the nonlinear function and the uncertain number of layers and neural units used in the DNNs, the network structure shows nonlinearity and complexity. With the typical DNNs named Convolutional Arithmetic Circuits (ConvACs), the complex DNNs can be converted into mathematical formula. Thus, we can use rigorous mathematical theory especially the information theory to analyse the complicated DNNs. In this paper, we propose a novel information scaling law scheme that can interpret the network's inner organization by information theory. First, we show the informational interpretation of the activation function. Secondly, we prove that the information entropy increases when the information is transmitted through the ConvACs. Finally, we propose the information scaling law of ConvACs through making a reasonable assumption.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑