With the rapid development of Deep Neural Networks (DNNs), various network models that show strong computing power and impressive expressive power are proposed. However, there is no comprehensive informational interpretation of DNNs from the perspective of information theory. Due to the nonlinear function and the uncertain number of layers and neural units used in the DNNs, the network structure shows nonlinearity and complexity. With the typical DNNs named Convolutional Arithmetic Circuits (ConvACs), the complex DNNs can be converted into mathematical formula. Thus, we can use rigorous mathematical theory especially the information theory to analyse the complicated DNNs. In this paper, we propose a novel information scaling law scheme that can interpret the network's inner organization by information theory. First, we show the informational interpretation of the activation function. Secondly, we prove that the information entropy increases when the information is transmitted through the ConvACs. Finally, we propose the information scaling law of ConvACs through making a reasonable assumption.
Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approaches
The University of Chicago(芝加哥大学);Guilin University of Electronic Technology(桂林电子科技大学);University of New South Wales(新南威尔士大学);The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);The Simons Institute for the Theory of Computing at UC Berkeley(加州大学伯克利分校西蒙斯理论计算研究所)
Institut für Theoretische Physik, Universität Heidelberg(海德堡大学理论物理研究所);Department of Physics, Harvard University(哈佛大学物理系);LPNHE, Sorbonne Université, Université Paris Cité, CNRS/IN2P3(巴黎大学LPNHE)
ELLIS Institute Tübingen(埃利斯研究所蒂宾根分所);OpenEuroLLM;Eindhoven University of Technology(埃因霍温理工大学);University of Freiburg(弗赖堡大学);Zuse School ELIZA(楚塞ELIZA学校);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Tübingen AI Center(蒂宾根人工智能中心)