arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.01045cs.LGcs.NE
收藏

计算训练好的神经网络的信息量

Computing the Information Content of Trained Neural Networks

Jeremy Bernstein, Yisong Yue

摘要:

学习算法从训练数据中提取多少信息并存储在神经网络的权重中?太多了,网络就会过拟合到训练数据。太少了,网络根本不适合任何东西。简单地说,网络存储的信息量应该与可训练权重的数量成比例。这就提出了一个问题:权重远远超过训练数据的神经网络如何仍然能够泛化?解决这个难题的一个简单方法是,权重的数量通常不能代表实际存储的信息量。例如,典型的权重向量可以是高度可压缩的。然后另一个问题出现了:是否有可能计算出存储的实际信息量?本文导出了无限宽神经网络信息量的一个相容估计和一个封闭形式的上界。推导是基于神经信息内容和负对数概率的高斯正交之间的识别。这种识别产生的边界,分析控制无限宽的网络的整个解决方案空间的泛化行为。边界简单地依赖于网络架构和训练数据。证实了Valle-Pérez等人的发现。(2019)使用近似高斯积分技术进行了类似的分析,发现边界既不空洞,又与有限宽度的经验概括行为相关。

英文摘要:

How much information does a learning algorithm extract from the training data and store in a neural network's weights? Too much, and the network would overfit to the training data. Too little, and the network would not fit to anything at all. Naïvely, the amount of information the network stores should scale in proportion to the number of trainable weights. This raises the question: how can neural networks with vastly more weights than training data still generalise? A simple resolution to this conundrum is that the number of weights is usually a bad proxy for the actual amount of information stored. For instance, typical weight vectors may be highly compressible. Then another question occurs: is it possible to compute the actual amount of information stored? This paper derives both a consistent estimator and a closed-form upper bound on the information content of infinitely wide neural networks. The derivation is based on an identification between neural information content and the negative log probability of a Gaussian orthant. This identification yields bounds that analytically control the generalisation behaviour of the entire solution space of infinitely wide networks. The bounds have a simple dependence on both the network architecture and the training data. Corroborating the findings of Valle-Pérez et al. (2019), who conducted a similar analysis using approximate Gaussian integration techniques, the bounds are found to be both non-vacuous and correlated with the empirical generalisation behaviour at finite width.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑