arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1802.07044cs.LG
收藏

深度学习模型的描述长度

The Description Length of Deep Learning Models

École Normale Supérieure(巴黎高等师范学院);Facebook Artificial Intelligence Research(Facebook人工智能研究院)

Léonard Blier, Yann Ollivier

摘要:

Solomonoff的一般推理理论和最小描述长度原则形式化了奥卡姆剃刀,并认为一个好的数据模型是一个善于对数据进行有损压缩的模型,包括描述模型本身的代价。深度神经网络似乎违背了这一原则,因为需要编码大量的参数。我们通过实验证明了深度神经网络压缩训练数据的能力,即使在考虑参数编码时也是如此。压缩的观点最初激发了变分方法在神经网络中的使用。出乎意料的是,我们发现这些变分方法提供了令人惊讶的差压缩边界,尽管显式地建立,以尽量减少这样的界限。这可能解释了变分方法在深度学习中相对较差的实际性能。另一方面,简单的增量编码方法在深度网络上产生了出色的压缩值,证明了Solomonoff的方法。

英文摘要:

Solomonoff's general theory of inference and the Minimum Description Length principle formalize Occam's razor, and hold that a good model of data is a model that is good at losslessly compressing the data, including the cost of describing the model itself. Deep neural networks might seem to go against this principle given the large number of parameters to be encoded. We demonstrate experimentally the ability of deep neural networks to compress the training data even when accounting for parameter encoding. The compression viewpoint originally motivated the use of variational methods in neural networks. Unexpectedly, we found that these variational methods provide surprisingly poor compression bounds, despite being explicitly built to minimize such bounds. This might explain the relatively poor practical performance of variational methods in deep learning. On the other hand, simple incremental encoding methods yield excellent compression values on deep networks, vindicating Solomonoff's approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑