arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1804.00057cs.LGcs.ITmath.ITstat.ML
收藏

用信息论概念理解自动编码器

Understanding Autoencoders with Information Theoretic Concepts

University of Florida(佛罗里达大学)

Shujian Yu, Jose C. Principe

摘要:

尽管它们在实际应用中取得了巨大的成功,但仍然缺乏理论和系统的方法来分析深层神经网络。在本文中,我们说明了一种先进的信息理论方法,以了解学习的动态和自动编码器的设计,这是一种特殊类型的深度学习架构,类似于通信信道。通过将信息平面推广到任何成本函数,并使用逐层信息量检查不同层的角色和动态,我们强调了互信息在量化数据学习中的作用。我们进一步建议,并通过实验验证,均方误差训练,三个基本属性的逐层信息流和内在维度的瓶颈层,分别使用的数据处理不等式和识别的分叉点的信息平面中,是由给定的数据控制。我们的观察结果对自动编码器的优化设计,替代前馈训练方法的设计,甚至泛化问题都有直接影响。

英文摘要:

Despite their great success in practical applications, there is still a lack of theoretical and systematic methods to analyze deep neural networks. In this paper, we illustrate an advanced information theoretic methodology to understand the dynamics of learning and the design of autoencoders, a special type of deep learning architectures that resembles a communication channel. By generalizing the information plane to any cost function, and inspecting the roles and dynamics of different layers using layer-wise information quantities, we emphasize the role that mutual information plays in quantifying learning from data. We further suggest and also experimentally validate, for mean square error training, three fundamental properties regarding the layer-wise flow of information and intrinsic dimensionality of the bottleneck layer, using respectively the data processing inequality and the identification of a bifurcation point in the information plane that is controlled by the given data. Our observations have a direct impact on the optimal design of autoencoders, the design of alternative feedforward training methods, and even in the problem of generalization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑