回顾学习函数与语义信息测度的演化以理解深度学习
Reviewing Evolution of Learning Functions and Semantic Information Measures for Understanding Deep Learning
AI总结:
本文回顾语义信息测度与学习函数的演化,介绍语义信息G理论及率保真函数R(G),探讨其在深度学习中的应用,并指出结合语义信息理论可加速深度学习发展。
摘要:
以互信息神经估计(MINE)和信息噪声对比估计(InfoNCE)为代表的深度学习新趋势正在兴起。在这一趋势中,相似性函数和估计互信息(EMI)被用作学习函数和目标函数。巧合的是,EMI本质上与作者30年前提出的语义互信息(SeMI)相同。本文首先回顾了语义信息测度和学习函数的演化历史。然后,简要介绍了作者的语义信息G理论及其率保真函数R(G)(G表示SeMI,R(G)扩展了R(D)),以及其在多标签学习、最大互信息(MI)分类和混合模型中的应用。接着,讨论了我们应该如何从R(G)函数或G理论的角度理解SeMI与香农MI、两种广义熵(模糊熵和覆盖熵)、自编码器、吉布斯分布和配分函数之间的关系。一个重要结论是,混合模型和受限玻尔兹曼机之所以收敛,是因为SeMI被最大化,而香农的MI被最小化,使得信息效率G/R接近1。一个潜在机遇是,利用高斯信道混合模型预训练深度神经网络的潜层,而无需考虑梯度,从而简化深度学习。本文还讨论了如何将SeMI测度用作强化学习的奖励函数(反映目的性)。G理论有助于解释深度学习,但还远远不够。结合语义信息理论和深度学习将加速它们的发展。
英文摘要:
A new trend in deep learning, represented by Mutual Information Neural Estimation (MINE) and Information Noise Contrast Estimation (InfoNCE), is emerging. In this trend, similarity functions and Estimated Mutual Information (EMI) are used as learning and objective functions. Coincidentally, EMI is essentially the same as Semantic Mutual Information (SeMI) proposed by the author 30 years ago. This paper first reviews the evolutionary histories of semantic information measures and learning functions. Then, it briefly introduces the author's semantic information G theory with the rate-fidelity function R(G) (G denotes SeMI, and R(G) extends R(D)) and its applications to multi-label learning, the maximum Mutual Information (MI) classification, and mixture models. Then it discusses how we should understand the relationship between SeMI and Shan-non's MI, two generalized entropies (fuzzy entropy and coverage entropy), Autoencoders, Gibbs distributions, and partition functions from the perspective of the R(G) function or the G theory. An important conclusion is that mixture models and Restricted Boltzmann Machines converge because SeMI is maximized, and Shannon's MI is minimized, making information efficiency G/R close to 1. A potential opportunity is to simplify deep learning by using Gaussian channel mixture models for pre-training deep neural networks' latent layers without considering gradients. It also discusses how the SeMI measure is used as the reward function (reflecting purposiveness) for reinforcement learning. The G theory helps interpret deep learning but is far from enough. Combining semantic information theory and deep learning will accelerate their development.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。