最大相关性与最优学习机
Maximal Relevance and Optimal Learning Machines
摘要：
我们证明了学习机的表示和它从数据中提取的隐藏特征之间的互信息是由相关性从下到上限定的，相关性是模型能量分布的熵。具有最大相关性的模型-我们称之为最优学习机（OLM）-因此期望提取最大信息表示。我们在一系列模型中探索这一原则。对于完全连接的伊辛模型，我们表明，{ \em i）} OLM的特点是耦合的非均匀分布，和{ \em ii）}他们的学习性能是由子广泛的功能，是难以捉摸的热力学治疗的影响。在特定的学习任务，我们发现，可能性最大化是通过具有最大相关性的模型来实现的。在MNIST基准上对受限玻尔兹曼机的训练表明，学习与能级谱的加宽有关，并且隐藏层的内部表示接近在有限数据集中可以实现的最大相关性。最后，我们讨论了高斯学习机，澄清了学习隐藏的功能是从概念上不同于参数估计。
英文摘要：
We show that the mutual information between the representation of a learning machine and the hidden features that it extracts from data is bounded from below by the relevance, which is the entropy of the model's energy distribution. Models with maximal relevance -- that we call Optimal Learning Machines (OLM) -- are hence expected to extract maximally informative representations. We explore this principle in a range of models. For fully connected Ising models and we show that {\em i)} OLM are characterised by inhomogeneous distributions of couplings, and that {\em ii)} their learning performance is affected by sub-extensive features that are elusive to a thermodynamic treatment. On specific learning tasks, we find that likelihood maximisation is achieved by models with maximal relevance. Training of Restricted Boltzmann Machines on the MNIST benchmark shows that learning is associated with a broadening of the spectrum of energy levels and that the internal representation of the hidden layer approaches the maximal relevance that can be achieved in a finite dataset. Finally, we discuss a Gaussian learning machine that clarifies that learning hidden features is conceptually different from parameter estimation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
