arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.07110cond-mat.dis-nnstat.ML
收藏

基于第一性原理的 Hebb 学习

Hebbian Learning from First Principles

Linda Albanese, Adriano Barra, Pierluigi Bianco, Fabrizio Durante, Diego Pallara

AI总结:

本文用 Jaynes 最大熵原理从第一性原理推导 Hopfield 模型及其稠密推广的 Hebb 学习规则,证明其在大数据极限恢复经典存储规则与统计力学描述。

摘要:

最近,Hopfield 神经网络模型(及其稠密推广)最初的存储规则,通过在监督和无监督协议下假定其 Hamiltonian 表达式,已被转化为一条真正的 Hebb 学习规则。在这些笔记中,我们首先借助 Jaynes 式的最大熵极值化获得这些显式表达式。除了为这些 Hebb 学习配方提供形式化推导之外,该构造还强调了熵极值化中的 Lagrangian 约束如何迫使网络的输出体现神经关联:这些关联试图模仿提供给网络训练的数据集中隐藏的经验对应物,而且网络越稠密,它能够捕捉的关联就越长程。接下来,我们证明,在大数据极限下,无论是否存在教师(或缺少教师),这些 Hebb 学习规则不仅收敛到 Hopfield 模型最初的存储规则,而且其相关自由能(因而由 Amit、Gutfreund 和 Sompolinsky 给出的统计力学图景)也得到完全恢复。作为附带结果,我们展示了统计力学习惯使用的标准 Cost 函数(Hamiltonian)与机器学习术语中偏好的二次 Loss 函数之间的数学等价性。文中还给出了关于指数 Hopfield 模型(作为密度发散的稠密网络极限)和半监督协议的评述。

英文摘要:

Recently, the original storage prescription for the Hopfield model of neural networks -- as well as for its dense generalizations -- has been turned into a genuine Hebbian learning rule by postulating the expression of its Hamiltonian for both the supervised and unsupervised protocols. In these notes, first, we obtain these explicit expressions by relying upon maximum entropy extremization à la Jaynes. Beyond providing a formal derivation of these recipes for Hebbian learning, this construction also highlights how Lagrangian constraints within entropy extremization force network's outcomes on neural correlations: these try to mimic the empirical counterparts hidden in the datasets provided to the network for its training and, the denser the network, the longer the correlations that it is able to capture. Next, we prove that, in the big data limit, whatever the presence of a teacher (or its lacking), not only these Hebbian learning rules converge to the original storage prescription of the Hopfield model but also their related free energies (and, thus, the statistical mechanical picture provided by Amit, Gutfreund and Sompolinsky is fully recovered). As a sideline, we show mathematical equivalence among standard Cost functions (Hamiltonian), preferred in Statistical Mechanical jargon, and quadratic Loss Functions, preferred in Machine Learning terminology. Remarks on the exponential Hopfield model (as the limit of dense networks with diverging density) and semi-supervised protocols are also provided.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑