We propose a modification of the cost function of the Hopfield model whose salient features shine in its Taylor expansion and result in more than pairwise interactions with alternate signs, suggesting a unified framework for handling both with deep learning and network pruning. In our analysis, we heavily rely on the Hamilton-Jacobi correspondence relating the statistical model with a mechanical system. In this picture, our model is nothing but the relativistic extension of the original Hopfield model (whose cost function is a quadratic form in the Mattis magnetization which mimics the non-relativistic Hamiltonian for a free particle). We focus on the low-storage regime and solve the model analytically by taking advantage of the mechanical analogy, thus obtaining a complete characterization of the free energy and the associated self-consistency equations in the thermodynamic limit. On the numerical side, we test the performances of our proposal with MC simulations, showing that the stability of spurious states (limiting the capabilities of the standard Hebbian construction) is sensibly reduced due to presence of unlearning contributions in this extended framework.
Do Hopfield Networks Dream of Stored Patterns? A Statistical-Mechanical Theory of Dreaming in Multidirectional Associative Memories
Sapienza Università di Roma(罗马大学);Istituto Nazionale di Fisica Nucleare(意大利国家核物理研究所);Istituto Nazionale d’Alta Matematica, GNFM(意大利高等数学研究所);Università del Salento(莱切萨伦托大学)
Semi-supervised Hopfield model: Theoretical and Numerical results
Università del Salento(萨伦托大学);Dipartimento di Matematica e Fisica "Ennio De Giorgi", Università del Salento(萨伦托大学恩尼奥·德乔治数学与物理系);Istituto Nazionale di Alta Matematica Francesco Severi (INdAM)(弗朗切斯科·塞韦里高等数学国家研究所);Istituto Nazionale di Fisica Nucleare, Sezione di Lecce(意大利国家核物理研究所莱切分部);Sapienza Università di Roma(罗马大学);Dipartimento di Scienze di Base e Applicazioni all’Ingegneria, Sapienza Università di Roma(罗马大学基础科学与工程应用系);Dipartimento di Matematica, Sapienza Università di Roma(罗马大学数学系)