Sapienza Università di Roma(罗马第一大学);GNFM-INdAM Sezione di Roma(意大利国家数学物理研究所罗马分会);Università del Salento(萨兰托大学);GNFM-INdAM Sezione di Lecce(意大利国家数学物理研究所莱切分会);INFN, Istituto Nazionale di Fisica Nucleare(意大利国家核物理研究所)
Elena Agliari, Francesco Alemanno, Adriano Barra, Alberto Fachechi
Recently a daily routine for associative neural networks has been proposed: the network Hebbian-learns during the awake state (thus behaving as a standard Hopfield model), then, during its sleep state, optimizing information storage, it consolidates pure patterns and removes spurious ones: this forces the synaptic matrix to collapse to the projector one (ultimately approaching the Kanter-Sompolinksy model). This procedure keeps the learning Hebbian-based (a biological must) but, by taking advantage of a (properly stylized) sleep phase, still reaches the maximal critical capacity (for symmetric interactions). So far this emerging picture (as well as the bulk of papers on unlearning techniques) was supported solely by mathematically-challenging routes, e.g. mainly replica-trick analysis and numerical simulations: here we rely extensively on Guerra's interpolation techniques developed for neural networks and, in particular, we extend the generalized stochastic stability approach to the case. Confining our description within the replica symmetric approximation (where the previous ones lie), the picture painted regarding this generalization (and the previously existing variations on theme) is here entirely confirmed. Further, still relying on Guerra's schemes, we develop a systematic fluctuation analysis to check where ergodicity is broken (an analysis entirely absent in previous investigations). We find that, as long as the network is awake, ergodicity is bounded by the Amit-Gutfreund-Sompolinsky critical line (as it should), but, as the network sleeps, sleeping destroys spin glass states by extending both the retrieval as well as the ergodic region: after an entire sleeping session the solely surviving regions are retrieval and ergodic ones and this allows the network to achieve the perfect retrieval regime (the number of storable patterns equals the number of neurons in the network).
Do Hopfield Networks Dream of Stored Patterns? A Statistical-Mechanical Theory of Dreaming in Multidirectional Associative Memories
Sapienza Università di Roma(罗马大学);Istituto Nazionale di Fisica Nucleare(意大利国家核物理研究所);Istituto Nazionale d’Alta Matematica, GNFM(意大利高等数学研究所);Università del Salento(莱切萨伦托大学)
The Gaussian equivalence of generative models for learning with shallow neural networks
International School of Advanced Studies (SISSA)(国际高等研究学院(SISSA));Duke University(杜克大学);Université PSL(巴黎文理研究大学);Sorbonne Université(索邦大学);Université Paris-Diderot(巴黎第七大学)