We introduce a bipartite, diluted and frustrated, network as a sparse restricted Boltzman machine and we show its thermodynamical equivalence to an associative working memory able to retrieve multiple patterns in parallel without falling into spurious states typical of classical neural networks. We focus on systems processing in parallel a finite (up to logarithmic growth in the volume) amount of patterns, mirroring the low-level storage of standard Amit-Gutfreund-Sompolinsky theory. Results obtained trough statistical mechanics, signal-to-noise technique and Monte Carlo simulations are overall in perfect agreement and carry interesting biological insights. Indeed, these associative networks pave new perspectives in the understanding of multitasking features expressed by complex systems, e.g. neural and immune networks.
The retrieval capabilities of associative neural networks can be impaired by different kinds of noise: the fast noise (which makes neurons more prone to failure), the slow noise (stemming from interference among stored memories), and synaptic noise (due to possible flaws during the learning or the storing stage). In this work we consider dense associative neural networks, where neurons can interact in $p$-plets, in the absence of fast noise, and we investigate the interplay of slow and synaptic noise. In particular, leveraging on the duality between associative neural networks and restricted B…
In this work we develop analytical techniques to investigate a broad class of associative neural networks set in the high-storage regime. These techniques translate the original statistical-mechanical problem into an analytical-mechanical one which implies solving a set of partial differential equations, rather than tackling the canonical probabilistic route. We test the method on the classical Hopfield model - where the cost function includes only two-body interactions (i.e., quadratic terms) - and on the "relativistic" Hopfield model - where the (expansion of the) cost function includes p-bo…
Associative memory storing an extensive number of patterns based on a network of oscillators with distributed natural frequencies in the presence of external white noise
Partial annealing and pattern decorrelation in associative neural networks
Università del Salento（萨伦托大学）；Istituto Nazionale di Fisica Nucleare, Sezione di Lecce（意大利国家核物理研究所莱切分部）；Sapienza Università di Roma（罗马第一大学）；Istituto Nazionale di Fisica Nucleare, Sezione di Roma 1（意大利国家核物理研究所罗马一分部）；CNR-Nanotec, unità di Lecce（国家研究委员会纳米技术研究所莱切单元）；Université Paris-Saclay（巴黎萨克雷大学）
We derive the Gardner storage capacity for associative networks of threshold linear units, and show that with Hebbian learning they can operate closer to such Gardner bound than binary networks, and even surpass it. This is largely achieved through a sparsification of the retrieved patterns, which we analyze for theoretical and empirical distributions of activity. As reaching the optimal capacity via non-local learning rules like backpropagation requires slow and neurally implausible training procedures, our results indicate that one-shot self-organized Hebbian learning can be just as efficien…
Hopfield model is one of the few neural networks for which analytical results can be obtained. However, most of them are derived under the assumption of random uncorrelated patterns, while in real life applications data to be stored show non-trivial correlations. In the present paper we study how the retrieval capability of the Hopfield network at null temperature is affected by spatial correlations in the data we feed to it. In particular, we use as patterns to be stored the configurations of a linear Ising model at inverse temperature $β$. Exploiting the signal to noise technique we obtain a…
A Federated Many-to-One Hopfield model for associative Neural Networks
Università del Salento（萨莱诺大学）；Istituto Nazionale di Fisica Nucleare (INFN), Sezione di Lecce（意大利国家核物理研究所（INFN）莱切分部）；Istituto Nazionale di Alta Matematica Francesco Severi (INdAM)（弗朗切斯科·塞韦里高等数学国立研究所（INdAM））；Sapienza Università di Roma（罗马智慧大学）
A Hardware-aware Hopfield Network with a Nonlinear Memristor Array for Robust Associative Memory with Superlinear Capacity
Korea Institute of Science and Technology (KIST)（韩国科学技术院）；Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；Seoul National University（首尔大学）；Korea University（韩国大学）；KIST School, University of Science and Technology（韩国科学技术院科学技术研究生院）
We show that macro-molecular self-assembly can recognize and classify high-dimensional patterns in the concentrations of $N$ distinct molecular species. Similar to associative neural networks, the recognition here leverages dynamical attractors to recognize and reconstruct partially corrupted patterns. Traditional parameters of pattern recognition theory, such as sparsity, fidelity, and capacity are related to physical parameters, such as nucleation barriers, interaction range, and non-equilibrium assembly forces. Notably, we find that self-assembly bears greater similarity to continuous attra…
Storing Infinite Dynamical Attractors in Nonreciprocal Associative Neural Networks
BCAM – Basque Center for Applied Mathematics（巴斯克应用数学中心）；IKERBASQUE, Basque Foundation for Science（伊克尔巴斯基克科学基金会）；Biofisika Institute (CSIC, EHU)（生物物理研究所（西班牙国家研究委员会、巴斯克大学））
Unsupervised learning in a generalized Hopfield associative-memory network is investigated in this work. First, we prove that the (generalized) Hopfield model is equivalent to a semi-restricted Boltzmann machine with a layer of visible neurons and another layer of hidden binary neurons, so it could serve as the building block for a multilayered deep-learning system. We then demonstrate that the Hopfield network can learn to form a faithful internal representation of the observed samples, with the learned memory patterns being prototypes of the input data. Furthermore, we propose a spectral met…
MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；IBM Research（IBM研究院）
Dense Associative Memories or Modern Hopfield Networks have many appealing properties of associative memory. They can do pattern completion, store a large number of memories, and can be described using a recurrent neural network with a degree of biological plausibility and rich feedback between the neurons. At the same time, up until now all the models of this class have had only one hidden layer, and have only been formulated with densely connected network architectures, two aspects that hinder their machine learning applications. This paper tackles this gap and describes a fully recurrent mo…
Simons Center for Systems Biology（西蒙斯系统生物学中心）；Institute for Advanced Study（高等研究院）；Princeton Neuroscience Institute（普林斯顿神经科学研究所）；Princeton University（普林斯顿大学）
A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of models that smoothly interpolates between two limiting cases can be constructed. One limit is referred to as the feature-matching mode of pattern recognition, and the other one as the prototype regime. On the deep learning side of the duality, this family corresponds to feedforward…
National Institute of Standards and Technology（美国国家标准与技术研究院）
Simulations of complex-valued Hopfield networks based on spin-torque oscillators can recover phase-encoded images. Sequences of memristor-augmented inverters provide tunable delay elements that implement complex weights by phase shifting the oscillatory output of the oscillators. Pseudo-inverse training suffices to store at least 12 images in a set of 192 oscillators, representing 16$\times$12 pixel images. The energy required to recover an image depends on the desired error level. For the oscillators and circuitry considered here, 5 % root mean square deviations from the ideal image require a…
We introduce a near-term experimental platform for realizing an associative memory. It can simultaneously store many memories by using spinful bosons coupled to a degenerate multimode optical cavity. The associative memory is realized by a confocal cavity QED neural network, with the cavity modes serving as the synapses, connecting a network of superradiant atomic spin ensembles, which serve as the neurons. Memories are encoded in the connectivity matrix between the spins, and can be accessed through the input and output of patterns of light. Each aspect of the scheme is based on recently demo…
Firing Rate Models as Associative Memory: Excitatory-Inhibitory Balance for Robust Retrieval
Department of Information Engineering（信息工程系）；Università degli Studi di Padova（帕多瓦大学）；Center for Control, Dynamical Systems and Computation（控制、动态系统与计算中心）；University of California at Santa Barbara（加州大学圣巴巴拉分校）
MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；IBM Research（IBM研究院）；Princeton Neuroscience Institute（普林斯顿神经科学研究所）；Princeton University（普林斯顿大学）
Dense Associative Memories or modern Hopfield networks permit storage and reliable retrieval of an exponentially large (in the dimension of feature space) number of memories. At the same time, their naive implementation is non-biological, since it seemingly requires the existence of many-body synaptic junctions between the neurons. We show that these models are effective descriptions of a more microscopic (written in terms of biological degrees of freedom) theory that has additional (hidden) neurons and only requires two-body interactions between them. For this reason our proposed microscopic …
