Dense Associative Memories or modern Hopfield networks permit storage and reliable retrieval of an exponentially large (in the dimension of feature space) number of memories. At the same time, their naive implementation is non-biological, since it seemingly requires the existence of many-body synaptic junctions between the neurons. We show that these models are effective descriptions of a more microscopic (written in terms of biological degrees of freedom) theory that has additional (hidden) neurons and only requires two-body interactions between them. For this reason our proposed microscopic theory is a valid model of large associative memory with a degree of biological plausibility. The dynamics of our network and its reduced dimensional equivalent both minimize energy (Lyapunov) functions. When certain dynamical variables (hidden neurons) are integrated out from our microscopic theory, one can recover many of the models that were previously discussed in the literature, e.g. the model presented in "Hopfield Networks is All You Need" paper. We also provide an alternative derivation of the energy function and the update rule proposed in the aforementioned paper and clarify the relationships between various models of this class.
In their recent paper titled "Large Associative Memory Problem in Neurobiology and Machine Learning" [arXiv:2008.06996] the authors gave a biologically plausible microscopic theory from which one can recover many dense associative memory models discussed in the literature. We show that the layers of the recent "MLP-mixer" [arXiv:2105.01601] as well as the essentially equivalent model in [arXiv:2105.02723] are amongst them.
Simons Center for Systems Biology（西蒙斯系统生物学中心）；Institute for Advanced Study（高等研究院）；Princeton Neuroscience Institute（普林斯顿神经科学研究所）；Princeton University（普林斯顿大学）
A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of models that smoothly interpolates between two limiting cases can be constructed. One limit is referred to as the feature-matching mode of pattern recognition, and the other one as the prototype regime. On the deep learning side of the duality, this family corresponds to feedforward…
MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；IBM Research（IBM研究院）
Dense Associative Memories or Modern Hopfield Networks have many appealing properties of associative memory. They can do pattern completion, store a large number of memories, and can be described using a recurrent neural network with a degree of biological plausibility and rich feedback between the neurons. At the same time, up until now all the models of this class have had only one hidden layer, and have only been formulated with densely connected network architectures, two aspects that hinder their machine learning applications. This paper tackles this gap and describes a fully recurrent mo…
The associative memory model is a typical neural network model, which can store discretely distributed fixed-point attractors as memory patterns. When the network stores the memory patterns extensively, however, the model has other attractors besides the memory patterns. These attractors are called spurious memories. Both spurious states and memory states are equilibrium, so there is little difference between their dynamics. Recent physiological experiments have shown that short-term dynamic synapse called synaptic depression decreases its transmission efficacy to postsynaptic neurons accordin…
Memorization to Generalization: Emergence of Diffusion Models from Associative Memory
Department of Computer Science, Rensselaer Polytechnic Institute ( RPI )（计算机科学系，罗切斯特理工学院（RPI））；Jheronimus Academy of Data Science, Tilburg University（数据科学学院，蒂尔堡大学）；Department of Physics, University of Rome Sapienza（物理系，罗马萨皮恩扎大学）；Donders Institute for Brain, Cognition, and Behaviour, Radboud University（大脑、认知与行为研究所，拉德堡德大学）；IBM Research（IBM研究院）
We study a model of associative memory based on a neural network with small-world structure. The efficacy of the network to retrieve one of the stored patterns exhibits a phase transition at a finite value of the disorder. The more ordered networks are unable to recover the patterns, and are always attracted to mixture states. Besides, for a range of the number of stored patterns, the efficacy has a maximum at an intermediate value of the disorder. We also give a statistical characterization of the attractors for all values of the disorder of the network.
Conversion of temporal to spatial correlations in the cortex is one of the most intriguing functions in the brain. The learning at synapses triggering the correlation conversion can take place in a wide integration window, whose influence on the correlation conversion remains elusive. Here, we propose a generalized associative memory model with arbitrary Hebbian length. The model can be analytically solved, and predicts that a small Hebbian length can already significantly enhance the correlation conversion, i.e., the stimulus-induced attractor can be highly correlated with a significant numbe…
Firing Rate Models as Associative Memory: Excitatory-Inhibitory Balance for Robust Retrieval
Department of Information Engineering（信息工程系）；Università degli Studi di Padova（帕多瓦大学）；Center for Control, Dynamical Systems and Computation（控制、动态系统与计算中心）；University of California at Santa Barbara（加州大学圣巴巴拉分校）
One of the most well established brain principles, hebbian learning, has led to the theoretical concept of neural assemblies. Based on it, many interesting brain theories have spawned. Palm's work implements this concept through binary associative memory, in a model that not only has a wide cognitive explanatory power but also makes neuroscientific predictions. Yet, associative memory can only work with logarithmic sparse representations, which makes it extremely difficult to apply the model to real data. We propose a biologically plausible network that encodes images into codes that are suita…
Associative Memory and Generative Diffusion in the Zero-noise Limit
Computational & Systems Biology, Sloan Kettering Institute, Memorial Sloan Kettering Cancer Center（计算生物学与系统生物学，斯隆凯特灵研究所，纪念斯隆凯特灵癌症中心）；Tri-Institutional Training Program in Computational Biology & Medicine（计算生物学与医学三机构联合培训计划）
Spike-timing-dependent plasticity(STDP) is a biological process in which the precise order and timing of neuronal spikes affect the degree of synaptic modification. While there have been numerous research focusing on the role of STDP in neural coding, the functional implications of STDP at the macroscopic level in the brain have not been fully explored yet. In this work, we propose a neurodynamical model based on STDP that renders storage and retrieval of a group of associative memories. We showed that the function of STDP at the macroscopic level is to form a "memory plane" in the neural stat…
ISAM-MTL: Cross-subject multi-task learning model with identifiable spikes and associative memory networks
School of Future Technology, South China University of Technology（华南理工大学未来技术学院）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）
A synfire chain is a network that can generate repeated spike patterns with millisecond precision. Although synfire chains with only one activity propagation mode have been intensively analyzed with several neuron models, those with several stable propagation modes have not been thoroughly investigated. By using the leaky integrate-and-fire neuron model, we constructed a layered associative network embedded with memory patterns. We analyzed the network dynamics with the Fokker-Planck equation. First, we addressed the stability of one memory pattern as a propagating spike volley. We showed that…
An associative memory has been discussed of neural networks consisting of spiking N (=100) Hodgkin-Huxley (HH) neurons with time-delayed couplings, which memorize P patterns in their synaptic weights. In addition to excitatory synapses whose strengths are modified after the Willshaw-type learning rule with the 0/1 code for quiescent/active states, the network includes uniform inhibitory synapses which are introduced to reduce cross-talk noises. Our simulations of the HH neuron network for the noise-free state have shown to yield a fairly good performance with the storage capacity of $α_c = P_{…
Semantically-correlated memories in a dense associative model
Institute for Computational and Experimental Research in Mathematics, Brown University（布朗大学计算与实验数学研究所）；SciAI Center, Cornell University（康奈尔大学SciAI中心）；Neural Coding and Brain Computing Unit, OIST Graduate University（冲绳科学技术大学院大学神经编码与脑计算单元）
Hippocampal CA3 is crucial for the formation of long-term associative memory. It has a heavily recurrent connectivity, and memories are thought to be stored as memory engrams in the CA3. However, despite its importance for memory storage and retrieval, spiking network models of the CA3 to date are relatively small-scale, and exist as only proof-of-concept models. Specifically, how neurogenesis in the dentate gyrus affects memory encoding and retrieval in the CA3 is not studied in such spiking models. Our work is the first to develop a biologically plausible spiking neural network model of hipp…
Several guiding principles for thought processes are proposed and a neural-network-type model implementing these principles is presented and studied. We suggest to consider thinking within an associative network built-up of overlapping memory states. We consider a homogeneous associative network as biological considerations rule out distinct conjunction units between the information (the memories) stored in the brain. We therefore propose that memory states have a dual functionality: They represent on one side the stored information and serve, on the other side, as the associative links in bet…
When someone mentions the name of a known person we immediately recall her face and possibly many other traits. This is because we possess the so-called associative memory, that is the ability to correlate different memories to the same fact or event. Associative memory is such a fundamental and encompassing human ability (and not just human) that the network of neurons in our brain must perform it quite easily. The question is then whether electronic neural networks (electronic schemes that act somewhat similarly to human brains) can be built to perform this type of function. Although the fie…
We present an algorithm to store binary memories in a Hopfield neural network using minimum probability flow, a recent technique to fit parameters in energy-based probabilistic models. In the case of memories without noise, our algorithm provably achieves optimal pattern storage (which we show is at least one pattern per neuron) and outperforms classical methods both in speed and memory recovery. Moreover, when trained on noisy or corrupted versions of a fixed set of binary patterns, our algorithm finds networks which correctly store the originals. We also demonstrate this finding visually wit…
A synfire chain is a simple neural network model which can propagate stable synchronous spikes called a pulse packet and widely researched. However how synfire chains coexist in one network remains to be elucidated. We have studied the activity of a layered associative network of Leaky Integrate-and-Fire neurons in which connection we embed memory patterns by the Hebbian Learning. We analyzed their activity by the Fokker-Planck method. In our previous report, when a half of neurons belongs to each memory pattern (memory pattern rate $F=0.5$), the temporal profiles of the network activity is sp…
Neurodegenerative diseases and traumatic brain injuries (TBI) are among the main causes of cognitive dysfunction in humans. Both manifestations exhibit the extensive presence of focal axonal swellings (FAS). FAS compromises the information encoded in spike trains, thus leading to potentially severe functional deficits. Complicating our understanding of the impact of FAS is our inability to access small scale injuries with non-invasive methods, the overall complexity of neuronal pathologies, and our limited knowledge of how networks process biological signals. Building on Hopfield's pioneering …
The University of Chicago（芝加哥大学）；The James Franck Institute, The University of Chicago（芝加哥大学詹姆斯·弗兰克研究所）；Max Planck Institute for Neurobiology of Behavior—caesar（马克斯·普朗克行为神经生物学研究所——caesar）
