A large number of neural network models of associative memory have been proposed in the literature. These include the classical Hopfield networks (HNs), sparse distributed memories (SDMs), and more recently the modern continuous Hopfield networks (MCHNs), which possesses close links with self-attention in machine learning. In this paper, we propose a general framework for understanding the operation of such memory networks as a sequence of three operations: similarity, separation, and projection. We derive all these memory models as instances of our general framework with differing similarity and separation functions. We extend the mathematical framework of Krotov et al (2020) to express general associative memory models using neural network dynamics with only second-order interactions between neurons, and derive a general energy function that is a Lyapunov function of the dynamics. Finally, using our framework, we empirically investigate the capacity of using different similarity functions for these associative memory models, beyond the dot product similarity measure, and demonstrate empirically that Euclidean or Manhattan distance similarity metrics perform substantially better in practice on many tasks, enabling a more robust retrieval and higher memory capacity than existing models.
Redundancy Maximization as a Principle of Associative Memory Learning in Hopfield Networks
Complex Systems Theory, Max Planck Institute for Dynamics and Self-Organization（复杂系统理论，马克斯·普朗克动态与自组织研究所）；Faculty of Physics, Institute for the Dynamics of Complex Systems, University of Göttingen（物理系，复杂系统动力学研究所，哥廷根大学）；Göttingen Campus Institute for Dynamics of Biological Networks, University of Göttingen（哥廷根校区生物网络动力学研究所，哥廷根大学）；Champalimaud Centre for the Unknown, Lisbon, Portugal（查帕利马德未知中心，里斯本，葡萄牙）；Max Planck Institute for Dynamics and Self-Organization（马克斯·普朗克动态与自组织研究所）
Unsupervised learning in a generalized Hopfield associative-memory network is investigated in this work. First, we prove that the (generalized) Hopfield model is equivalent to a semi-restricted Boltzmann machine with a layer of visible neurons and another layer of hidden binary neurons, so it could serve as the building block for a multilayered deep-learning system. We then demonstrate that the Hopfield network can learn to form a faithful internal representation of the observed samples, with the learned memory patterns being prototypes of the input data. Furthermore, we propose a spectral met…
Hopfield neural networks are a possible basis for modelling associative memory in living organisms. After summarising previous studies in the field, we take a new look at learning rules, exhibiting them as descent-type algorithms for various cost functions. We also propose several new cost functions suitable for learning. We discuss the role of biases (the external inputs) in the learning process in Hopfield networks. Furthermore, we apply Newtons method for learning memories, and experimentally compare the performances of various learning rules. Finally, to add to the debate whether allowing …
Associative memories store content in such a way that the content can be later retrieved by presenting the memory with a small portion of the content, rather than presenting the memory with an address as in more traditional memories. Associative memories are used as building blocks for algorithms within database engines, anomaly detection systems, compression algorithms, and face recognition systems. A classical example of an associative memory is the Hopfield neural network. Recently, Gripon and Berrou have introduced an alternative construction which builds on ideas from the theory of error …
In the present paper, an effort has been made for storing and recalling images with Hopfield Neural Network Model of auto-associative memory. Images are stored by calculating a corresponding weight matrix. Thereafter, starting from an arbitrary configuration, the memory will settle on exactly that stored image, which is nearest to the starting configuration in terms of Hamming distance. Thus given an incomplete or corrupted version of a stored image, the network is able to recall the corresponding original image. The storing of the objects has been performed according to the Hopfield algorithm…
Hopfield networks and Boltzmann machines (BMs) are fundamental energy-based neural network models. Recent studies on modern Hopfield networks have broaden the class of energy functions and led to a unified perspective on general Hopfield networks including an attention module. In this letter, we consider the BM counterparts of modern Hopfield networks using the associated energy functions, and study their salient properties from a trainability perspective. In particular, the energy function corresponding to the attention module naturally introduces a novel BM, which we refer to as the attentio…
Dynamics of Structured Complex-Valued Hopfield Neural Networks
Mahindra University（马hindra大学）；Ecole Centrale School of Engineering（Ecole Centrale工程学院）；Universidade Estadual de Campinas（圣保罗州立大学）；International Institute of Information Technology（国际信息科技研究所）；Indian Institute of Technology（印度理工学院）
Thermodynamics of bidirectional associative memories
Università del Salento（萨伦托大学）；Istituto Nazionale di Fisica Nucleare（国家核物理研究所）；Universidad Complutense de Madrid（马德里康普顿斯大学）；Université Paris-Saclay（巴黎萨克雷大学）；CNRS（法国国家科学研究中心）；INRIA（法国国家信息与自动化研究所）
Johannes Kepler University Linz（林茨约翰开普勒大学）；Institute of Advanced Research in Artificial Intelligence (IARAI)（高级人工智能研究所（IARAI））；University of Oslo（奥斯陆大学）
We introduce a modern Hopfield network with continuous states and a corresponding update rule. The new Hopfield network can store exponentially (with the dimension of the associative space) many patterns, retrieves the pattern with one update, and has exponentially small retrieval errors. It has three types of energy minima (fixed points of the update): (1) global fixed point averaging over all patterns, (2) metastable states averaging over a subset of patterns, and (3) fixed points which store a single pattern. The new update rule is equivalent to the attention mechanism used in transformers.…
Phases in a class of associative memories via hidden neurons
CyberAgent AI Lab（CyberAgent AI实验室）；RIKEN iTHEMS（RIKEN跨学科理论科学研究所）；Graduate School of Artificial Intelligence and Science, Rikkyo University（立教大学人工智能与科学研究科）
The network embedding task is to represent the node in the network as a low-dimensional vector while incorporating the topological and structural information. Most existing approaches solve this problem by factorizing a proximity matrix, either directly or implicitly. In this work, we introduce a network embedding method from a new perspective, which leverages Modern Hopfield Networks (MHN) for associative learning. Our network learns associations between the content of each node and that node's neighbors. These associations serve as memories in the MHN. The recurrent dynamics of the network m…
John A. Paulson School of Engineering and Applied Sciences（约翰·A·保尔森工程与应用科学学院）；Center for Brain Science（脑科学中心）；Department of Physics（物理系）；MIT-IBM Watson AI Lab, IBM Research（MIT-IBM沃森人工智能实验室，IBM研究院）；Kempner Institute for the Study of Natural and Artificial Intelligence（肯普纳自然与人工智能研究所）
MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；IBM Research（IBM研究院）
Dense Associative Memories or Modern Hopfield Networks have many appealing properties of associative memory. They can do pattern completion, store a large number of memories, and can be described using a recurrent neural network with a degree of biological plausibility and rich feedback between the neurons. At the same time, up until now all the models of this class have had only one hidden layer, and have only been formulated with densely connected network architectures, two aspects that hinder their machine learning applications. This paper tackles this gap and describes a fully recurrent mo…
Institute of Mathematics, Statistics, and Scientific Computing（数学、统计与科学计算研究所）；University of Campinas（坎皮纳斯大学）
Recurrent correlation neural networks (RCNNs), introduced by Chiueh and Goodman as an improved version of the bipolar correlation-based Hopfield neural network, can be used to implement high-capacity associative memories. In this paper, we extend the bipolar RCNNs for processing hypercomplex-valued data. Precisely, we present the mathematical background for a broad class of hypercomplex-valued RCNNs. Then, we provide the necessary conditions which ensure that a hypercomplex-valued RCNN always settles at an equilibrium using either synchronous or asynchronous update modes. Examples with bipolar…
McGovern Institute for Brain Research, Massachusetts Institute of Technology（麻省理工学院麦戈文脑科学研究所）；The NSF AI Institute for Artificial Intelligence and Fundamental Interactions (IAIFI), Massachusetts Institute of Technology（麻省理工学院美国国家科学基金会人工智能基础相互作用研究所）
We study the problem of learning associative memory -- a system which is able to retrieve a remembered pattern based on its distorted or incomplete version. Attractor networks provide a sound model of associative memory: patterns are stored as attractors of the network dynamics and associative retrieval is performed by running the dynamics starting from a query pattern until it converges to an attractor. In such models the dynamics are often implemented as an optimization procedure that minimizes an energy function, such as in the classical Hopfield network. In general it is difficult to deriv…
Long-Term Visual Object Tracking with Event Cameras: An Associative Memory Augmented Tracker and A Benchmark Dataset
School of Computer Science and Technology, Anhui University（计算机科学与技术学院，安徽大学）；School of Electronic and Information Engineering, Anhui University（电子与信息工程学院，安徽大学）
