Learning deep representations to solve complex machine learning tasks has become the prominent trend in the past few years. Indeed, Deep Neural Networks are now the golden standard in domains as various as computer vision, natural language processing or even playing combinatorial games. However, problematic limitations are hidden behind this surprising universal capability. Among other things, explainability of the decisions is a major concern, especially since deep neural networks are made up of a very large number of trainable parameters. Moreover, computational complexity can quickly become a problem, especially in contexts constrained by real time or limited resources. Therefore, understanding how information is stored and the impact this storage can have on the system remains a major and open issue. In this chapter, we introduce a method to transform deep neural network models into deep associative memories, with simpler, more explicable and less expensive operations. We show through experiments that these transformations can be done without penalty on predictive performance. The resulting deep associative memories are excellent candidates for artificial intelligence that is easier to theorize and manipulate.
University of Oxford（牛津大学）；Hebei University of Technology（河北工业大学）
Associative memories in the brain receive and store patterns of activity registered by the sensory neurons, and are able to retrieve them when necessary. Due to their importance in human intelligence, computational models of associative memories have been developed for several decades now. They include autoassociative memories, which allow for storing data points and retrieving a stored data point $s$ when provided with a noisy or partial variant of $s$, and heteroassociative memories, able to store and recall multi-modal data. In this paper, we present a novel neural model for realizing assoc…
An ensemble method should cleverly combine a group of base classifiers to yield an improved classifier. The majority vote is an example of a methodology used to combine classifiers in an ensemble method. In this paper, we propose to combine classifiers using an associative memory model. Precisely, we introduce ensemble methods based on recurrent correlation associative memories (RCAMs) for binary classification problems. We show that an RCAM-based ensemble classifier can be viewed as a majority vote classifier whose weights depend on the similarity between the base classifiers and the resultin…
In this paper, we introduce a novel layer designed to be used as the output of pre-trained neural networks in the context of classification. Based on Associative Memories, this layer can help design Deep Neural Networks which support incremental learning and that can be (partially) trained in real time on embedded devices. Experiments on the ImageNet dataset and other different domain specific datasets show that it is possible to design more flexible and faster-to-train Neural Networks at the cost of a slight decrease in accuracy.
Telecom Bretagne（布列塔尼电信学校）；University of Münster（明斯特大学）；Université de Bretagne Occidentale（布列塔尼西方大学）
Nearest neighbor search is a very active field in machine learning for it appears in many application cases, including classification and object retrieval. In its canonical version, the complexity of the search is linear with both the dimension and the cardinal of the collection of vectors the search is performed in. Recently many works have focused on reducing the dimension of vectors using quantization techniques or hashing, while providing an approximate result. In this paper we focus instead on tackling the cardinal of the collection of vectors. Namely, we introduce a technique that partit…
Hyperparameter Transfer for Dense Associative Memories
Rudolf Peierls Centre for Theoretical Physics, University of Oxford, Oxford OX1 3PU, UK（鲁道夫·皮尔士理论物理中心，牛津大学，牛津 OX1 3PU, 英国）；Dynamical Mind, IBM Research（动态思维，IBM研究）；Princeton ORFE（普林斯顿ORFE）
School of Computer；Communication Sciences Ecole Polyetechnique Federale de Lausanne (EPFL) Switzerland
We propose a novel architecture to design a neural associative memory that is capable of learning a large number of patterns and recalling them later in presence of noise. It is based on dividing the neurons into local clusters and parallel plains, very similar to the architecture of the visual cortex of macaque brain. The common features of our proposed architecture with those of spatially-coupled codes enable us to show that the performance of such networks in eliminating noise is drastically better than the previous approaches while maintaining the ability of learning an exponentially large…
Semantically-correlated memories in a dense associative model
Institute for Computational and Experimental Research in Mathematics, Brown University（布朗大学计算与实验数学研究所）；SciAI Center, Cornell University（康奈尔大学SciAI中心）；Neural Coding and Brain Computing Unit, OIST Graduate University（冲绳科学技术大学院大学神经编码与脑计算单元）
National Institute of Standards and Technology（美国国家标准与技术研究院）
Simulations of complex-valued Hopfield networks based on spin-torque oscillators can recover phase-encoded images. Sequences of memristor-augmented inverters provide tunable delay elements that implement complex weights by phase shifting the oscillatory output of the oscillators. Pseudo-inverse training suffices to store at least 12 images in a set of 192 oscillators, representing 16$\times$12 pixel images. The energy required to recover an image depends on the desired error level. For the oscillators and circuitry considered here, 5 % root mean square deviations from the ideal image require a…
MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；IBM Research（IBM研究院）
Dense Associative Memories or Modern Hopfield Networks have many appealing properties of associative memory. They can do pattern completion, store a large number of memories, and can be described using a recurrent neural network with a degree of biological plausibility and rich feedback between the neurons. At the same time, up until now all the models of this class have had only one hidden layer, and have only been formulated with densely connected network architectures, two aspects that hinder their machine learning applications. This paper tackles this gap and describes a fully recurrent mo…
Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplings
Physics Department, Sapienza University of Rome（罗马大学物理系）；Institute of Nanotechnology, National Research Council of Italy, CNR-NANOTEC, Rome Unit（意大利国家研究委员会纳米技术研究所，CNR-NANOTEC，罗马分部）
Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data
Department of Computer Science, Rensselaer Polytechnic Institute ( RPI )（里士满理工学院计算机科学系）；Donders Institute for Brain, Cognition, and Behaviour, Radboud University（拉布德大学脑科学与行为研究所）；Laboratoire de Physique Théorique et Modélisation, CY Cergy Paris Université（巴黎中央理工学院物理理论与建模实验室）
Posgrado en Ciencia e Ingeniería de la Computación, UNAM（墨西哥国立自治大学计算机科学与工程研究生院）；Universidad Nacional Autónoma de México（墨西哥国立自治大学）；IIMAS（数学研究中心）；Universidad de Guadalajara（瓜达拉哈拉大学）
VERSES AI Research Lab（VERSES AI研究实验室）；Medical Research Council Brain Network Dynamics Unit（医学研究委员会脑网络动力学单元）；University of Oxford（牛津大学）；Institute of Logic and Computation（逻辑与计算研究所）；Vienna University of Technology（维也纳技术大学）；Department of Computer Science, University of Oxford（牛津大学计算机科学系）
Phases in a class of associative memories via hidden neurons
CyberAgent AI Lab（CyberAgent AI实验室）；RIKEN iTHEMS（RIKEN跨学科理论科学研究所）；Graduate School of Artificial Intelligence and Science, Rikkyo University（立教大学人工智能与科学研究科）
Associative memory plays an important role in human intelligence and its mechanisms have been linked to attention in machine learning. While the machine learning community's interest in associative memories has recently been rekindled, most work has focused on memory recall ($read$) over memory learning ($write$). In this paper, we present BayesPCN, a hierarchical associative memory capable of performing continual one-shot memory writes without meta-learning. Moreover, BayesPCN is able to gradually forget past observations ($forget$) to free its memory. Experiments show that BayesPCN can recal…
Thermodynamics of bidirectional associative memories
Università del Salento（萨伦托大学）；Istituto Nazionale di Fisica Nucleare（国家核物理研究所）；Universidad Complutense de Madrid（马德里康普顿斯大学）；Université Paris-Saclay（巴黎萨克雷大学）；CNRS（法国国家科学研究中心）；INRIA（法国国家信息与自动化研究所）
Carnegie Mellon University（卡内基梅隆大学）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Massachusetts Institute of Technology（麻省理工学院）
An associative memory is a framework of content-addressable memory that stores a collection of message vectors (or a dataset) over a neural network while enabling a neurally feasible mechanism to recover any message in the dataset from its noisy version. Designing an associative memory requires addressing two main tasks: 1) learning phase: given a dataset, learn a concise representation of the dataset in the form of a graphical model (or a neural network), 2) recall phase: given a noisy version of a message vector from the dataset, output the correct message vector via a neurally feasible algo…
MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；IBM Research（IBM研究院）；Princeton Neuroscience Institute（普林斯顿神经科学研究所）；Princeton University（普林斯顿大学）
Dense Associative Memories or modern Hopfield networks permit storage and reliable retrieval of an exponentially large (in the dimension of feature space) number of memories. At the same time, their naive implementation is non-biological, since it seemingly requires the existence of many-body synaptic junctions between the neurons. We show that these models are effective descriptions of a more microscopic (written in terms of biological degrees of freedom) theory that has additional (hidden) neurons and only requires two-body interactions between them. For this reason our proposed microscopic …
Intel AI Lab, Artificial Intelligence Products Group, Intel Corporation（英特尔人工智能实验室，人工智能产品组，英特尔公司）
Associative memory using fast weights is a short-term memory mechanism that substantially improves the memory capacity and time scale of recurrent neural networks (RNNs). As recent studies introduced fast weights only to regular RNNs, it is unknown whether fast weight memory is beneficial to gated RNNs. In this work, we report a significant synergy between long short-term memory (LSTM) networks and fast weight associative memories. We show that this combination, in learning associative retrieval tasks, results in much faster training and lower test error, a performance boost most prominent at …
CY Cergy Paris Université（CY塞尔吉-巴黎大学）；ENSEA（国立高等电子及应用工程师学院）；CNRS（法国国家科学研究中心）；ETIS UMR 8051（信息处理与系统实验室（联合研究单位8051））
In this article, we propose a variational inference formulation of auto-associative memories, allowing us to combine perceptual inference and memory retrieval into the same mathematical framework. In this formulation, the prior probability distribution onto latent representations is made memory dependent, thus pulling the inference process towards previously stored representations. We then study how different neural network approaches to variational inference can be applied in this framework. We compare methods relying on amortized inference such as Variational Auto Encoders and methods relyin…
Institute of Mathematics, Statistics, and Scientific Computing（数学、统计与科学计算研究所）；University of Campinas（坎皮纳斯大学）
Recurrent correlation neural networks (RCNNs), introduced by Chiueh and Goodman as an improved version of the bipolar correlation-based Hopfield neural network, can be used to implement high-capacity associative memories. In this paper, we extend the bipolar RCNNs for processing hypercomplex-valued data. Precisely, we present the mathematical background for a broad class of hypercomplex-valued RCNNs. Then, we provide the necessary conditions which ensure that a hypercomplex-valued RCNN always settles at an equilibrium using either synchronous or asynchronous update modes. Examples with bipolar…
Memorization to Generalization: Emergence of Diffusion Models from Associative Memory
Department of Computer Science, Rensselaer Polytechnic Institute ( RPI )（计算机科学系，罗切斯特理工学院（RPI））；Jheronimus Academy of Data Science, Tilburg University（数据科学学院，蒂尔堡大学）；Department of Physics, University of Rome Sapienza（物理系，罗马萨皮恩扎大学）；Donders Institute for Brain, Cognition, and Behaviour, Radboud University（大脑、认知与行为研究所，拉德堡德大学）；IBM Research（IBM研究院）
The network embedding task is to represent the node in the network as a low-dimensional vector while incorporating the topological and structural information. Most existing approaches solve this problem by factorizing a proximity matrix, either directly or implicitly. In this work, we introduce a network embedding method from a new perspective, which leverages Modern Hopfield Networks (MHN) for associative learning. Our network learns associations between the content of each node and that node's neighbors. These associations serve as memories in the MHN. The recurrent dynamics of the network m…
MRC Brain Network Dynamics Unit（MRC脑网络动力学单元）；University of Oxford（牛津大学）
A large number of neural network models of associative memory have been proposed in the literature. These include the classical Hopfield networks (HNs), sparse distributed memories (SDMs), and more recently the modern continuous Hopfield networks (MCHNs), which possesses close links with self-attention in machine learning. In this paper, we propose a general framework for understanding the operation of such memory networks as a sequence of three operations: similarity, separation, and projection. We derive all these memory models as instances of our general framework with differing similarity …
Recent research developing neural network architectures with external memory have often used the benchmark bAbI question and answering dataset which provides a challenging number of tasks requiring reasoning. Here we employed a classic associative inference task from the memory-based reasoning neuroscience literature in order to more carefully probe the reasoning capacity of existing memory-augmented architectures. This task is thought to capture the essence of reasoning -- the appreciation of distant relationships among elements distributed across multiple facts or memories. Surprisingly, we …
On associative neural networks for sparse patterns with huge capacities
Fachbereich Mathematik und Informatik, University of M\"unster, Einsteinstra e 62, 48149 M\"unster, Germany；Univ Brest, CNRS UMR 6205, Laboratoire de Math\'ematiques de Bretagne Atlantique, 6, avenue Victor Le Gorgeu CS 93837 F-29238 BREST Cedex 3 France
Simons Center for Systems Biology（西蒙斯系统生物学中心）；Institute for Advanced Study（高等研究院）；Princeton Neuroscience Institute（普林斯顿神经科学研究所）；Princeton University（普林斯顿大学）
A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of models that smoothly interpolates between two limiting cases can be constructed. One limit is referred to as the feature-matching mode of pattern recognition, and the other one as the prototype regime. On the deep learning side of the duality, this family corresponds to feedforward…
Inferring missing facts in temporal knowledge graphs is a critical task and has been widely explored. Extrapolation in temporal reasoning tasks is more challenging and gradually attracts the attention of researchers since no direct history facts for prediction. Previous works attempted to apply evolutionary representation learning to solve the extrapolation problem. However, these techniques do not explicitly leverage various time-aware attribute representations, i.e. the reasoning performance is significantly affected by the history length. To alleviate the time dependence when reasoning futu…
We study the problem of learning associative memory -- a system which is able to retrieve a remembered pattern based on its distorted or incomplete version. Attractor networks provide a sound model of associative memory: patterns are stored as attractors of the network dynamics and associative retrieval is performed by running the dynamics starting from a query pattern until it converges to an attractor. In such models the dynamics are often implemented as an optimization procedure that minimizes an energy function, such as in the classical Hopfield network. In general it is difficult to deriv…
Redundancy Maximization as a Principle of Associative Memory Learning in Hopfield Networks
Complex Systems Theory, Max Planck Institute for Dynamics and Self-Organization（复杂系统理论，马克斯·普朗克动态与自组织研究所）；Faculty of Physics, Institute for the Dynamics of Complex Systems, University of Göttingen（物理系，复杂系统动力学研究所，哥廷根大学）；Göttingen Campus Institute for Dynamics of Biological Networks, University of Göttingen（哥廷根校区生物网络动力学研究所，哥廷根大学）；Champalimaud Centre for the Unknown, Lisbon, Portugal（查帕利马德未知中心，里斯本，葡萄牙）；Max Planck Institute for Dynamics and Self-Organization（马克斯·普朗克动态与自组织研究所）
