We investigate a new method to augment recurrent neural networks with extra memory without increasing the number of network parameters. The system has an associative memory based on complex-valued vectors and is closely related to Holographic Reduced Representations and Long Short-Term Memory networks. Holographic Reduced Representations have limited capacity: as they store more information, each retrieval becomes noisier due to interference. Our system in contrast creates redundant copies of stored information, which enables retrieval with reduced noise. Experiments demonstrate faster learning on multiple memorization tasks.
We propose an in silico molecular associative memory model for pattern learning, storage and denoising using Pairwise Markov Random Field (PMRF) model. Our PMRF-based molecular associative memory model extracts locally distributed features from the exposed examples, learns and stores the patterns in the molecular associative memory and denoises the given noisy patterns via DNA computation based operations. Thus, our computational molecular model demonstrates the functionalities of content-addressability of human memory. Our molecular simulation results show that the averaged mean squared error…
In this paper, we present a neural network system related to about memory and recall that consists of one neuron group (the "cue ball") and a one-layer neural net (the "recall net"). This system realizes the bidirectional memorization learning between one cue neuron in the cue ball and the neurons in the recall net. It can memorize many patterns and recall these patterns or those that are similar at any time. Furthermore, the patterns are recalled at most the same time. This model's recall situation seems to resemble human recall of a variety of similar things almost simultaneously when one th…
Spike Timing Dependent Plasticity is form of learning that has been demonstrated in real cortical tissue, but attempts to use it for artificial systems have not produced good results. This paper seeks to remedy this with two significant advances. The first is the development a simple stochastic learning rule called cyclic STDP that can extract patterns encoded in the precise spiking times of a group of neurons. We show that a population of neurons endowed with this learning rule can act as an effective short-term associative memory, storing and reliably recalling a large set of pattern associa…
The Swiss AI Lab IDSIA / USI / SUPSI（瑞士人工智能实验室IDSIA / 瑞士意大利语区大学 / 瑞士南部应用科学大学）；Microsoft Research（微软研究院）
Humans can quickly associate stimuli to solve problems in novel contexts. Our novel neural network model learns state representations of facts that can be composed to perform such associative inference. To this end, we augment the LSTM model with an associative memory, dubbed Fast Weight Memory (FWM). Through differentiable operations at every step of a given input sequence, the LSTM updates and maintains compositional associations stored in the rapidly changing FWM weights. Our model is trained end-to-end by gradient descent and yields excellent performance on compositional language reasoning…
One of the most well established brain principles, hebbian learning, has led to the theoretical concept of neural assemblies. Based on it, many interesting brain theories have spawned. Palm's work implements this concept through binary associative memory, in a model that not only has a wide cognitive explanatory power but also makes neuroscientific predictions. Yet, associative memory can only work with logarithmic sparse representations, which makes it extremely difficult to apply the model to real data. We propose a biologically plausible network that encodes images into codes that are suita…
This paper presents the design of an associative memory with feedback that is capable of on-line temporal sequence learning. A framework for on-line sequence learning has been proposed, and different sequence learning models have been analysed according to this framework. The network model is an associative memory with a separate store for the sequence context of a symbol. A sparse distributed memory is used to gain scalability. The context store combines the functionality of a neural layer with a shift register. The sensitivity of the machine to the sequence context is controllable, resulting…
This article underlines the learning and discrimination capabilities of a model of associative memory based on artificial networks of spiking neurons. Inspired from neuropsychology and neurobiology, the model implements top-down modulations, as in neocortical layer V pyramidal neurons, with a learning rule based on synaptic plasticity (STDP), for performing a multimodal association learning task. A temporal correlation method of analysis proves the ability of the model to associate specific activity patterns to different samples of stimulation. Even in the absence of initial learning and with …
Associative memories store content in such a way that the content can be later retrieved by presenting the memory with a small portion of the content, rather than presenting the memory with an address as in more traditional memories. Associative memories are used as building blocks for algorithms within database engines, anomaly detection systems, compression algorithms, and face recognition systems. A classical example of an associative memory is the Hopfield neural network. Recently, Gripon and Berrou have introduced an alternative construction which builds on ideas from the theory of error …
Despite recent progress in memory augmented neural network (MANN) research, associative memory networks with a single external memory still show limited performance on complex relational reasoning tasks. Especially the content-based addressable memory networks often fail to encode input data into rich enough representation for relational reasoning and this limits the relation modeling performance of MANN for long temporal sequence data. To address these problems, here we introduce a novel Distributed Associative Memory architecture (DAM) with Memory Refreshing Loss (MRL) which enhances the rel…
MRC Brain Network Dynamics Unit（MRC脑网络动力学单元）；University of Oxford（牛津大学）
A large number of neural network models of associative memory have been proposed in the literature. These include the classical Hopfield networks (HNs), sparse distributed memories (SDMs), and more recently the modern continuous Hopfield networks (MCHNs), which possesses close links with self-attention in machine learning. In this paper, we propose a general framework for understanding the operation of such memory networks as a sequence of three operations: similarity, separation, and projection. We derive all these memory models as instances of our general framework with differing similarity …
Intel AI Lab, Artificial Intelligence Products Group, Intel Corporation（英特尔人工智能实验室，人工智能产品组，英特尔公司）
Associative memory using fast weights is a short-term memory mechanism that substantially improves the memory capacity and time scale of recurrent neural networks (RNNs). As recent studies introduced fast weights only to regular RNNs, it is unknown whether fast weight memory is beneficial to gated RNNs. In this work, we report a significant synergy between long short-term memory (LSTM) networks and fast weight associative memories. We show that this combination, in learning associative retrieval tasks, results in much faster training and lower test error, a performance boost most prominent at …
Password authentication is a very important system security procedure to gain access to user resources. In the Traditional password authentication methods a server has check the authenticity of the users. In our proposed method users can freely select their passwords from a predefined character set. They can also use a graphical image as password. The password may be a character or an image it will be converted into binary form and the binary values will be normalized. Associative memories have been used recently for password authentication in order to overcome drawbacks of the traditional pas…
National Institute of Science and Technology（国立科学技术学院）
In this paper, an efficient Offline Hand Written Character Recognition algorithm is proposed based on Associative Memory Net (AMN). The AMN used in this work is basically auto associative. The implementation is carried out completely in 'C' language. To make the system perform to its best with minimal computation time, a Parallel algorithm is also developed using an API package OpenMP. Characters are mainly English alphabets (Small (26), Capital (26)) collected from system (52) and from different persons (52). The characters collected from system are used to train the AMN and characters collec…
Many important NLP problems can be posed as dual-sequence or sequence-to-sequence modeling tasks. Recent advances in building end-to-end neural architectures have been highly successful in solving such tasks. In this work we propose a new architecture for dual-sequence modeling that is based on associative memory. We derive AM-RNNs, a recurrent associative memory (AM) which augments generic recurrent neural networks (RNN). This architecture is extended to the Dual AM-RNN which operates on two AMs at once. Our models achieve very competitive results on textual entailment. A qualitative analysis…
Simons Center for Systems Biology（西蒙斯系统生物学中心）；Institute for Advanced Study（高等研究院）；Princeton Neuroscience Institute（普林斯顿神经科学研究所）；Princeton University（普林斯顿大学）
A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of models that smoothly interpolates between two limiting cases can be constructed. One limit is referred to as the feature-matching mode of pattern recognition, and the other one as the prototype regime. On the deep learning side of the duality, this family corresponds to feedforward…
Unsupervised learning in a generalized Hopfield associative-memory network is investigated in this work. First, we prove that the (generalized) Hopfield model is equivalent to a semi-restricted Boltzmann machine with a layer of visible neurons and another layer of hidden binary neurons, so it could serve as the building block for a multilayered deep-learning system. We then demonstrate that the Hopfield network can learn to form a faithful internal representation of the observed samples, with the learned memory patterns being prototypes of the input data. Furthermore, we propose a spectral met…
MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；IBM Research（IBM研究院）
Dense Associative Memories or Modern Hopfield Networks have many appealing properties of associative memory. They can do pattern completion, store a large number of memories, and can be described using a recurrent neural network with a degree of biological plausibility and rich feedback between the neurons. At the same time, up until now all the models of this class have had only one hidden layer, and have only been formulated with densely connected network architectures, two aspects that hinder their machine learning applications. This paper tackles this gap and describes a fully recurrent mo…
ISAM-MTL: Cross-subject multi-task learning model with identifiable spikes and associative memory networks
School of Future Technology, South China University of Technology（华南理工大学未来技术学院）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）
Long-Term Visual Object Tracking with Event Cameras: An Associative Memory Augmented Tracker and A Benchmark Dataset
School of Computer Science and Technology, Anhui University（计算机科学与技术学院，安徽大学）；School of Electronic and Information Engineering, Anhui University（电子与信息工程学院，安徽大学）
Semantically-correlated memories in a dense associative model
Institute for Computational and Experimental Research in Mathematics, Brown University（布朗大学计算与实验数学研究所）；SciAI Center, Cornell University（康奈尔大学SciAI中心）；Neural Coding and Brain Computing Unit, OIST Graduate University（冲绳科学技术大学院大学神经编码与脑计算单元）
Fast contextual adaptation has shown to be effective in improving Automatic Speech Recognition (ASR) of rare words and when combined with an on-device personalized training, it can yield an even better recognition result. However, the traditional re-scoring approaches based on an external language model is prone to diverge during the personalized training. In this work, we introduce a model-based end-to-end contextual adaptation approach that is decoder-agnostic and amenable to on-device personalization. Our on-device simulation experiments demonstrate that the proposed approach outperforms th…
Redundancy Maximization as a Principle of Associative Memory Learning in Hopfield Networks
Complex Systems Theory, Max Planck Institute for Dynamics and Self-Organization（复杂系统理论，马克斯·普朗克动态与自组织研究所）；Faculty of Physics, Institute for the Dynamics of Complex Systems, University of Göttingen（物理系，复杂系统动力学研究所，哥廷根大学）；Göttingen Campus Institute for Dynamics of Biological Networks, University of Göttingen（哥廷根校区生物网络动力学研究所，哥廷根大学）；Champalimaud Centre for the Unknown, Lisbon, Portugal（查帕利马德未知中心，里斯本，葡萄牙）；Max Planck Institute for Dynamics and Self-Organization（马克斯·普朗克动态与自组织研究所）
