We consider the problem of neural association for a network of non-binary neurons. Here, the task is to first memorize a set of patterns using a network of neurons whose states assume values from a finite number of integer levels. Later, the same network should be able to recall previously memorized patterns from their noisy versions. Prior work in this area consider storing a finite number of purely random patterns, and have shown that the pattern retrieval capacities (maximum number of patterns that can be memorized) scale only linearly with the number of neurons in the network.
In our formulation of the problem, we concentrate on exploiting redundancy and internal structure of the patterns in order to improve the pattern retrieval capacity. Our first result shows that if the given patterns have a suitable linear-algebraic structure, i.e. comprise a sub-space of the set of all possible patterns, then the pattern retrieval capacity is in fact exponential in terms of the number of neurons. The second result extends the previous finding to cases where the patterns have weak minor components, i.e. the smallest eigenvalues of the correlation matrix tend toward zero. We will use these minor components (or the basis vectors of the pattern null space) to both increase the pattern retrieval capacity and error correction capabilities.
An iterative algorithm is proposed for the learning phase, and two simple neural update algorithms are presented for the recall phase. Using analytical results and simulations, we show that the proposed methods can tolerate a fair amount of errors in the input while being able to memorize an exponentially large number of patterns.
Phases in a class of associative memories via hidden neurons
CyberAgent AI Lab(CyberAgent AI实验室);RIKEN iTHEMS(RIKEN跨学科理论科学研究所);Graduate School of Artificial Intelligence and Science, Rikkyo University(立教大学人工智能与科学研究科)
Semantically-correlated memories in a dense associative model
Institute for Computational and Experimental Research in Mathematics, Brown University(布朗大学计算与实验数学研究所);SciAI Center, Cornell University(康奈尔大学SciAI中心);Neural Coding and Brain Computing Unit, OIST Graduate University(冲绳科学技术大学院大学神经编码与脑计算单元)