Understanding the memory capacity of neural networks remains a challenging problem in implementing artificial intelligence systems. In this paper, we address the notion of capacity with respect to Hopfield networks and propose a dynamic approach to monitoring a network's capacity. We define our understanding of capacity as the maximum number of stored patterns which can be retrieved when probed by the stored patterns. Prior work in this area has presented static expressions dependent on neuron count $N$, forcing network designers to assume worst-case input characteristics for bias and correlation when setting the capacity of the network. Instead, our model operates simultaneously with the learning Hopfield network and concludes on a capacity estimate based on the patterns which were stored. By continuously updating the crosstalk associated with the stored patterns, our model guards the network from overwriting its memory traces and exceeding its capacity. We simulate our model using artificially generated random patterns, which can be set to a desired bias and correlation, and observe capacity estimates between 93% and 97% accurate. As a result, our model doubles the memory efficiency of Hopfield networks in comparison to the static and worst-case capacity estimate while minimizing the risk of lost patterns.
Redundancy Maximization as a Principle of Associative Memory Learning in Hopfield Networks
Complex Systems Theory, Max Planck Institute for Dynamics and Self-Organization(复杂系统理论,马克斯·普朗克动态与自组织研究所);Faculty of Physics, Institute for the Dynamics of Complex Systems, University of Göttingen(物理系,复杂系统动力学研究所,哥廷根大学);Göttingen Campus Institute for Dynamics of Biological Networks, University of Göttingen(哥廷根校区生物网络动力学研究所,哥廷根大学);Champalimaud Centre for the Unknown, Lisbon, Portugal(查帕利马德未知中心,里斯本,葡萄牙);Max Planck Institute for Dynamics and Self-Organization(马克斯·普朗克动态与自组织研究所)
John A. Paulson School of Engineering and Applied Sciences(约翰·A·保尔森工程与应用科学学院);Center for Brain Science(脑科学中心);Department of Physics(物理系);MIT-IBM Watson AI Lab, IBM Research(MIT-IBM沃森人工智能实验室,IBM研究院);Kempner Institute for the Study of Natural and Artificial Intelligence(肯普纳自然与人工智能研究所)
Department of Computer Science, Northwestern University, Evanston, IL, USA(计算机科学系,西北大学,伊利诺伊州埃文斯顿);Department of Physics and Computer Science, National Taiwan University, Taipei, Taiwan(物理与计算机科学系,国立台湾大学,台湾台北);Department of Statistics and Data Science, Northwestern University, Evanston, IL, USA(统计与数据科学系,西北大学,伊利诺伊州埃文斯顿)
McGovern Institute for Brain Research, Massachusetts Institute of Technology(麻省理工学院麦戈文脑科学研究所);The NSF AI Institute for Artificial Intelligence and Fundamental Interactions (IAIFI), Massachusetts Institute of Technology(麻省理工学院美国国家科学基金会人工智能基础相互作用研究所)
Institute of Zoology, Chinese Academy of Sciences(中国科学院动物研究所);Institute for Stem Cell and Regeneration, Chinese Academy of Sciences(中国科学院干细胞与再生医学研究院);Beijing Institute for Stem Cell and Regenerative Medicine(北京干细胞与再生医学研究院);University of Chinese Academy of Sciences(中国科学院大学);School of Electronic and Information Engineering, Hebei University of Technology(河北工业大学电子与信息工程学院)
Queens College of the City University of New York(纽约市立大学皇后学院);The Graduate Center, City University of New York(纽约市立大学研究生中心);City College of the City University of New York(纽约市立学院)