联想记忆 Hopfield 模型的近期推广能够存储数量为 P 的随机模式,该数量随神经元数量 N 呈指数增长,即 P=exp(αN)。除了巨大的存储容量外,这些网络的另一个有趣特征是其与注意力机制的联系,而该机制是深度学习中广泛应用的 Transformer 架构的组成部分。在本研究中,我们使用统计力学分析研究了一个通用的模式集合族,给出了典型模式检索的精确渐近阈值 α_1,以及所有模式均可被检索的负载 α 最大值的下界 α_c 和吸引域大小。我们详细讨论了 Gaussian 和 spherical 模式的情况,并表明它们展现出丰富且定性不同的相图。
英文摘要:
Recent generalizations of the Hopfield model of associative memories are able to store a number $P$ of random patterns that grows exponentially with the number $N$ of neurons, $P=\exp(αN)$. Besides the huge storage capacity, another interesting feature of these networks is their connection to the attention mechanism which is part of the Transformer architectures widely applied in deep learning. In this work, we study a generic family of pattern ensembles using a statistical mechanics analysis which gives exact asymptotic thresholds for the retrieval of a typical pattern, $α_1$, and lower bounds for the maximum of the load $α$ for which all patterns can be retrieved, $α_c$, as well as sizes of attraction basins. We discuss in detail the cases of Gaussian and spherical patterns, and show that they display rich and qualitatively different phase diagrams.
Dense auto-hetero associative memories applied to noisy communication channels
Sapienza Università di Roma(罗马第一大学);Istituto Nazionale d’Alta Matematica, Sezione di Roma(罗马高等数学研究所);INFN, Istituto Nazionale di Fisica Nucleare, Sezione di Roma(罗马核物理国家研究所);CNR, Nanotec, Salento Unit(莱切纳米技术国家研究委员会);Università del Salento(萨伦托大学);INFN, Istituto Nazionale di Fisica Nucleare, Sezione di Lecce(莱切核物理国家研究所)
Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplings
Physics Department, Sapienza University of Rome(罗马大学物理系);Institute of Nanotechnology, National Research Council of Italy, CNR-NANOTEC, Rome Unit(意大利国家研究委员会纳米技术研究所,CNR-NANOTEC,罗马分部)
Phases in a class of associative memories via hidden neurons
CyberAgent AI Lab(CyberAgent AI实验室);RIKEN iTHEMS(RIKEN跨学科理论科学研究所);Graduate School of Artificial Intelligence and Science, Rikkyo University(立教大学人工智能与科学研究科)
Thermodynamics of bidirectional associative memories
Università del Salento(萨伦托大学);Istituto Nazionale di Fisica Nucleare(国家核物理研究所);Universidad Complutense de Madrid(马德里康普顿斯大学);Université Paris-Saclay(巴黎萨克雷大学);CNRS(法国国家科学研究中心);INRIA(法国国家信息与自动化研究所)
Do Hopfield Networks Dream of Stored Patterns? A Statistical-Mechanical Theory of Dreaming in Multidirectional Associative Memories
Sapienza Università di Roma(罗马大学);Istituto Nazionale di Fisica Nucleare(意大利国家核物理研究所);Istituto Nazionale d’Alta Matematica, GNFM(意大利高等数学研究所);Università del Salento(莱切萨伦托大学)
Large Associative Memory Problem in Neurobiology and Machine Learning
MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室);IBM Research(IBM研究院);Princeton Neuroscience Institute(普林斯顿神经科学研究所);Princeton University(普林斯顿大学)
密集关联记忆或现代Hopfield网络允许存储和可靠检索指数级大(在特征空间的维度上)数量的记忆。与此同时,它们的朴素实现是非生物的,因为它似乎需要神经元之间存在多体突触连接。我们表明,这些模型是一个更微观的(书面的生物自由度)理论,有额外的(隐藏的)神经元,只需要两个身体之间的相互作用的有效描述。由于这个原因,我们提出的微观理论是一个有效的模型,大联想记忆与一定程度的生物相容性。我们的网络的动力学和它的降维等效都最小化能量(李雅普诺夫)函数。当某些动态变量(隐藏神经元)从我们的微观理论中整合出来时,可以恢复许多以前在文献中讨论过的模型,例如“Hopfield Networks is All You Need”论文中提出的模型。我们还提供了另一种推导的能量函数和更新规则中提出的上述文件,并澄清这一类的各种模型之间的关系。
A Hardware-aware Hopfield Network with a Nonlinear Memristor Array for Robust Associative Memory with Superlinear Capacity
Korea Institute of Science and Technology (KIST)(韩国科学技术院);Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院);Seoul National University(首尔大学);Korea University(韩国大学);KIST School, University of Science and Technology(韩国科学技术院科学技术研究生院)
Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(哈佛学院肯普纳自然与人工智能研究所);Zuckerman Institute, Columbia University(哥伦比亚大学祖克曼研究所);Kavli Institute for Brain Science, Columbia University(哥伦比亚大学卡弗里脑科学研究所)
Memorization to Generalization: Emergence of Diffusion Models from Associative Memory
Department of Computer Science, Rensselaer Polytechnic Institute ( RPI )(计算机科学系,罗切斯特理工学院(RPI));Jheronimus Academy of Data Science, Tilburg University(数据科学学院,蒂尔堡大学);Department of Physics, University of Rome Sapienza(物理系,罗马萨皮恩扎大学);Donders Institute for Brain, Cognition, and Behaviour, Radboud University(大脑、认知与行为研究所,拉德堡德大学);IBM Research(IBM研究院)
Hiroshima City University(广岛市立大学);RIKEN AIP(理化学研究所先进智能研究中心);Kyushu University(九州大学);Institute for Physics of Intelligence(智能物理学研究所);The University of Tokyo(东京大学);Saddle Point Science Europe and DCN Donders Institute, Radboud University(拉布德大学唐德斯研究所与鞍点科学欧洲分部)
Storing Infinite Dynamical Attractors in Nonreciprocal Associative Neural Networks
BCAM – Basque Center for Applied Mathematics(巴斯克应用数学中心);IKERBASQUE, Basque Foundation for Science(伊克尔巴斯基克科学基金会);Biofisika Institute (CSIC, EHU)(生物物理研究所(西班牙国家研究委员会、巴斯克大学))
Simons Center for Systems Biology(西蒙斯系统生物学中心);Institute for Advanced Study(高等研究院);Princeton Neuroscience Institute(普林斯顿神经科学研究所);Princeton University(普林斯顿大学)