We investigate the computational limits of the memory retrieval dynamics of modern Hopfield models from the fine-grained complexity analysis. Our key contribution is the characterization of a phase transition behavior in the efficiency of all possible modern Hopfield models based on the norm of patterns. Specifically, we establish an upper bound criterion for the norm of input query patterns and memory patterns. Only below this criterion, sub-quadratic (efficient) variants of the modern Hopfield model exist, assuming the Strong Exponential Time Hypothesis (SETH). To showcase our theory, we provide a formal example of efficient constructions of modern Hopfield models using low-rank approximation when the efficient criterion holds. This includes a derivation of a lower bound on the computational time, scaling linearly with $\max\{$# of stored memory patterns, length of input query sequence$\}$. In addition, we prove its memory retrieval error bound and exponential memory capacity.
Department of Computer Science, Northwestern University, Evanston, IL, USA(计算机科学系,西北大学,伊利诺伊州埃文斯顿);Department of Physics and Computer Science, National Taiwan University, Taipei, Taiwan(物理与计算机科学系,国立台湾大学,台湾台北);Department of Statistics and Data Science, Northwestern University, Evanston, IL, USA(统计与数据科学系,西北大学,伊利诺伊州埃文斯顿)
Modern Hopfield Networks for Few- and Zero-Shot Reaction Template Prediction
Johannes Kepler University Linz(约翰内斯·开普勒大学林茨分校);Institute of Advanced Research in Artificial Intelligence (IARAI)(人工智能高级研究所);Janssen Research & Development, Janssen Pharmaceutica N.V.(杨森制药有限公司杨森研发部);Microsoft Research(微软研究院)
Modern Hopfield Networks with Continuous-Time Memories
Instituto de Telecomunicações(电信研究所);Instituto Superior Técnico(高等技术学院);Universidade de Lisboa(里斯本大学);Champalimaud Research(尚帕利莫研究中心);Unbabel(昂巴贝尔公司);ELLIS Unit Lisbon(欧洲学习与智能系统研究所里斯本分部)
Modern Hopfield Networks and Attention for Immune Repertoire Classification
ELLIS Unit Linz(ELLIS林茨单元);LIT AI Lab(LIT人工智能实验室);Institute for Machine Learning(机器学习研究所);Johannes Kepler University Linz(约翰内斯·开普勒大学林茨分校);Institute of Advanced Research in Artificial Intelligence (IARAI)(高级人工智能研究所(IARAI));University of Oslo(奥斯陆大学)
Modern Hopfield Networks Require Chain-of-Thought to Solve $\mathsf{NC}^1$-Hard Problems
Wyoming Seminary(怀俄明神学院);University of New South Wales(新南威尔士大学);The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Simons Institute for the Theory of Computing, University of California, Berkeley(伯克利加州大学计算理论研究所)
Johannes Kepler University Linz(林茨约翰开普勒大学);Institute of Advanced Research in Artificial Intelligence (IARAI)(高级人工智能研究所(IARAI));University of Oslo(奥斯陆大学)
Instituto Superior Técnico, Universidade de Lisboa(里斯本大学高等技术学院);Instituto de Telecomunicações(电信研究所);Language Technology Lab, University of Amsterdam(阿姆斯特丹大学语言技术实验室);Champalimaud Research(尚帕利莫德研究院);Unbabel(昂巴贝尔公司)
Hopfield-Fenchel-Young Networks: A Unified Framework for Associative Memory Retrieval
Instituto Superior Técnico, Universidade de Lisboa(里斯本大学技术学院);Instituto de Telecomunicações(电信研究所);Language Technology Lab, University of Amsterdam(阿姆斯特丹大学语言技术实验室);Champalimaud Research(恰帕拉姆德研究);ELLIS Unit Lisbon(里斯本ELLIS单位)
Input-Driven Dynamics for Robust Memory Retrieval in Hopfield Networks
Department of Information Engineering(信息工程系);Università degli Studi di Padova(帕多瓦大学);Center for Control, Dynamical Systems and Computation(控制、动态系统与计算中心);University of California at Santa Barbara(加州大学圣巴巴拉分校)
Phase Kernel Lifts Capacity of Dense Associative Memory
Institute of Fundamental and Frontier Sciences, University of Electronic Science and Technology of China(电子科技大学基础与前沿研究院);School of Physics, University of Electronic Science and Technology of China(电子科技大学物理学院);Non-classical Information Science Basic Discipline Research Center of Sichuan Province, University of Electronic Science and Technology of China(电子科技大学四川省非经典信息科学基础学科研究中心)
Large Associative Memory Problem in Neurobiology and Machine Learning
MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室);IBM Research(IBM研究院);Princeton Neuroscience Institute(普林斯顿神经科学研究所);Princeton University(普林斯顿大学)
密集关联记忆或现代Hopfield网络允许存储和可靠检索指数级大(在特征空间的维度上)数量的记忆。与此同时,它们的朴素实现是非生物的,因为它似乎需要神经元之间存在多体突触连接。我们表明,这些模型是一个更微观的(书面的生物自由度)理论,有额外的(隐藏的)神经元,只需要两个身体之间的相互作用的有效描述。由于这个原因,我们提出的微观理论是一个有效的模型,大联想记忆与一定程度的生物相容性。我们的网络的动力学和它的降维等效都最小化能量(李雅普诺夫)函数。当某些动态变量(隐藏神经元)从我们的微观理论中整合出来时,可以恢复许多以前在文献中讨论过的模型,例如“Hopfield Networks is All You Need”论文中提出的模型。我们还提供了另一种推导的能量函数和更新规则中提出的上述文件,并澄清这一类的各种模型之间的关系。
Demographic-aware fine-grained visual recognition of pediatric wrist pathologies
Department of Computer Science (IDI), Norwegian University of Science \& Technology (NTNU), Gjøvik, 2815, Norway Department of Informatics, Linnaeus University, V\"axj\"o, 351 95, Sweden Department of Computer Science, Sukkur IBA University, Sukkur, 65200, Pakistan
Enhancing Talent Search Ranking with Role-Aware Expert Mixtures and LLM-based Fine-Grained Job Descriptions
HKUST (GZ)(香港科技大学(广州));Alibaba Group(阿里巴巴集团);HKUST(香港科技大学);Zhijiang Lab(浙江实验室);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室(深圳))
Revisiting Evaluation Metrics for Semantic Segmentation: Optimization and Evaluation of Fine-grained Intersection over Union
KU Leuven(鲁汶大学);Google(谷歌公司);Google DeepMind(谷歌DeepMind);University of Oxford(牛津大学);EPFL(洛桑联邦理工学院);ETH Zürich(苏黎世联邦理工学院);INSAIT(INSAIT研究院);Samsung Research(三星研究院)
Associative Memory and Generative Diffusion in the Zero-noise Limit
Computational & Systems Biology, Sloan Kettering Institute, Memorial Sloan Kettering Cancer Center(计算生物学与系统生物学,斯隆凯特灵研究所,纪念斯隆凯特灵癌症中心);Tri-Institutional Training Program in Computational Biology & Medicine(计算生物学与医学三机构联合培训计划)