Our work combines aspects of three promising paradigms in machine learning, namely, attention mechanism, energy-based models, and associative memory. Attention is the power-house driving modern deep learning successes, but it lacks clear theoretical foundations. Energy-based models allow a principled approach to discriminative and generative tasks, but the design of the energy functional is not straightforward. At the same time, Dense Associative Memory models or Modern Hopfield Networks have a well-established theoretical foundation, and allow an intuitive design of the energy function. We propose a novel architecture, called the Energy Transformer (or ET for short), that uses a sequence of attention layers that are purposely designed to minimize a specifically engineered energy function, which is responsible for representing the relationships between the tokens. In this work, we introduce the theoretical foundations of ET, explore its empirical capabilities using the image completion task, and obtain strong quantitative results on the graph anomaly detection and graph classification tasks.
Memorization to Generalization: Emergence of Diffusion Models from Associative Memory
Department of Computer Science, Rensselaer Polytechnic Institute ( RPI )（计算机科学系，罗切斯特理工学院（RPI））；Jheronimus Academy of Data Science, Tilburg University（数据科学学院，蒂尔堡大学）；Department of Physics, University of Rome Sapienza（物理系，罗马萨皮恩扎大学）；Donders Institute for Brain, Cognition, and Behaviour, Radboud University（大脑、认知与行为研究所，拉德堡德大学）；IBM Research（IBM研究院）
Associative Memory and Generative Diffusion in the Zero-noise Limit
Computational & Systems Biology, Sloan Kettering Institute, Memorial Sloan Kettering Cancer Center（计算生物学与系统生物学，斯隆凯特灵研究所，纪念斯隆凯特灵癌症中心）；Tri-Institutional Training Program in Computational Biology & Medicine（计算生物学与医学三机构联合培训计划）
Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data
Department of Computer Science, Rensselaer Polytechnic Institute ( RPI )（里士满理工学院计算机科学系）；Donders Institute for Brain, Cognition, and Behaviour, Radboud University（拉布德大学脑科学与行为研究所）；Laboratoire de Physique Théorique et Modélisation, CY Cergy Paris Université（巴黎中央理工学院物理理论与建模实验室）
Tensor Cache: Eviction-conditioned Associative Memory for Transformers
Massachusetts Institute of Technology, Cambridge, MA, USA（麻省理工学院）；IBM Research, Cambridge, MA, USA（IBM研究院）；University of Toronto, Toronto, Canada（多伦多大学）
Distributed Associative Memory via Online Convex Optimization
KCLIP, CIIPS, Department of Engineering, King’s College London, London WC2R 2LS, UK（KCLIP、CIIPS、工程系、伦敦国王学院，伦敦 WC2R 2LS，英国）；Communication Systems Department, EURECOM, 06904 Sophia Antipolis, France（通信系统部门、EURECOM、法国索菲亚抗皮斯，06904）；Intelligent Networked Systems Institute, Northeastern University London, London E1 8PH, UK（智能网络系统研究所、伦敦东北大学、伦敦 E1 8PH，英国）
Examining Reject Relations in Stimulus Equivalence Simulations
Dep de Ps Clín, Psbiol y Metod Universidad de la Laguna（心理临床、心理学与方法学大学拉古纳大学）；Department of Informatics University of Bergen（信息学院卑尔根大学）；Department of Computer Science Oslo Metropolitan University（计算机科学学院奥斯陆 Metropolitan 大学）
Large Associative Memory Problem in Neurobiology and Machine Learning
MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；IBM Research（IBM研究院）；Princeton Neuroscience Institute（普林斯顿神经科学研究所）；Princeton University（普林斯顿大学）
密集关联记忆或现代Hopfield网络允许存储和可靠检索指数级大（在特征空间的维度上）数量的记忆。与此同时，它们的朴素实现是非生物的，因为它似乎需要神经元之间存在多体突触连接。我们表明，这些模型是一个更微观的（书面的生物自由度）理论，有额外的（隐藏的）神经元，只需要两个身体之间的相互作用的有效描述。由于这个原因，我们提出的微观理论是一个有效的模型，大联想记忆与一定程度的生物相容性。我们的网络的动力学和它的降维等效都最小化能量（李雅普诺夫）函数。当某些动态变量（隐藏神经元）从我们的微观理论中整合出来时，可以恢复许多以前在文献中讨论过的模型，例如“Hopfield Networks is All You Need”论文中提出的模型。我们还提供了另一种推导的能量函数和更新规则中提出的上述文件，并澄清这一类的各种模型之间的关系。
Simons Center for Systems Biology（西蒙斯系统生物学中心）；Institute for Advanced Study（高等研究院）；Princeton Neuroscience Institute（普林斯顿神经科学研究所）；Princeton University（普林斯顿大学）
Distributed Dynamic Associative Memory via Online Convex Optimization
Department of Engineering, King’s College London（伦敦国王学院工程系）；Communication Systems Department, EURECOM（EURECOM通信系统部）；Institute for Intelligent Networked Systems (INSI) at Northeastern University London（伦敦东北大学智能网络系统研究所）
Dense Associative Memory is Robust to Adversarial Inputs
Simons Center for Systems Biology, Institute for Advanced Study（高等研究院西蒙斯系统生物学中心）；Princeton Neuroscience Institute, Princeton University（普林斯顿大学普林斯顿神经科学研究所）
