Several mechanisms to focus attention of a neural network on selected parts of its input or memory have been used successfully in deep learning models in recent years. Attention has improved image classification, image captioning, speech recognition, generative models, and learning algorithmic tasks, but it had probably the largest impact on neural machine translation.
Recently, similar improvements have been obtained using alternative mechanisms that do not focus on a single part of a memory but operate on all of it in parallel, in a uniform way. Such mechanism, which we call active memory, improved over attention in algorithmic tasks, image processing, and in generative modelling.
So far, however, active memory has not improved over attention for most natural language processing tasks, in particular for machine translation. We analyze this shortcoming in this paper and propose an extended model of active memory that matches existing attention models on neural machine translation and generalizes better to longer sentences. We investigate this model and explain why previous active memory models did not succeed. Finally, we discuss when active memory brings most benefits and where attention can be a better choice.
Non-equilibrium active noise enhances generative memory in diffusion models
Department of Chemistry, University of Chicago, Chicago, IL, 60637（化学系，芝加哥大学，芝加哥，IL，60637）；The James Franck Institute, University of Chicago, Chicago, IL, 60637（弗朗克研究所，芝加哥大学，芝加哥，IL，60637）；Department of Physics, University of Chicago, Chicago, IL, 60637（物理系，芝加哥大学，芝加哥，IL，60637）
Class-wise Federated Unlearning: Harnessing Active Forgetting with Teacher-Student Memory Generation
School of Communication Engineering, Hangzhou Dianzi University（杭州电子科技大学通信工程学院）；College of Computer Science & Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Cyberspace, Hangzhou Dianzi University（杭州电子科技大学网络空间学院）
Multidimensional Bayesian Active Machine Learning of Working Memory Task Performance
Department of Biomedical Engineering, Washington University（生物医学工程系，华盛顿大学）；Department of Computer Science and Engineering, Washington University（计算机科学与工程系，华盛顿大学）
