Pretrained Transformers keep what they learned in their weights and lose what they observe once a session ends. The first version of this paper proposed a Retention Layer, a persistent memory that a Transformer block reads with attention and writes during use. Because most of what a deployed model could retain is produced by other agents, this revision treats deciding what to keep as a social learning problem: when to rely on observed behaviour, whom to learn from and how much independent agreement to require. We give a corrected specification of the layer, which reduces exactly to the base Transformer when its memory is empty. We derive the memory's lifecycle from social learning strategies: encoding gated by surprise, observed outcomes and earned credibility; consolidation by a credibility weighted quorum of distinct, recent sources that must also outweigh every rival behaviour; and reconsolidation by the outcomes of reproduction. We prove that raising the quorum lowers the risk of consolidating a coordinated false template exponentially while delaying true templates only linearly, and that relative consolidation protects only while credible honest evidence arrives faster than adversarial evidence. In a simulation with world drift and three memory-poisoning attacks, the lifecycle reached accuracies of 0.989 to 0.996, against 0.62 to 0.63 for the ungated first version design, and kept attack success at or below 0.07 when 30% of the observations about a target were adversarial. As predicted, it amplified attacks once adversarial evidence outpaced honest evidence. Experience with a long running assistant adds two rules: a model's own outputs must not count as support, and a user's testimony should be kept after one mention. We close with an evaluation protocol for language models.
Transactional Attention: Semantic Sponsorship for KV-Cache Retention
National Institute of Electronics and Information Technology (NIELIT)（国家电子与信息技术研究院）；Indian Institute of Information Technology, Allahabad (IIITA)（阿拉哈巴德印度信息技术学院）
SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language Models
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Chongqing University（重庆大学）；Peng Cheng Laboratory（鹏城实验室）；Wuhan AI Research（武汉人工智能研究院）
Intelligence necessitates memory. Without memory, humans fail to perform various nontrivial tasks such as reading novels, playing games or solving maths. As the ultimate goal of machine learning is to derive intelligent systems that learn and act automatically just like human, memory construction for machine is inevitable. Artificial neural networks model neurons and synapses in the brain by interconnecting computational units via weights, which is a typical class of machine learning algorithms that resembles memory structure. Their descendants with more complicated modeling techniques (a.k.a …
Risk Analysis in Customer Relationship Management via Quantile Region Convolutional Neural Network-Long Short-Term Memory and Cross-Attention Mechanism
Longyan University（龙岩学院）；National Central University（中央大学）；University of Portsmouth（朴茨茅斯大学）；Institute of Semiconductors, Chinese Academy of Sciences（中国科学院半导体研究所）；Northeastern University（东北大学）
Efficient Low Rank Attention for Long-Context Inference in Large Language Models
Guangdong University of Technology（广东工业大学）；RIKEN AIP（理化学研究所AIP）；Key Laboratory of Intelligent Detection and the Internet of Things in Manufacturing, Ministry of Education, Guangzhou, CHINA（教育部智能制造智能化检测与物联网重点实验室）；RIKEN iTHEMS（理化学研究所iTHEMS）；RIKEN IMS（理化学研究所IMS）；Chiba University（千叶大学）
Sliding Window Attention Training for Efficient Large Language Models
City University of Hong Kong（香港城市大学）；Xi’an Jiaotong University（西安交通大学）；Jarvis Research Center, Tencent YouTu Lab（腾讯优图实验室研究部）；Westlake University（西湖大学）；University of Science and Technology of China（中国科学技术大学）；Southern University of Science and Technology（南方科技大学）
Vision-Transformers are widely used in various vision tasks. Meanwhile, there is another line of works starting with the MLP-mixer trying to achieve similar performance using mlp-based architectures. Interestingly, until now those mlp-based architectures have not been adapted for NLP tasks. Additionally, until now, mlp-based architectures have failed to achieve state-of-the-art performance in vision tasks. In this paper, we analyze the expressive power of mlp-based architectures in modeling dependencies between multiple different inputs simultaneously, and show an exponential gap between the a…
Recurrent Neural Networks were, until recently, one of the best ways to capture the timely dependencies in sequences. However, with the introduction of the Transformer, it has been proven that an architecture with only attention-mechanisms without any RNN can improve on the results in various sequence processing tasks (e.g. NLP). Multiple studies since then have shown that similar approaches can be applied for images, point clouds, video, audio or time series forecasting. Furthermore, solutions such as the Perceiver or the Informer have been introduced to expand on the applicability of the Tra…
