混合架构结合状态空间模型与注意力机制已实现了良好的效率-质量权衡,但现有方法要么统一应用注意力,要么学习静态稀疏模式。这错过了一个关键机会:随着重复模式变得熟悉,注意力需求应随时间减少。我们从分析GPT-2模型中得出一个令人惊讶的发现:88%的注意力操作从模型的隐藏状态中检索信息,且这种冗余在训练过程中不减少。受此观察启发,我们引入了\ours{}(基于巩固的路由用于适应性记忆),一种生物启发式记忆巩固机制,逐步将事件检索转化为参数化语义记忆。与以往稀疏注意力方法不同,\ours{}在训练过程中表现出注意力利用率的下降,通过约3000步的尖锐相变,实现了37.8倍的减少。我们证明这种能力在没有巩固的情况下是不可能的:任何静态路由方案对于具有频率为f的重复模式的任务需要Ω(f·n)的注意力。在我们提出的新SRCD基准上,\ours{}在1.6%的注意力计算下实现了100%的检索准确率(相比基线为68%),并且巩固的模式在未见过的任务中无需重新训练即可实现48-52%的注意力减少。值得注意的是,学习的巩固动态定量匹配认知心理学中的事件到语义记忆转换曲线(γ=0.43 vs. γ_human≈0.4-0.5)。代码和基准在[匿名化]处可用。
英文摘要:
Hybrid architectures combining state-space models with attention have achieved strong efficiency-quality tradeoffs, yet existing approaches either apply attention uniformly or learn static sparse patterns. This misses a key opportunity: \emph{attention demand should decrease over time as recurring patterns become familiar}. We present a surprising finding from analyzing GPT-2 models: \textbf{88\%} of attention operations retrieve information already predictable from the model's hidden state, and this redundancy does \emph{not} decrease during training. Motivated by this observation, we introduce \textbf{\ours{}} (\textbf{C}onsolidation-based \textbf{R}outing for \textbf{A}daptive \textbf{M}emory), a biologically inspired memory consolidation mechanism that gradually distills episodic retrievals into parametric semantic memory. Unlike prior sparse attention methods, \ours{} exhibits \emph{decreasing attention utilization} over training, achieving a \textbf{37.8$\times$} reduction through a sharp phase transition at approximately 3K steps. We prove that this capability is \emph{impossible} without consolidation: any static routing scheme requires $Ω(f \cdot n)$ attention for tasks with recurring patterns of frequency $f$. On our proposed SRCD benchmark, \ours{} achieves \textbf{100\% retrieval accuracy} at 1.6\% attention compute (vs.\ 68\% for baselines), and consolidated patterns transfer to unseen tasks with \textbf{48--52\%} attention reduction without retraining. Remarkably, the learned consolidation dynamics quantitatively match human episodic-to-semantic memory transition curves from cognitive psychology ($γ= 0.43$ vs.\ $γ_{\text{human}} \approx 0.4$--$0.5$). Code and benchmarks are available at [anonymized].
Continual Policy Consolidation for Lifelong Robot Learning
Department of XXX, University of YYY, Location, Country;School of ZZZ, Institute of WWW, Location, Country;Zhejiang University, Zhejiang, China;South University of Science;Hong Kong University, Hong Kong, China;INFIFORCE Intelligent Technology Co., Ltd., Hangzhou, China;The Hong Kong Polytechnic University, Hong Kong, China;School of Computer Science, Carnegie Mellon University
RecMem: Recurrence-based Memory Consolidation for Efficient and Effective Long-Running LLM Agents
Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程系);School of Computer Science, Beijing University of Posts and Telecommunications(北京邮电大学计算机学院);Huawei Cloud(华为云);Huawei Theory Lab(华为理论实验室);Institute for Math and AI, Wuhan University(武汉大学数学与人工智能研究院)
Task-Core Memory Management and Consolidation for Long-term Continual Learning
School of Computer Science and Technology, East China Normal University(东华师范大学计算机科学与技术学院);School of Electrical and Electronic Engineering, Nanyang Technological University(南洋理工大学电子与电气工程学院);Computation and Artificial Intelligence Innovative College, Fudan University(复旦大学计算与人工智能创新学院)
Semi-parametric Memory Consolidation: Towards Brain-like Deep Continual Learning
Centre for Artificial Intelligence and Robotics(人工智能与机器人中心);Hong Kong Institute of Science and Innovation(香港科学与创新研究院);Chinese Academy of Sciences(中国科学院);Department of Computer Science(计算机科学系);City University of Hong Kong(香港城市大学);The Department of Neurosurgery(神经外科部门);Peking University First Hospital(北京大学第一医院);State Key Laboratory of Multimodal Artificial Intelligence Systems(多模态人工智能系统国家重点实验室);Institute of Automation(自动化研究所);Hong Kong SAR, China(香港特别行政区,中国)