arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.12204cs.LG
收藏

学习遗忘注意力:适应性计算减少的记忆巩固

Learning to Forget Attention: Memory Consolidation for Adaptive Compute Reduction

Iowa State University(爱荷华州立大学)

Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma

AI总结:

本文提出了一种基于记忆巩固的注意力路由机制,通过减少冗余注意力计算,显著提升模型效率,同时在未见过的任务中保持高准确性。

摘要:

混合架构结合状态空间模型与注意力机制已实现了良好的效率-质量权衡,但现有方法要么统一应用注意力,要么学习静态稀疏模式。这错过了一个关键机会:随着重复模式变得熟悉,注意力需求应随时间减少。我们从分析GPT-2模型中得出一个令人惊讶的发现:88%的注意力操作从模型的隐藏状态中检索信息,且这种冗余在训练过程中不减少。受此观察启发,我们引入了\ours{}(基于巩固的路由用于适应性记忆),一种生物启发式记忆巩固机制,逐步将事件检索转化为参数化语义记忆。与以往稀疏注意力方法不同,\ours{}在训练过程中表现出注意力利用率的下降,通过约3000步的尖锐相变,实现了37.8倍的减少。我们证明这种能力在没有巩固的情况下是不可能的:任何静态路由方案对于具有频率为f的重复模式的任务需要Ω(f·n)的注意力。在我们提出的新SRCD基准上,\ours{}在1.6%的注意力计算下实现了100%的检索准确率(相比基线为68%),并且巩固的模式在未见过的任务中无需重新训练即可实现48-52%的注意力减少。值得注意的是,学习的巩固动态定量匹配认知心理学中的事件到语义记忆转换曲线(γ=0.43 vs. γ_human≈0.4-0.5)。代码和基准在[匿名化]处可用。

英文摘要:

Hybrid architectures combining state-space models with attention have achieved strong efficiency-quality tradeoffs, yet existing approaches either apply attention uniformly or learn static sparse patterns. This misses a key opportunity: \emph{attention demand should decrease over time as recurring patterns become familiar}. We present a surprising finding from analyzing GPT-2 models: \textbf{88\%} of attention operations retrieve information already predictable from the model's hidden state, and this redundancy does \emph{not} decrease during training. Motivated by this observation, we introduce \textbf{\ours{}} (\textbf{C}onsolidation-based \textbf{R}outing for \textbf{A}daptive \textbf{M}emory), a biologically inspired memory consolidation mechanism that gradually distills episodic retrievals into parametric semantic memory. Unlike prior sparse attention methods, \ours{} exhibits \emph{decreasing attention utilization} over training, achieving a \textbf{37.8$\times$} reduction through a sharp phase transition at approximately 3K steps. We prove that this capability is \emph{impossible} without consolidation: any static routing scheme requires $Ω(f \cdot n)$ attention for tasks with recurring patterns of frequency $f$. On our proposed SRCD benchmark, \ours{} achieves \textbf{100\% retrieval accuracy} at 1.6\% attention compute (vs.\ 68\% for baselines), and consolidated patterns transfer to unseen tasks with \textbf{48--52\%} attention reduction without retraining. Remarkably, the learned consolidation dynamics quantitatively match human episodic-to-semantic memory transition curves from cognitive psychology ($γ= 0.43$ vs.\ $γ_{\text{human}} \approx 0.4$--$0.5$). Code and benchmarks are available at [anonymized].

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑