arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.14754cs.AI
收藏

GateSight:循环记忆预测混合大语言模型中与查询无关的键值缓存淘汰的注意力记忆

GateSight: Recurrent Memory Anticipates Attention Memory for Query-Agnostic KV Eviction in Hybrid LLMs

Shanghai Jiao Tong University(上海交通大学)

Junjie Li, Jiong Lou, Jie Li

AI总结:

GateSight利用GDN层预填充阶段的信号预测KV条目效用,以轻量每头MLP实现与查询无关的KV缓存淘汰,在仅需十分之一注意力KV缓存时,在多基准上优于现有方法。

摘要:

混合语言模型将具有固定大小循环记忆的门控DeltaNet(GDN)层与键值(KV)缓存随上下文增长且主导服务内存的全注意力层交错排列。本文探究循环记忆是否已预先知晓注意力记忆所需内容。首先,我们证明Qwen3.8-27B的哪些GDN头保留长记忆由权重而非文档决定。接着,我们定义KV条目的未来效用为移除该条目对模型自身后续生成的注意力输出的影响程度,发现GDN层在预填充阶段输出的写门、衰减和写幅度在任何查询出现前即可预测该效用。由于该关系分布在多个GDN头和层中,且线性映射可恢复其趋势但对高效用尾部的捕捉弱于非线性映射,我们提出GateSight,其通过轻量的每头多层感知器学习该关系,这些感知器读取上述信号且不使用任何注意力分数。其预测的效用决定每个注意力层保留哪些条目,而各层保留的数量由模型原生注意力输出门设定,该门的均值在不同层间相差一个数量级以上。最后,在服务阶段,GateSight在预填充后对缓存评分一次并物理释放被丢弃的KV条目。在SCBench和RULER-4096上使用Qwen3.8-27B,以及采用相同方案的稀疏注意力模型Qwen3.8-Flash-Next(达到模型的262K令牌限制)的实验结果显示,仅使用十分之一的注意力KV缓存时,GateSight在四组模型-基准对中的三组上是五种与查询无关方法中最准确的,且在所有四组上均领先KVzip,优势达9.5至47.2个百分点。值得注意的是,其对全长上下文评分的开销相对于预填充可忽略不计。

英文摘要:

Hybrid language models interleave Gated DeltaNet (GDN) layers, whose recurrent memory has a fixed size, with full-attention layers, whose key-value (KV) cache grows with the context and dominates serving memory. In this paper, we ask whether the recurrent memory already knows what the attention memory will need. We first show that which GDN heads of Qwen3.8-27B keep long memories is fixed by the weights rather than by the document. Next, we define the future utility of a KV entry by how much removing it changes the attention output over the model's own continuations, and find that the write gate, decay and write magnitude that GDN layers emit during prefill anticipate this utility before any query is seen. Because the relation is spread across many GDN heads and layers, and a linear map recovers its trend but less of its high-utility tail than a nonlinear one, we propose GateSight, which learns it with lightweight per-head multilayer perceptrons that read these signals and no attention score. Their predicted utility decides which entries each attention layer keeps, and how much each layer keeps is set by the model's native attention output gate, whose mean differs by more than an order of magnitude across layers. Finally, at serving time, GateSight scores the cache once after prefill and physically releases discarded KV entries. Experimental results on SCBench and RULER-4096 with Qwen3.8-27B and, with the same recipe, the sparse-attention model Qwen3.8-Flash-Next, up to the 262K-token limit of the models, show that with a tenth of the attention KV cache, GateSight is the most accurate of five query-agnostic methods on three of the four model-benchmark pairs and leads KVzip on all four, by 9.5 to 47.2 points. Notably, it scores a full-length context with negligible overhead relative to prefill.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑