AI 中文总结
KVFetch提出时间预取通道，通过量化冷层和单调指针检测顺序复制，在不增加注意力成本下恢复KV缓存压缩中的逐字复制，显著提升顺序访问任务性能。
AI 中文摘要
随着上下文窗口扩展到数万或数十万个令牌，KV缓存压缩已成为高效LLM推理的关键。现有方法分为三类：基于分数的驱逐、摘要补偿和卸载-召回。然而，这三类方法都根据内容与当前查询的相关性来决定保留或召回什么。我们表明，这种共享设计在结构上是不完整的。缓存支持两种访问模式：按内容的关联查找和按位置的顺序遍历；当前的压缩器只实现了第一种。这种差距在实践中很重要：检索增强生成、代码补全和结构化数据提取都要求模型从上下文中逐字复制标识符、字段值或代码令牌。在压缩下，基于内容的驱逐保留了此类序列的头部，但丢弃了其后续部分，导致逐字复制在中途不可逆地中断，我们将这种失败称为顺序遗忘。这种失败抵抗更好的评分、更大的预算、摘要补偿和动态重新评分；它是压缩下剩余质量损失的主要来源。我们提出KVFetch，一个无需训练、即插即用的框架，为任何基于分数的压缩器打开一个时间召回通道。它将驱逐的候选降级到量化的冷层，通过单调读取指针检测活动复制，并将位置后继预取到固定大小的热层槽中，而不增加注意力成本。在RULER-16K上，在等预算控制下，KVFetch将逐字复制从0.8恢复到78.4，并将13任务平均分提高+8.4，收益集中在需要顺序访问的任务上。在LongBench上，没有任务需要顺序访问，该通道保持休眠且不产生成本。
英文摘要
As context windows scale to tens or hundreds of thousands of tokens, KV cache compression has become essential for efficient LLM inference. Existing methods fall into three families: score-based eviction, summary compensation, and offload-and-recall. Yet all three decide what to keep or recall by content relevance to the current query. We show this shared design is structurally incomplete. A cache supports two access modes: associative lookup by content and sequential traversal by position; current compressors implement only the first. The gap matters in practice: retrieval-augmented generation, code completion, and structured-data extraction all require the model to reproduce identifiers, field values, or code tokens verbatim from the context. Under compression, content-based eviction retains the head of such a sequence but discards its continuation, causing verbatim copying to break irreversibly midway, a failure we call sequential forgetting. This failure resists better scoring, larger budgets, summary compensation, and dynamic re-scoring; it is the dominant source of remaining quality loss under compression. We propose KVFetch, a training-free, drop-in framework that opens a temporal recall channel for any score-based compressor. It demotes evicted candidates to a quantized cold tier, detects active copying through a monotone read pointer, and prefetches positional successors into fixed-size hot-tier slots without increasing attention cost. On RULER-16K under an iso-budget control, KVFetch recovers verbatim copying from 0.8 to 78.4 and raises the 13-task average by +8.4, with gains concentrating on tasks that require sequential access. On LongBench, where no task requires sequential access, the channel remains dormant and imposes no cost.
Comments21 pages, 7 figures. Submitted to ICLR 2027
