关键-值（KV）缓存中更重要的是什么：通过位置感知的伪查询实现解码对齐的KV缓存压缩
Where Matters More Than What: Decoding-aligned KV Cache Compression via Position-aware Pseudo Queries
AI总结：
本文提出DapQ，通过位置感知的伪查询实现解码对齐的KV缓存压缩，有效减少内存占用并提升生成性能。
摘要：
关键-值（KV）缓存对于高效的大语言模型（LLM）推理至关重要，但过长的上下文会显著增加KV缓存的内存占用。现有的KV缓存压缩方法通常依赖于提示观察窗口内的输入侧注意力模式，以在prefill阶段估计token的重要性。这些方法无法保留对后续生成至关重要的token，因为这些评估并非源自解码过程。直观上，一个有效的观察窗口应与解码阶段的查询相匹配，以准确反映生成过程将关注哪些token。然而，真实的解码查询在推理过程中本质上是不可用的。为了构建伪查询来近似它们，我们发现位置信息比语义内容更重要。受这一见解的启发，我们提出了通过位置感知的伪查询实现解码对齐的KV缓存压缩（DapQ），这是一种新颖且轻量的淘汰框架，利用位置感知的伪查询来模拟输出token，从而建立一个有效的观察窗口以进行重要性评估。它与实际生成上下文紧密对齐，并能够实现精确的token淘汰。在多个基准和LLM上的广泛评估表明，DapQ在多个基准和LLM上取得了优越的性能，特别是在严格的内存限制下（例如，在NIAH上使用3%的KV缓存预算时，性能达到近无损的99.5%）
英文摘要：
The Key-Value (KV) cache is crucial for efficient Large Language Models (LLMs) inference, but excessively long contexts drastically increase KV cache memory footprint. Existing KV cache compression methods typically rely on input-side attention patterns within a prompt observation window to estimate token importance during the prefill stage. They fail to preserve critical tokens for future generation since these assessments are not derived from the decoding process. Intuitively, an effective observation window should mirror the decoding-stage queries to accurately reflect which tokens the generation process will attend to. However, ground-truth decoding queries are inherently unavailable during inference. For constructing pseudo queries to approximate them, we find that positional information plays a more critical role than semantic content. Motivated by this insight, we propose decoding-aligned KV cache compression via position-aware pseudo queries (DapQ), a novel and lightweight eviction framework that leverages position-aware pseudo queries to simulate the output tokens, thereby establishing an effective observation window for importance assessment. It aligns closely with the actual generation context and enables precise token eviction. Extensive evaluations across multiple benchmarks and LLMs demonstrate that DapQ achieves superior performance, particularly under strict memory constraints (e.g., up to nearly lossless performance 99.5% on NIAH with 3% KV cache budgets).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
