为长上下文LLM推理重构KV缓存淘汰问题
Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
AI总结：
本文提出一种基于输出的分层矩阵乘法近似方法，通过LaProx策略量化token贡献并考虑跨头依赖，实验表明在仅使用5%的KV缓存时保持性能并优于现有方法。
摘要：
大型语言模型（LLMs）支持长上下文推理，但因键值（KV）缓存增长导致显著的内存和运行时间开销。现有KV缓存淘汰方法主要依赖局部注意力权重，忽视了值表示、输出投影和跨头交互的影响。本文将KV缓存淘汰从传统的头-wise、权重平均方法重构为基于输出的分层矩阵乘法近似问题。我们引入LaProx，一种新的淘汰策略，显式建模注意力图与投影值状态之间的乘法交互，以准确量化token贡献并考虑跨头依赖。基于此度量，我们提出了首个统一的淘汰策略，为token分配全局可比的重要性分数，实现模型层面的选择而非局部、头级别的决策。在19个长上下文基准测试集（LongBench和Needle-In-A-Haystack）上进行实验，结果表明，我们的方法在仅使用5%的KV缓存时保持模型性能，并在所有配置中均优于现有方法。值得注意的是，在极端压缩场景下，我们的方法相比现有最先进的基线，实现了高达2倍的准确率损失减少，且具有极小的开销。
英文摘要：
Large language models (LLMs) support long-context inference but suffer from substantial memory and runtime overhead due to Key-Value (KV) Cache growth. Existing KV Cache eviction methods primarily rely on local attention weights, neglecting the influence of value representations, output projection, and inter-head interactions. In this work, we reformulate KV Cache eviction from a conventional head-wise, weight-averaging approach into an output-aware, layer-wise matrix multiplication approximation problem. We introduce LaProx, a novel eviction strategy that explicitly models the multiplicative interaction between attention maps and projected value states to accurately quantify token contributions while accounting for inter-head dependencies. Building on this metric, we propose the first unified eviction strategy that assigns globally comparable importance scores to tokens, enabling model-wide selection instead of local, head-wise decisions. Experimental results across 19 datasets on long-context benchmarks LongBench and Needle-In-A-Haystack demonstrate that our approach maintains model performance with only 5\% of the KV cache and consistently outperforms prior works across all configurations. Notably, our method achieves up to 2$\times$ accuracy loss reduction under extreme compression scenarios compared to existing state-of-the-art baselines with minimal overhead.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
