发表机构
Kim Jaechul Graduate School of AI, KAIST; KT Corporation（韩国科学技术院金宰澈人工智能研究生院; KT公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对长上下文推理中KV缓存压缩问题，提出行为保持的压缩框架，通过KL散度评估驱逐影响，在匹配预算下显著提升下游任务质量，并保持端到端加速。
AI 中文摘要
KV缓存是大语言模型在长上下文推理和长文本生成中的主要瓶颈。现有的免训练驱逐策略主要依赖代理重要性信号（如注意力权重）来决定保留哪些过去的令牌。我们认为，缓存压缩应转而保持全缓存模型的可预测行为，即保留那些移除后会显著改变模型输出分布的条目。我们提出了行为保持的KV缓存压缩，这是一个免训练框架，通过估计移除候选条目后压缩缓存的逻辑值，并评估其与全缓存下一个令牌分布的KL散度来对候选驱逐进行评分。利用驱逐前的正向统计，该方法避免了为每个候选执行单独的掩码正向传递。在多种架构以及预填充时间和生成时间压缩中，我们的方法在匹配的保留KV预算下，相比基于注意力的轻量级启发式方法，在下游任务质量上取得了显著提升，且在激进压缩下提升最大。它通过额外的压缩时间计算实现了这些增益，同时在我们评估的设置中保持了相对于全缓存推理的端到端加速。
英文摘要
KV caches are a major bottleneck in long-context inference and long-form generation with large language models. Existing training-free eviction policies largely rely on proxy importance signals, such as attention mass, to decide which past tokens to retain. We argue that cache compression should instead preserve the predictive behavior of the full-cache model, retaining entries whose removal would substantially change the model's output distribution. We propose Behavior-Preserving KV Cache Compression, a training-free framework that scores candidate evictions by estimating the compressed-cache logits induced by their removal and evaluating the resulting KL to the full-cache next-token distribution. Using pre-eviction forward statistics, the method avoids running separate masked forward passes for each candidate. Across diverse architectures and both prefill-time and generation-time compression, our method delivers substantial gains in downstream task quality over lightweight attention-based heuristics at matched retained-KV budgets, with the largest gains under aggressive compression. It achieves these gains with additional compression-time computation while retaining an end-to-end speedup over full-cache inference in our evaluated settings.
Journal refEMNLP 2026 Main Conference Published
