EMS：基于全局-局部重要性的自适应逐出-合并策略用于头级KV缓存压缩
EMS: Adaptive Evict-then-Merge Strategy for Head-wise KV Cache Compression Based on Global-Local Importance
AI总结：
EMS提出基于全局-局部重要性的自适应逐出-合并策略，实现头级KV缓存压缩，在极端压缩比下保持低困惑度并提升长上下文任务性能。
摘要：
随着大型语言模型（LLMs）的持续进步，各应用对长上下文的高质量和快速处理需求日益增长。KV缓存被广泛采用，因为它存储先前生成的关键和值标记，有效减少推理过程中的冗余计算。然而，随着内存开销成为重要问题，KV缓存的高效压缩受到越来越多的关注。现有大多数方法从两个角度进行压缩：识别重要标记和设计压缩策略。然而，这些方法往往由于累积注意力分数或位置编码的影响而产生重要标记的偏态分布。此外，它们忽略了不同头之间的稀疏性和冗余性，导致在头级保留最有效信息方面存在困难。为此，我们提出EMS以克服这些局限，同时在极端压缩比下实现更好的KV缓存压缩。具体而言，我们引入一种全局-局部分数，结合来自全局和局部KV标记的累积注意力分数，以更好地识别标记重要性。对于压缩策略，我们设计了一个自适应且统一的逐出-合并框架，该框架考虑了不同头间KV标记的稀疏性和冗余性。此外，我们通过零类机制实现头级并行压缩以提高效率。大量实验表明，即使在极端压缩比下，我们的SOTA性能依然出色。EMS始终实现最低的困惑度，在256缓存预算下，在LongBench上四个LLM的得分提升超过1.28点，并在Needle-in-a-Haystack任务中，在缓存预算小于上下文长度2%的情况下保持95%的检索准确率。
英文摘要：
As large language models (LLMs) continue to advance, the demand for higher quality and faster processing of long contexts across various applications is growing. KV cache is widely adopted as it stores previously generated key and value tokens, effectively reducing redundant computations during inference. However, as memory overhead becomes a significant concern, efficient compression of KV cache has gained increasing attention. Most existing methods perform compression from two perspectives: identifying important tokens and designing compression strategies. However, these approaches often produce biased distributions of important tokens due to the influence of accumulated attention scores or positional encoding. Furthermore, they overlook the sparsity and redundancy across different heads, which leads to difficulties in preserving the most effective information at the head level. To this end, we propose EMS to overcome these limitations, while achieving better KV cache compression under extreme compression ratios. Specifically, we introduce a Global-Local score that combines accumulated attention scores from both global and local KV tokens to better identify the token importance. For the compression strategy, we design an adaptive and unified Evict-then-Merge framework that accounts for the sparsity and redundancy of KV tokens across different heads. Additionally, we implement the head-wise parallel compression through a zero-class mechanism to enhance efficiency. Extensive experiments demonstrate our SOTA performance even under extreme compression ratios. EMS consistently achieves the lowest perplexity, improves scores by over 1.28 points across four LLMs on LongBench under a 256 cache budget, and preserves 95% retrieval accuracy with a cache budget less than 2% of the context length in the Needle-in-a-Haystack task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
