arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.02812cs.CLcs.AI
收藏

Q-Filters:利用 QK 几何特性进行高效的 KV Cache 压缩

Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression

Inria(法国国家信息与自动化研究所);Sapienza University of Rome(罗马大学);University of Edinburgh(爱丁堡大学)

Nathan Godey, Alessio Devoto, Yu Zhao, Simone Scardapane, Pasquale Minervini, Éric de la Clergerie, Benoît Sagot

AI总结:

本文提出了一种免训练的 KV Cache 压缩方法 Q-Filters,利用 Query 和 Key 向量特性无需计算注意力图即可近似注意力分数,兼容 FlashAttention,在长上下文任务中表现优异。

摘要:

自回归语言模型依赖 Key-Value (KV) Cache,这避免了在生成过程中重新计算过去的隐藏状态,从而加快了速度。随着模型规模和上下文长度的增长,KV Cache 成为一个显著的内存瓶颈,这需要压缩方法来限制其在生成过程中的大小。在本文中,我们发现了 Query (Q) 和 Key (K) 向量的惊人特性,使我们能够在不计算注意力图的情况下高效地近似注意力分数。我们提出了 Q-Filters,这是一种免训练的 KV Cache 压缩方法,它基于单一的与上下文无关的投影来过滤掉不太关键的 Key-Value 对。与许多替代方案不同,Q-Filters 兼容 FlashAttention,因为它不需要直接访问注意力权重。在长上下文设置下的实验结果表明,Q-Filters 在检索任务中与基于注意力的压缩方法(如 SnapKV)具有竞争力,同时在生成设置中始终优于高效压缩方案(如 Streaming-LLM)。值得注意的是,Q-Filters 在 x32 压缩级别下的“大海捞针”任务中实现了 99% 的准确率,同时与 Streaming-LLM 相比,在文本生成中将生成困惑度的下降幅度减少了高达 65%。

英文摘要:

Autoregressive language models rely on a Key-Value (KV) Cache, which avoids re-computing past hidden states during generation, making it faster. As model sizes and context lengths grow, the KV Cache becomes a significant memory bottleneck, which calls for compression methods that limit its size during generation. In this paper, we discover surprising properties of Query (Q) and Key (K) vectors that allow us to efficiently approximate attention scores without computing the attention maps. We propose Q-Filters, a training-free KV Cache compression method that filters out less crucial Key-Value pairs based on a single context-agnostic projection. Contrarily to many alternatives, Q-Filters is compatible with FlashAttention, as it does not require direct access to attention weights. Experimental results in long-context settings demonstrate that Q-Filters is competitive with attention-based compression methods such as SnapKV in retrieval tasks while consistently outperforming efficient compression schemes such as Streaming-LLM in generation setups. Notably, Q-Filters achieves a 99% accuracy in the needle-in-a-haystack task with a x32 compression level while reducing the generation perplexity drop by up to 65% in text generation compared to Streaming-LLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑