arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.02252cs.CL
收藏

压缩长上下文LLM推理中的KV缓存:基于层间注意力相似性

Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity

Da Ma, Lu Chen, Situo Zhang, Yuxun Miao, Su Zhu, Zhi Chen, Hongshen Xu, Hanqi Li, Shuai Fan, Lei Pan, Kai Yu

AI总结:

针对长上下文LLM推理中KV缓存内存增长问题,提出PoD框架,基于层间注意力相似性共享远端令牌键状态,减少高达35%内存且不损失性能。

摘要:

大语言模型(LLMs)上下文窗口大小的快速扩展使其能够处理涉及长文档的日益复杂的任务。然而,这一进展以推理期间内存使用量大幅增加为代价,这主要是由于键值(KV)缓存的线性增长。现有的KV缓存压缩方法通常会丢弃相关性较低的令牌,当关键信息丢失时,这可能导致显著的性能下降。在本文中,我们提出PoD(近端令牌优于远端令牌),一种新颖的KV缓存压缩框架,根据令牌重要性分配内存,将不太重要的令牌以更紧凑的共享形式保留,而不是完全丢弃它们。我们的方法受到两个关键观察的启发:(1)近端令牌——即上下文开头和结尾的令牌——对于下一个令牌预测显著更重要,(2)远端令牌的注意力分数在连续层之间高度冗余。利用这些见解,PoD为近端令牌保留完整的KV缓存,而对于远端令牌,它在层间共享键状态。由于注意力分数由查询和键共同决定,共享键状态使多个层能够为远端令牌重用一组键,从而在不丢弃必要上下文的情况下大幅减少KV缓存内存。我们进一步引入了一种轻量级的训练后适应,使模型能够适应这种新的注意力共享结构。在合成(大海捞针)和真实世界长上下文基准上的大量实验表明,PoD在不影响性能的情况下将KV缓存内存使用量减少了高达35%。我们的方法与现有的基于令牌选择的技术正交,可以与之结合以进一步压缩KV缓存。

英文摘要:

The rapid expansion of context window sizes in Large Language Models~(LLMs) has enabled them to tackle increasingly complex tasks involving lengthy documents. However, this progress comes at the cost of a substantial increase in memory usage during inference, primarily due to the linear growth of the key-value~(KV) cache. Existing KV cache compression methods often discard less relevant tokens, which can lead to significant performance degradation when critical information is lost. In this paper, we propose \textsc{PoD}~(Proximal tokens over Distant tokens), a novel KV cache compression framework that allocates memory according to token importance, retaining less important tokens in a more compact, shared form rather than discarding them entirely. Our approach is motivated by two key observations: (1) proximal tokens -- those at the beginning and end of the context -- are significantly more important for next-token prediction, and (2) attention scores for distant tokens are highly redundant across consecutive layers. Leveraging these insights, \textsc{PoD} preserves the full KV cache for proximal tokens, while for distant tokens, it shares key states across layers. Since attention scores are determined by both queries and keys, sharing key states enables multiple layers to reuse a single set of keys for distant tokens, substantially reducing KV cache memory without discarding essential context. We further introduce a lightweight post-training adaptation to enable the model to adjust to this new attention-sharing structure. Extensive experiments on both synthetic~(Needle in a Haystack) and real-world long-context benchmarks demonstrate that \textsc{PoD} reduces KV cache memory usage by up to 35\% without compromising performance. Our method is orthogonal to existing token-selection-based techniques and can be combined with them for further KV cache compression.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑