arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.18096cs.LGcs.AI
收藏

不让任何 Token 被落下:通过重要性感知的混合精度量化实现可靠的 KV 缓存压缩

No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization

KAIST(韩国科学技术院);NAVER Cloud(NAVER 云);POSTECH(浦项科技大学);AITRICS

June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, Dongsoo Lee

AI总结:

本文针对 KV 缓存驱逐导致的安全、幻觉与上下文丢失问题,提出 MiKV 混合精度压缩框架,对重要 KV 高精度保留、被驱逐 KV 低精度留存,实现压缩率与性能的最优权衡。

摘要:

键值(Key-Value,KV)缓存已成为加速生成式大语言模型(LLM)推理速度和吞吐量的一项关键技术。然而,KV 缓存的内存占用构成了 LLM 部署中的严重瓶颈,因为缓存大小会随批量大小和序列长度增长,甚至常常超过模型本身的大小。尽管近期已有方法提出从缓存中选择并驱逐不重要的 KV 对以降低内存消耗,但驱逐对生成过程的潜在影响尚未得到充分研究。本文考察了缓存驱逐的有害影响,并观察到:当 KV 对中包含的信息被彻底丢弃时,会出现意料之外的风险,导致安全漏洞、幻觉和上下文丢失。令人惊讶的是,我们发现即使通过降低精度量化来保留被驱逐 KV 对中的少量信息,也能大幅恢复由此造成的性能退化。另一方面,我们观察到重要的 KV 对必须以相对更高的精度保存,以保障生成质量。受这些观察启发,我们提出了 Mixed-precision KV cache(MiKV),这是一种可靠的缓存压缩方法:它通过以低精度保留被驱逐的 KV 对来同时保留上下文细节,并通过以高精度保存重要的 KV 对来确保生成质量。在多种基准和 LLM 骨干模型上的实验表明,与其他基线相比,所提方法在压缩率与性能之间实现了最先进的权衡。

英文摘要:

Key-Value (KV) Caching has become an essential technique for accelerating the inference speed and throughput of generative Large Language Models~(LLMs). However, the memory footprint of the KV cache poses a critical bottleneck in LLM deployment as the cache size grows with batch size and sequence length, often surpassing even the size of the model itself. Although recent methods were proposed to select and evict unimportant KV pairs from the cache to reduce memory consumption, the potential ramifications of eviction on the generative process are yet to be thoroughly examined. In this paper, we examine the detrimental impact of cache eviction and observe that unforeseen risks arise as the information contained in the KV pairs is exhaustively discarded, resulting in safety breaches, hallucinations, and context loss. Surprisingly, we find that preserving even a small amount of information contained in the evicted KV pairs via reduced precision quantization substantially recovers the incurred degradation. On the other hand, we observe that the important KV pairs must be kept at a relatively higher precision to safeguard the generation quality. Motivated by these observations, we propose \textit{Mixed-precision KV cache}~(MiKV), a reliable cache compression method that simultaneously preserves the context details by retaining the evicted KV pairs in low-precision and ensure generation quality by keeping the important KV pairs in high-precision. Experiments on diverse benchmarks and LLM backbones show that our proposed method offers a state-of-the-art trade-off between compression ratio and performance, compared to other baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑