KeepKV: 实现高效的大型语言模型推理中的周期性无损KV缓存压缩
KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
AI总结：
KeepKV通过自适应KV缓存合并方法，在严格内存限制下实现高效LLM推理，实现无损压缩并提升生成质量。
摘要：
大型语言模型（LLMs）高效推理受到不断增长的键值（KV）缓存的阻碍，使KV缓存压缩成为关键研究方向。传统方法选择性地驱逐不重要的KV缓存条目，导致信息丢失和幻觉。最近，基于合并的策略已被探索，通过合并会被丢弃的KV对来保留更多信息；然而，这些现有方法不可避免地在合并前后引入注意力分布的不一致，导致生成质量下降。为克服这一挑战，我们提出了KeepKV，一种新的自适应KV缓存合并方法，旨在在严格内存约束下保持性能，实现单步无损压缩，并为多步压缩提供误差界。KeepKV引入了选举投票机制，记录合并历史并自适应调整注意力分数。此外，它进一步利用一种新的零推理扰动合并方法，补偿由于缓存合并导致的注意力损失。在各种基准和LLM架构上的广泛实验表明，KeepKV显著减少了内存使用，同时成功保留了关键上下文信息，实现了超过2倍的推理吞吐量提升，并在仅使用10%的KV缓存预算时仍保持优越的生成质量。
英文摘要：
Efficient inference of large language models (LLMs) is hindered by an ever-growing key-value (KV) cache, making KV cache compression a critical research direction. Traditional methods selectively evict less important KV cache entries, which leads to information loss and hallucinations. Recently, merging-based strategies have been explored to retain more information by merging KV pairs that would be discarded; however, these existing approaches inevitably introduce inconsistencies in attention distributions before and after merging, causing degraded generation quality. To overcome this challenge, we propose KeepKV, a novel adaptive KV cache merging method designed to preserve performance under strict memory constraints, achieving single-step lossless compression and providing error bounds for multi-step compression. KeepKV introduces the Electoral Votes mechanism that records merging history and adaptively adjusts attention scores. Moreover, it further leverages a novel Zero Inference-Perturbation Merging method, compensating for attention loss resulting from cache merging. Extensive experiments on various benchmarks and LLM architectures demonstrate that KeepKV substantially reduces memory usage while successfully retaining essential context information, achieving over 2x inference throughput improvement and maintaining superior generation quality even with only 10% KV cache budgets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
