ChunkKV：面向高效长上下文LLM推理的语义保持KV缓存压缩
ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
AI总结：
ChunkKV以语义块为压缩单元，通过逐层索引复用技术，在保持相同压缩比下将长上下文LLM推理精度提升高达8.7%，吞吐量提高26.5%，有效解决KV缓存内存瓶颈。
摘要：
大型语言模型（LLMs）在处理长文本时需要大量GPU内存，其中键值（KV）缓存在推理过程中消耗高达总内存的70%。尽管现有的压缩方法通过评估单个标记的重要性来减少内存使用，但它们忽略了标记之间关键的语义关系，导致上下文碎片化和性能下降。我们引入了ChunkKV，它从根本上重新构想了KV缓存压缩，将语义块——而非孤立的标记——作为基本压缩单元。这种方法保留了完整的语言结构和上下文完整性，确保即使在激进压缩下也能保留基本含义。我们的创新包括一种新颖的逐层索引复用技术，该技术利用了ChunkKV中保留索引的更高跨层相似性，减少了计算开销并将吞吐量提高了26.5%。在具有挑战性的基准测试（LongBench、Needle-In-A-HayStack、GSM8K和JailbreakV）上的全面评估表明，在保持相同压缩比的同时，ChunkKV在精度上比最先进的方法高出高达8.7%。这些结果证实，语义感知压缩显著提高了长上下文LLM推理的效率和性能，为内存瓶颈问题提供了一种简单而有效的解决方案。代码可在\href{https://github.com/NVIDIA/kvpress}{链接}获取。
英文摘要：
Large Language Models (LLMs) require significant GPU memory when processing long texts, with the key value (KV) cache consuming up to 70\% of total memory during inference. Although existing compression methods reduce memory by evaluating the importance of individual tokens, they overlook critical semantic relationships between tokens, resulting in fragmented context and degraded performance. We introduce ChunkKV, which fundamentally reimagines KV cache compression by treating semantic chunks - rather than isolated tokens - as basic compression units. This approach preserves complete linguistic structures and contextual integrity, ensuring that essential meaning is retained even under aggressive compression. Our innovation includes a novel layer-wise index reuse technique that exploits the higher cross-layer similarity of preserved indices in ChunkKV, reducing computational overhead and improving throughput by 26.5\%. Comprehensive evaluations on challenging benchmarks: LongBench, Needle-In-A-HayStack, GSM8K, and JailbreakV demonstrate that ChunkKV outperforms state-of-the-art methods by up to 8.7\% in precision while maintaining the same compression ratio. These results confirm that semantic-aware compression significantly enhances both efficiency and performance for long-context LLM inference, providing a simple yet effective solution to the memory bottleneck problem. The code is available at \href{https://github.com/NVIDIA/kvpress}{link}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
