ClusterKV：在语义空间中操纵大语言模型KV缓存以实现可召回压缩
ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression
AI总结：
针对大语言模型长上下文KV缓存内存成本高、现有压缩方法准确率低的问题，提出以语义簇为粒度召回令牌的ClusterKV方法，在低缓存预算下实现高精度与高推理效率。
摘要：
大语言模型（LLM）已被广泛部署于各类应用中，且上下文长度正快速增长，以处理长文档问答、复杂逻辑推理等任务。然而，长上下文给推理效率带来了严峻挑战，包括键值（KV）缓存的高内存成本，以及大量内存访问导致的延迟上升。近期研究提出压缩KV缓存来近似计算，但这些方法要么永久驱逐令牌、后续推理时永远无法召回，要么以按文本位置划分的页面为粒度召回先前令牌，两类方法都会降低模型准确率和输出质量。为实现高效且准确的可召回KV缓存压缩，我们提出ClusterKV，它以语义簇为粒度召回令牌。我们设计并实现了用于聚类、选择、索引和缓存的高效算法与系统。实验结果表明，在32k上下文长度的各类任务上，ClusterKV仅使用1k至2k的KV缓存预算，就实现了可忽略的准确率损失，延迟最高提升2倍，解码吞吐量最高提升2.5倍。与当前最优（SoTA）的可召回KV压缩方法相比，ClusterKV在保持或超越推理效率的同时，展现出更高的模型准确率和输出质量。我们的代码可在https://github.com/sjtu-zhao-lab/ClusterKV获取。
英文摘要：
Large Language Models (LLMs) have been widely deployed in a variety of applications, and the context length is rapidly increasing to handle tasks such as long-document QA and complex logical reasoning. However, long context poses significant challenges for inference efficiency, including high memory costs of key-value (KV) cache and increased latency due to extensive memory accesses. Recent works have proposed compressing KV cache to approximate computation, but these methods either evict tokens permanently, never recalling them for later inference, or recall previous tokens at the granularity of pages divided by textual positions. Both approaches degrade the model accuracy and output quality. To achieve efficient and accurate recallable KV cache compression, we introduce ClusterKV, which recalls tokens at the granularity of semantic clusters. We design and implement efficient algorithms and systems for clustering, selection, indexing and caching. Experiment results show that ClusterKV attains negligible accuracy loss across various tasks with 32k context lengths, using only a 1k to 2k KV cache budget, and achieves up to a 2$\times$ speedup in latency and a 2.5$\times$ improvement in decoding throughput. Compared to SoTA recallable KV compression methods, ClusterKV demonstrates higher model accuracy and output quality, while maintaining or exceeding inference efficiency. Our code is available at https://github.com/sjtu-zhao-lab/ClusterKV.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
