arXiv 2510.225562025-10-26cs.CL

SABlock：具有自适应压缩块大小的语义感知KV缓存驱逐

SABlock: Semantic-Aware KV Cache Eviction with Adaptive Compression Block Size

School of Computer Science and Technology, University of Science and Technology of China（计算机科学与技术学院，科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China（苏州先进研究院，科学技术大学）

Jinhan Chen, Jianchun Liu, Hongli Xu, Xianjun Gao, Shilong Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

SABlock通过语义感知的自适应块大小KV缓存驱逐，在长上下文LLM推理中平衡语义完整性与内存效率，显著提升检索准确率并降低内存与解码延迟。

摘要：

键值（KV）缓存不断增长的内存占用对长上下文大型语言模型（LLM）推理构成了严重的可扩展性瓶颈。虽然KV缓存驱逐通过丢弃不太关键的令牌已成为一种有效的解决方案，但现有的令牌级、块级和句子级压缩方法难以在语义连贯性和内存效率之间取得平衡。为此，我们引入了SABlock，一个具有自适应块大小的语义感知KV缓存驱逐框架。具体来说，SABlock首先执行语义分割，使压缩边界与语言结构对齐，然后应用段引导的令牌评分来细化令牌重要性估计。最后，对于每个段，预算驱动的搜索策略自适应地确定最佳块大小，在给定的缓存预算下保持语义完整性，同时提高压缩效率。在长上下文基准上的大量实验表明，在相同的内存预算下，SABlock始终优于最先进的基线。例如，在“大海捞针”（NIAH）测试中，SABlock仅用96个KV条目就实现了99.9%的检索准确率，几乎与保留多达8K条目的全缓存基线的性能相当。在固定的1,024缓存预算下，SABlock进一步将峰值内存使用量减少了46.28%，并在128K上下文长度下实现了高达9.5倍的解码加速。

英文摘要：

The growing memory footprint of the Key-Value (KV) cache poses a severe scalability bottleneck for long-context Large Language Model (LLM) inference. While KV cache eviction has emerged as an effective solution by discarding less critical tokens, existing token-, block-, and sentence-level compression methods struggle to balance semantic coherence and memory efficiency. To this end, we introduce SABlock, a \underline{s}emantic-aware KV cache eviction framework with \underline{a}daptive \underline{block} sizes. Specifically, SABlock first performs semantic segmentation to align compression boundaries with linguistic structures, then applies segment-guided token scoring to refine token importance estimation. Finally, for each segment, a budget-driven search strategy adaptively determines the optimal block size that preserves semantic integrity while improving compression efficiency under a given cache budget. Extensive experiments on long-context benchmarks demonstrate that SABlock consistently outperforms state-of-the-art baselines under the same memory budgets. For instance, on Needle-in-a-Haystack (NIAH), SABlock achieves 99.9% retrieval accuracy with only 96 KV entries, nearly matching the performance of the full-cache baseline that retains up to 8K entries. Under a fixed cache budget of 1,024, SABlock further reduces peak memory usage by 46.28% and achieves up to 9.5x faster decoding on a 128K context length.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。