SkipKV: 为大推理模型高效推理选择性跳过KV生成与存储
SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Models
AI总结:
本文提出SkipKV方法,通过选择性跳过KV生成与存储,减少缓存大小,提升推理效率。方法引入句子评分指标,动态调整引导向量,实现高效推理,实验显示准确率提升26.7%,生成长度缩短1.6倍,吞吐量提升1.7倍。
摘要:
大推理模型(LRMs)由于其与详尽的链式推理(CoT)成线性增长的关系,导致显著的键值(KV)缓存开销,从而引发内存开销和吞吐量瓶颈,限制了高效部署。为减少推理过程中的KV缓存大小,我们首先研究了现有KV缓存驱逐方法在CoT推理中的有效性。有趣的是,我们发现由于令牌级评分不稳定以及由于填充导致的有效KV预算减少,最先进的(SoTA)驱逐方法在多批次设置中无法维持准确性。此外,这些方法往往生成比原始模型更长的序列,因为语义不敏感的令牌级驱逐导致推理过程中重复验证。为了解决这些问题,我们提出了SkipKV,一种训练无关的KV压缩方法,能够进行选择性驱逐和生成,以粗粒度、句子级别的序列移除实现高效的CoT推理。具体来说,它引入了一个句子评分度量来识别并移除高度相似的句子,同时保持语义一致性。为了抑制冗余生成,SkipKV动态调整一个引导向量以在推理过程中更新隐藏激活状态,强制LRM生成简洁的响应。在多个推理基准上的广泛评估表明,SkipKV在与基线方法相比时,准确率提高了高达26.7%,在相似的压缩预算下。此外,与SoTA相比,SkipKV生成长度缩短了高达1.6倍,吞吐量提高了高达1.7倍。我们的代码已发布在:https://github.com/TTTTTTris/SkipKV。
英文摘要:
Large reasoning models (LRMs) often incur significant key-value (KV) cache overhead, due to their linear growth with the verbose chain-of-thought (CoT) reasoning. This incurs both memory overhead and throughput bottlenecks, limiting efficient deployment. To reduce KV cache size during inference, we first investigate the effectiveness of existing KV cache eviction methods for CoT reasoning. Interestingly, we find that due to unstable token-wise scoring and reduced effective KV budget caused by padding, state-of-the-art (SoTA) eviction methods fail to maintain accuracy in multi-batch settings. Additionally, these methods often generate longer sequences than the original model without eviction, as semantic-unaware token-wise eviction leads to repeated revalidation during reasoning. To address these issues, we present \textbf{SkipKV}, a \textbf{\textit{training-free}} KV compression method that performs selective \textit{eviction} and \textit{generation}, operating at a coarse-grained, sentence-level sequence removal for efficient CoT reasoning. In specific, it introduces a \textit{sentence-scoring metric} to identify and remove highly similar sentences while maintaining semantic coherence. To suppress redundant generation, SkipKV dynamically adjusts a steering vector to update the hidden activation states during inference, enforcing the LRM to generate concise responses. Extensive evaluations on multiple reasoning benchmarks demonstrate that SkipKV achieves up to $\mathbf{26.7}\%$ higher accuracy compared to baseline methods, at a similar compression budget. Additionally, compared to SoTA, SkipKV yields up to $\mathbf{1.6}\times$ shorter generation length while improving throughput by up to $\mathbf{1.7}\times$. Our code is released at: \href{https://github.com/TTTTTTris/SkipKV}{https://github.com/TTTTTTris/SkipKV}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。