ThinKV: 基于思维适应的KV缓存压缩以提高推理模型效率
ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models
AI总结：
本文提出ThinKV，通过混合量化-驱逐策略，根据思维重要性动态调整token精度，减少KV缓存占用，提升推理效率和吞吐量。
摘要：
大推理模型的长输出上下文生成使得推理链（CoT）得以扩展，但同时也导致键值（KV）缓存迅速增长，快速超出GPU内存。为解决这一挑战，我们提出ThinKV，一种基于思维适应的KV缓存压缩框架。ThinKV基于观察到注意力稀疏性揭示了不同重要性的思维类型。它应用混合量化-驱逐策略，根据思维重要性分配token精度，并逐步驱逐不重要思维中的token。此外，为实现ThinKV，我们设计了一个内核，扩展PagedAttention以实现驱逐token内存槽的高效重用，消除压缩开销。在DeepSeek-R1-Distill、GPT-OSS和NVIDIA AceReason等模型上进行的广泛实验表明，ThinKV在KV缓存占用少于5%的情况下实现了接近无损的精度，同时在状态-of-the-art基线之上提升了高达5.8倍的推理吞吐量。
英文摘要：
The long-output context generation of large reasoning models enables extended chain of thought (CoT) but also drives rapid growth of the key-value (KV) cache, quickly overwhelming GPU memory. To address this challenge, we propose ThinKV, a thought-adaptive KV cache compression framework. ThinKV is based on the observation that attention sparsity reveals distinct thought types with varying importance within the CoT. It applies a hybrid quantization-eviction strategy, assigning token precision by thought importance and progressively evicting tokens from less critical thoughts as reasoning trajectories evolve. Furthermore, to implement ThinKV, we design a kernel that extends PagedAttention to enable efficient reuse of evicted tokens' memory slots, eliminating compaction overheads. Extensive experiments on DeepSeek-R1-Distill, GPT-OSS, and NVIDIA AceReason across mathematics and coding benchmarks show that ThinKV achieves near-lossless accuracy with less than 5% of the original KV cache, while improving performance with up to 5.8x higher inference throughput over state-of-the-art baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
