ZigZagkv：基于层不确定性的长上下文建模动态KV缓存压缩方法
ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty
AI总结：
针对大语言模型推理中KV缓存随长度增长易致内存不足的问题，提出基于层不确定性动态分配各层保留预算的ZigZagkv压缩方法，可将KV缓存内存降至全KV推理的约20%且性能近乎无损。
摘要：
大语言模型（LLMs）已成为研究热点。为加速LLMs推理，将计算得到的缓存存储在内存中已成为标准技术。然而，随着推理长度增加，不断增长的KV缓存可能引发内存不足问题。现有许多方法通过KV缓存压缩解决该问题，主要思路是在所有层中保留关键token以减少信息损失，且大多为每一层分配统一的保留预算规模。但我们从注意力和隐藏状态输出的角度观察发现，不同层和不同模型保留关键信息所需的最小预算规模存在差异。基于这一观察，本文提出一种简单且有效的KV缓存压缩方法，利用层不确定性为每一层分配预算规模。实验结果表明，与全KV推理相比，所提方法可将KV缓存的内存占用降至仅约20%，同时实现近乎无损的性能。
英文摘要：
Large Language models (LLMs) have become a research hotspot. To accelerate the inference of LLMs, storing computed caches in memory has become the standard technique. However, as the inference length increases, growing KV caches might lead to out-of-memory issues. Many existing methods address this issue through KV cache compression, primarily by preserving key tokens throughout all layers to reduce information loss. Most of them allocate a uniform budget size for each layer to retain. However, we observe that the minimum budget sizes needed to retain essential information vary across layers and models based on the perspectives of attention and hidden state output. Building on this observation, this paper proposes a simple yet effective KV cache compression method that leverages layer uncertainty to allocate budget size for each layer. Experimental results show that the proposed method can reduce memory usage of the KV caches to only $\sim$20\% when compared to Full KV inference while achieving nearly lossless performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
