KVReviver: 基于草图算法的可逆KV缓存压缩
KVReviver: Reversible KV Cache Compression with Sketch-Based Token Reconstruction
AI总结：
KVReviver通过基于草图算法的可逆方法，在有限内存下实现高效KV缓存压缩，保持推理准确性。
摘要：
随着当前大型语言模型（LLMs）的上下文长度迅速增加，KV缓存的内存需求已成为LLM部署和批量处理的瓶颈。传统KV缓存压缩方法通常涉及永久驱逐或不可逆地合并注意力分数低的“不重要”token。这种做法导致token信息的不可恢复丢失，我们称之为上下文性遗忘，显著降低模型的信息检索能力。为了解决这一问题，我们提出KVReviver，一种基于草图算法的可逆KV缓存压缩方法。该方法允许从额外的数据结构中重建压缩的token，从而在有限内存内实现全规模计算。实验显示，在2k长度的上下文中，它仅需10%的KV缓存预算即可维持相同的端到端推理准确性。在32k长度的上下文中，它使用仅25%的KV缓存预算，实现了等同或相近的准确性（约2%的准确性损失）。
英文摘要：
As the context length of current large language models (LLMs) rapidly increases, the memory demand for the Key-Value (KV) cache is becoming a bottleneck for LLM deployment and batch processing. Traditional KV cache compression methods typically involve permanently evicting or irreversibly merging "less important" tokens with low attention scores. This approach results in the unrecoverable loss of token information, which we call Contextual Amnesia, significantly degrading the model's information retrieval capability. To address this issue, we propose KVReviver, a reversible KV cache compression method based on the sketch algorithm. This method allows reconstructing compressed tokens from an additional data structure, thus enabling full-scale computation within limited memory. Experiments showed that in 2k-length contexts, it requires only 10% of KV Cache budget while maintaining identical end-to-end inference accuracy. For 32k-length contexts, it achieves equivalent or comparable accuracy ~2% accuracy loss) using merely 25% of KV Cache budget.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
