CaliDrop：基于校准的KV缓存压缩
CaliDrop: KV Cache Compression with Calibration
AI总结：
本文提出CaliDrop，一种通过校准增强令牌驱逐的KV缓存压缩策略，利用邻近查询相似性对丢弃令牌进行推测性校准，显著提升现有驱逐方法的精度。
摘要：
大型语言模型（LLMs）在生成过程中需要大量的计算资源。虽然键值（KV）缓存通过存储注意力中间结果显著加速了这一过程，但其内存占用随序列长度、批处理大小和模型规模线性增长，在长上下文场景中成为瓶颈。为缓解这一瓶颈，研究者提出了多种KV缓存压缩技术，包括令牌驱逐、量化和低秩投影，这些技术往往相互补充。本文聚焦于增强令牌驱逐策略。令牌驱逐利用了注意力模式通常稀疏的观察，允许移除不太关键的KV条目以节省内存。然而，这种缩减通常以显著的精度下降为代价，尤其是在高压缩比下。为解决此问题，我们提出了CaliDrop，一种通过校准增强令牌驱逐的新策略。我们的初步实验表明，邻近位置的查询表现出高度相似性。基于这一观察，CaliDrop对丢弃的令牌执行推测性校准，以减轻令牌驱逐引起的精度损失。大量实验证明，CaliDrop显著提高了现有令牌驱逐方法的精度。
英文摘要：
Large Language Models (LLMs) require substantial computational resources during generation. While the Key-Value (KV) cache significantly accelerates this process by storing attention intermediates, its memory footprint grows linearly with sequence length, batch size, and model size, creating a bottleneck in long-context scenarios. Various KV cache compression techniques, including token eviction, quantization, and low-rank projection, have been proposed to mitigate this bottleneck, often complementing each other. This paper focuses on enhancing token eviction strategies. Token eviction leverages the observation that the attention patterns are often sparse, allowing for the removal of less critical KV entries to save memory. However, this reduction usually comes at the cost of notable accuracy degradation, particularly under high compression ratios. To address this issue, we propose \textbf{CaliDrop}, a novel strategy that enhances token eviction through calibration. Our preliminary experiments show that queries at nearby positions exhibit high similarity. Building on this observation, CaliDrop performs speculative calibration on the discarded tokens to mitigate the accuracy loss caused by token eviction. Extensive experiments demonstrate that CaliDrop significantly improves the accuracy of existing token eviction methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
