发表机构
Lake Washington School District; XSchool; UMass Amherst（华盛顿湖学区; XSchool; 马萨诸塞大学阿默斯特分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
MetaKV提出自适应KV缓存压缩框架，根据用户指定的延迟和内存预算为每个提示词选择最优配置，在多种约束下显著提升受限成功率。
AI 中文摘要
键值（KV）缓存压缩是减少大型语言模型（LLM）推理内存开销的有效方法，尤其适用于长上下文工作负载。然而，现有的压缩方法在准确性、推理延迟和峰值KV缓存内存利用率之间做出了不同的权衡，使得单一的固定配置无法适用于不同的提示词和资源约束。我们提出了MetaKV，一个自适应框架，它根据用户指定的延迟和峰值内存预算，为每个输入提示词选择KV缓存压缩配置。MetaKV使用轻量级预测模型来估计每个候选配置的端到端延迟、峰值内存和正确响应的概率，并选择最能满足延迟-内存约束同时保持准确性的配置。我们在来自三种代表性KV缓存压缩方法（KVQuant、H$_2$O和RocketKV）的十种配置以及一种未压缩的FP16配置上评估了MetaKV，覆盖了数学、科学、常识推理和阅读理解四个数据集。在广泛的延迟和峰值内存约束范围内，MetaKV始终优于最佳静态配置，将受限成功率（CSR）——即同时满足两个约束条件下正确回答的提示词比例——平均提高了约0.07，最高提高了0.135。这些结果证明了根据单个提示词和延迟-内存约束自适应调整KV缓存压缩的益处。代码可在以下网址获取：此https URL。
英文摘要
Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-context workloads. However, existing compression methods make different trade-offs among accuracy, inference latency, and peak KV cache memory utilization, making a single fixed configuration unsuitable across different prompts and resource constraints. We introduce MetaKV, an adaptive framework that selects a KV cache compression configuration for each input prompt based on user-specified latency and peak memory budgets. MetaKV uses lightweight prediction models to estimate the end-to-end latency, peak memory, and probability of a correct response for each candidate configuration, and selects the configuration that best satisfies the latency-memory constraints while preserving accuracy. We evaluate MetaKV across ten configurations from three representative KV cache compression methods, KVQuant, H$_2$O, and RocketKV, together with an uncompressed FP16 configuration, on four datasets covering mathematics, science, commonsense reasoning, and reading comprehension. Across a wide range of latency and peak memory constraints, MetaKV consistently outperforms the best static configuration, improving constrained success rate (CSR), the fraction of prompts answered correctly while satisfying both constraints, by approximately 0.07 on average and up to 0.135. These results demonstrate the benefit of adapting KV cache compression to individual prompts and latency-memory constraints. Code is available at https://github.com/MichaelWang0505/MetaKV.git
