Compactor: 基于近似杠杆分数的校准查询无关KV缓存压缩
Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores
AI总结：
Compactor通过近似杠杆分数实现查询无关的KV缓存压缩，减少内存占用并提升性能，适用于多种语言模型部署场景。
摘要：
现代大型语言模型（LLM）正被越来越多地训练以支持非常大的上下文窗口。我们提出了Compactor，一种无需训练的、查询无关的KV压缩策略，利用近似杠杆分数来确定token的重要性。我们展示Compactor在合成和真实世界上下文任务中能够实现与竞争方法相同的表现，同时在保留20%更少token的同时更具任务鲁棒性。我们进一步介绍了一种上下文校准的压缩程序：在显著性能损失之前推断给定上下文支持的最大压缩。使用上下文校准的压缩，我们展示了Compactor在Longbench上实现完整的KV性能，同时平均将KV内存负担减少了68%。为了证明我们方法的有效性和通用性，我们将Compactor应用于来自RULER和Longbench的27个合成和真实世界任务，涉及Qwen 2.5和Llama 3.1家族的模型。最后，我们发布了compactor-vllm，一个推理引擎和一组优化的Triton内核套件，旨在高效支持压缩KV缓存固有的稀疏、非连续内存访问模式。本工作证明了Compactor为缓解现代LLM部署中的内存瓶颈提供了一种实用、高性能的解决方案。
英文摘要：
Modern Large Language Models (LLMs) are increasingly trained to support very large context windows. We present Compactor, a training-free, query-agnostic KV compression strategy that uses approximate leverage scores to determine token importance. We show that Compactor can achieve the same performance as competing methods while retaining 20% fewer tokens in both synthetic and real-world context tasks, while being more task-robust. We further introduce a procedure for context-calibrated compression: inferring the maximum compression a given context supports before significant performance loss. Using context-calibrated compression, we show that Compactor achieves full KV performance on Longbench while reducing the KV memory burden by 68%, on average. To demonstrate the efficacy and generalizability of our approach, we apply Compactor to 27 synthetic and real-world tasks from RULER and Longbench, with models from both the Qwen 2.5 and Llama 3.1 families. Finally, we release compactor-vllm, an inference engine and suite of optimized Triton kernels designed to efficiently support the sparse, non-contiguous memory access patterns inherent to compressed KV caches. This work demonstrates that Compactor offers a practical, high-performance solution for alleviating the memory bottleneck in modern LLM deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
