针对长上下文服务，跨任务质量和系统性能对KV缓存优化进行基准测试
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
AI总结：
研究长上下文服务中KV缓存优化，通过工作负载感知基准评估多种优化机制，测量多项指标。发现仅压缩率不能预测性能，不同机制表现各异，为长上下文服务系统提供工作负载感知的KV缓存机制选择及部署指导。
摘要：
大语言模型服务在长上下文工作负载下越来越受KV缓存增长的限制，而现有KV缓存压缩技术因在不同模型、任务、预算和服务栈上评估而难以比较。本文提出了一个工作负载感知基准，涵盖量化、剪枝和合并等代表性KV缓存优化机制，在多种工作负载和模型上进行评估。基准测量任务质量、平均输出吞吐量等指标。结果表明仅压缩率不能预测端到端性能，不同机制各有优劣，为长上下文服务系统提供了部署指导。
英文摘要：
Large language model serving is increasingly limited by KV-cache growth under long-context workloads, yet existing KV-cache compression techniques are difficult to compare because they were evaluated on different models, tasks, budgets, and serving stacks. This paper presents a workload-aware benchmark of representative KV-cache optimization mechanisms spanning quantization, pruning, and merging, including KIVI, TurboQuant, SnapKV, and CaM, evaluated on LongBench-style multi-document QA, single-document QA, few-shot learning, and summarization workloads using Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3. The benchmark measures task quality, mean output throughput, mean time-to-first-token, and realized compression ratio across context-length buckets. The results show that the compression ratio alone is a poor predictor of end-to-end performance. KIVI4 provides the most stable quality across models, SnapKV delivers the strongest long-context throughput, and CaM yields large gains on selected QA workloads but exhibits substantial workload sensitivity in both quality and realized compression ratio. These findings motivate workload-aware selection of KV-cache mechanisms rather than one-size-fits-all compression and provide deployment guidance for long-context serving systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
