KV-Repurpose:用于评估非前缀KV缓存复用的测试平台
KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse
AI总结:
KV-Repurpose是基于vLLM的开源测试平台,可评估非前缀KV缓存复用,其在30 GB缓存容量下的令牌级缓存命中率达63.9%,远高于前缀缓存的3.3%,且在单文档任务中基本保持输出质量。
摘要:
在大语言模型服务中,KV缓存复用通常仅限于精确的提示前缀,当相同的检索文档出现在不同的系统提示、对话历史或动态元数据之后时,便无法复用。我们提出KV-Repurpose,这是一个基于vLLM的开源系统,用于开发和评估超越精确前缀的分段KV缓存复用。该系统结合了感知缓存的分段调度、提示构建、容量管理、工作负载生成和运行时监控。在长上下文基准测试上的实验表明,朴素的分段复用在单文档问答和摘要任务中基本保持了输出质量,但在跨文档推理中暴露出明显的局限性。在Zipfian检索增强生成(RAG)风格的工作负载下,当20%的请求共享固定前缀时,在相同的30 GB缓存容量下,KV-Repurpose的令牌级缓存命中率远高于前缀缓存,分别为63.9%和3.3%,从而降低了首令牌延迟(TTFT)。即使缓存容量达到1 TB,前缀缓存在此工作负载下也仅达到17.9%的命中率。源代码根据Apache-2.0许可证可在该https URL获取。
英文摘要:
KV-cache reuse in large language model serving is commonly limited to exact prompt prefixes, preventing reuse when the same retrieved document appears after different system prompts, conversation histories, or dynamic metadata. We present KV-Repurpose, an open-source, vLLM-based system for developing and evaluating segment-wise KV-cache reuse beyond exact prefixes. The system combines cache-aware segment scheduling, prompt construction, capacity management, workload generation, and runtime monitoring. Experiments on long-context benchmarks show that naive segment reuse largely preserves output quality for single-document question answering and summarization, while exposing clear limitations in cross-document reasoning. Under a Zipfian RAG-style workload in which 20% of requests share a fixed prefix, KV-Repurpose achieves a substantially higher token-level cache hit rate than prefix caching at the same 30 GB cache capacity, 63.9% versus 3.3%, resulting in lower TTFT. Even with 1 TB of cache capacity, prefix caching reaches only a 17.9% hit rate in this workload. Source code is available under the Apache-2.0 license at https://github.com/taiga333/eval-kv-repurpose.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。