arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.09665cs.LG
收藏

LMCache: 一种高效的键值缓存层用于企业级大语言模型推理

LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference

Tensormesh Inc.(Tensormesh公司)

Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, Junchen Jiang

AI总结:

LMCache是一种高效的键值缓存解决方案,通过优化数据移动和模块化设计,提升大语言模型推理的性能和可扩展性。

摘要:

传统的键值缓存(KV cache)通常存储在GPU内存中,以加速大型语言模型(LLM)推理的解码阶段。然而,将键值缓存移出GPU设备变得越来越必要,以实现跨不同查询和推理引擎的缓存重用。我们的实际使用统计数据证实了这一趋势:随着时间的推移,用户存储的总键值缓存迅速增长,远远超过了GPU内存的容量。尽管有这种需求,但缺乏一种高效的解决方案来卸载和传输键值缓存。我们提出了LMCACHE,这是首个也是目前最高效的开源键值缓存解决方案,它从现代LLM引擎(vLLM和SGLang)中提取并存储生成的键值缓存,使其脱离GPU内存,并在不同引擎和查询之间共享。LMCACHE支持两种功能:缓存卸载(跨查询的前缀重用)和预填充-解码(PD)解耦(跨引擎/GPU缓存传输)。LMCACHE的高性能和广泛应用源于以下贡献:(1)由批量数据移动操作、计算和I/O管道推动的高效键值缓存数据移动;(2)一个模块化的键值缓存连接组件,将LMCACHE与推理引擎的快速演变解耦;(3)一个一流的控制API,用于在GPU、CPU、存储和网络层之间灵活地编排缓存。我们的评估显示,将LMCACHE与vLLM结合,可在多轮问答和文档分析等任务上实现高达15倍的吞吐量提升。在企业环境中大规模采用LMCACHE为我们提供了有价值的见解,例如从远程存储中获取键值缓存对预填充延迟有意外的好处,而上下文截断,这是工业中广泛使用的技术,可以将前缀缓存命中率减少一半。LMCACHE的源代码位于:https://github.com/LMCache/LMCache。

英文摘要:

KV cache has traditionally been stored in GPU memory to accelerate the decoding phase of large language model (LLM) inference. However, it is increasingly necessary to move KV caches outside GPU devices, to enable cache reuse across different queries and inference engines. Our real-world usage statistics confirm this trend: over time, the total KV cache stored by users has grown rapidly, far exceeding the capacity of GPU memory. Despite this need, there lacks an efficient solution for offloading and transferring KV caches. We present LMCACHE, the first and so far the most efficient open-source KV caching solution, which extracts and stores KV caches generated by modern LLM engines (vLLM and SGLang) out of the GPU memory and shares them across engines and queries. LMCACHE supports both cache offloading (prefix reuse across queries) and prefill-decode (PD) disaggregation (cross-engine/GPU cache transfer). LMCACHE's high performance and wide adoption stem from the following contributions: (1) highly optimized KV cache data movement powered by batched data movement operations, compute and I/O pipelining; (2) a modular KV cache connector component, decoupling LMCACHE from the rapid evolution of inference engines; (3) a first-class control API for flexible cache orchestration across GPU, CPU, storage, and network layers. Our evaluation shows that combining LMCACHE with vLLM achieves up to 15x improvement in throughput across workloads such as multi-round question answering and document analysis. Large-scale adoption of LMCACHE in enterprise settings provides us valuable insights, for example, fetching KV cache from remote storage has unsurprisingly benefits to prefill delay, and that context truncation, which is a widely applied technique in industry, can greatly reduce prefix cache hit ratio by half. The source code of LMCACHE is at: https://github.com/LMCache/LMCache.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑