Range:将KV缓存重用粒度与存储工作解耦
Range: Decoupling KV-Cache Reuse Granularity from Storage Work
AI总结:
Range是面向vLLM和LMCache的KV缓存存储引擎,将KV缓存存储路径成本与逻辑对象解耦,在6 GiB恢复任务中,其中位TTFT在32倍对象数量范围内稳定在1.79-1.80秒,性能优于LMCache及原生路径。
摘要:
大语言模型(LLM)服务通过复用缓存的键值(KV)状态来跳过预填充步骤,而不断增长的上下文会将该状态从GPU高带宽内存(HBM)、CPU动态随机存取存储器(DRAM)转移到非易失性内存 express(NVMe)固态硬盘(SSD)上,这使得前缀恢复成为首令牌生成时间(TTFT)关键路径的一部分。我们发现,在配置良好的主机发出路径中,SSD并非瓶颈:恢复相同6 GiB的KV数据时,以192个逻辑对象为单位需2.1秒,以6144个逻辑对象为单位则需3.0秒,分层加载模式下耗时更长,因为存储栈的每一层(查找、分配、I/O提交、GPU传输、存储、索引)都会按每个逻辑对象计费,而总字节数保持不变。该成本随KV占用量增长而上升:更长的上下文、分层加载下更深的模型、张量并行性都会使对象数量成倍增加。现有系统仅将这种按对象计算的工作从CPU转移到GPU,但该工作仍按对象计算。本文提出Range,这是一个面向vLLM和LMCache的KV缓存存储引擎,基于一个核心原则:存储路径上的所有成本按字节或物理批次扩展,绝不按逻辑对象扩展。Range用令牌元数据和物理租约替代按键的查找与所有权;字节条带化使介质上的布局独立于重用粒度;完成方所有的GPU传输库对传输与提取进行批处理,并在每一层就绪时将其释放给注意力机制。Range保留内核cuFile路径,仅修改vLLM的4行代码。在固定6 GiB恢复任务中,Range的中位TTFT在32倍对象数量范围内及分层传输模式下均保持在1.79-1.80秒,与前缀已驻留GPU内存的情况仅相差70毫秒,而调优后的进程内LMCache则从2.12秒升至3.01秒。Range在粗粒度下比最快的多进程DRAM中转路径快0.44秒,且不通过主机内存传输有效载荷,同时保持分层重用的成本与粗粒度重用相当,而最优的原生路径耗时为2.24秒。
英文摘要:
LLM serving reuses cached key-value (KV) state to skip prefill, and growing contexts push that state past GPU HBM and CPU DRAM onto NVMe SSDs, placing prefix restore on the critical path of time-to-first-token (TTFT). We find that on a well-configured host-issued path the SSD is not the bottleneck: restoring the same 6 GiB of KV takes 2.1 s as 192 objects but 3.0 s as 6,144 objects, and more under layerwise loading, because every layer of the storage stack - lookup, allocation, I/O submission, GPU delivery, store, index - charges per logical object while the bytes stay the same. This cost grows with the KV footprint: longer contexts, deeper models under layerwise loading, and tensor parallelism each multiply the object count. Prior systems move this per-object work from the CPU to the GPU; the work remains per object. This paper presents Range, a KV-cache storage engine for vLLM and LMCache built on one principle: every cost on the storage path scales with bytes or physical batches, never with logical objects. Range tickets replace per-key lookup and ownership with token metadata and physical leases; byte stripes make the on-media layout independent of reuse granularity; completion-owned GPU transfer banks batch delivery and extraction and release each layer to attention as it lands. Range keeps the kernel cuFile path and changes four lines of vLLM. On a fixed 6 GiB restore, Range's median TTFT stays at 1.79-1.80 s across a 32x range of object counts and under layerwise delivery, within 70 ms of the prefix already being resident in GPU memory, while tuned in-process LMCache rises from 2.12 s to 3.01 s. Range improves on the fastest multiprocess DRAM-bounced path by 0.44 s at coarse granularity while moving no payload through host memory, and holds layerwise reuse at the cost of coarse-grained reuse, where the best stock path takes 2.24 s.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。