arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.26557cs.DCcs.AIcs.PF
收藏

DUAL-BLADE:双路径NVMe直接KV缓存卸载用于边缘大语言模型推理

DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference

Sogang University(ソンガン大学);Florida State University(佛罗里达州立大学);Samsung Electronics Co.(三星电子公司)

Bodon Jeong, Hongsu Byun, Youngjae Kim, Weikuan Yu, Kyungkeun Lee, Jihoon Yang, Sungyong Park

AI总结:

本文提出DUAL-BLADE双路径框架,通过动态分配KV张量到页缓存或NVMe直接路径,减少I/O瓶颈,提升边缘LLM推理吞吐量。

摘要:

随着大型语言模型(LLM)在边缘AI系统中的部署增加,对在有限内存预算下高效执行的需求日益增长。键值(KV)缓存是关键挑战,因其常超出可用设备内存。尽管基于NVMe的卸载提供可扩展容量,但现有基于文件的设计依赖内核页缓存,导致缓存震荡、不可预测的延迟和高压软件开销。本文提出DUAL-BLADE,一种双路径KV驻留框架,根据运行时内存可用性动态分配KV张量到页缓存路径或NVMe直接路径。NVMe直接路径通过将KV张量映射到连续逻辑块地址(LBA)区域,实现低开销直接存储访问。DUAL-BLADE进一步结合自适应流水线并行,将存储I/O与GPU DMA重叠,提高推理吞吐量。评估显示,DUAL-BLADE显著缓解I/O瓶颈,预填和解码延迟分别降低33.1%和42.4%,同时在多样化的内存预算下提升SSD利用效率2.2倍。

英文摘要:

The increasing deployment of Large Language Model (LLM) inference on edge AI systems demands efficient execution under tight memory budgets. A key challenge arises from Key-Value (KV) caches, which often exceed available device memory. Although NVMe-based offloading offers scalable capacity, existing file-based designs rely heavily on the kernel page cache, leading to cache thrashing, unpredictable latency, and high software overhead under memory pressure. We present DUAL-BLADE, a dual-path KV residency framework that dynamically assigns KV tensors to either a page-cache path or an NVMe-direct path based on runtime memory availability. The NVMe-direct path bypasses the filesystem by mapping KV tensors to contiguous logical block address (LBA) regions, enabling low-overhead direct storage access. DUAL-BLADE further incorporates adaptive pipeline parallelism to overlap storage I/O with GPU DMA, improving inference throughput. Our evaluation shows that DUAL-BLADE substantially mitigates I/O bottlenecks, reducing prefill and decode latency by up to 33.1% and 42.4%, respectively, while improving SSD utilization by 2.2x across diverse memory budgets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑