arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.21548cs.DC
收藏

DualPath: 突破代理LLM推理中的存储带宽瓶颈

DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference

School of Computer Science, Peking University(北京大学计算机学院);Tsinghua University(清华大学);DeepSeek-AI(深度求索)

Yongtong Wu, Shaoyuan Chen, Yinmin Zhong, Rilin Huang, Yixuan Tan, Wentao Zhang, Liyue Zhang, Shangyan Zhou, Yuxuan Liu, Shunfeng Zhou, Mingxing Zhang, Xin Jin, Panpan Huang

AI总结:

DualPath通过双路径KV缓存加载技术,突破代理LLM推理中的存储带宽瓶颈,提升推理系统吞吐量。

摘要:

多轮代理LLM推理性能越来越受到KV缓存存储I/O的限制,而非计算。在普遍的离散架构中,从外部存储加载大量的KV缓存会造成根本性的不平衡:预填引擎上的存储网络接口成为带宽饱和,而解码引擎上的接口则保持空闲。这种不对称严重限制了整体系统的吞吐量。\n我们提出了DualPath,一种推理系统,通过引入双路径KV缓存加载来打破这个瓶颈。除了传统的存储到预填路径外,DualPath还启用了新的存储到解码路径,在此路径中,KV缓存被加载到解码引擎中,然后通过计算网络上的RDMA高效地传输到预填引擎。DualPath结合了这种优化的数据路径——该路径本质上避免了网络拥堵,并避免了与延迟关键的模型执行通信的干扰——与一个全局调度器,该调度器动态平衡预填和解码引擎之间的负载。\n我们在三个具有生产代理工作负载的模型上进行评估,结果显示,DualPath在我们内部的推理系统中将离线推理吞吐量提高了高达1.87倍。它还能在不违反SLO的情况下,将在线服务吞吐量提高平均1.96倍。

英文摘要:

The performance of multi-turn, agentic LLM inference is increasingly dominated by KV-Cache storage I/O rather than computation. In prevalent disaggregated architectures, loading the massive KV-Cache from external storage creates a fundamental imbalance: storage NICs on prefill engines become bandwidth-saturated, while those on decoding engines remain idle. This asymmetry severely constrains overall system throughput. We present DualPath, an inference system that breaks this bottleneck by introducing dual-path KV-Cache loading. Beyond the traditional storage-to-prefill path, DualPath enables a novel storage-to-decode path, in which the KV-Cache is loaded into decoding engines and then efficiently transferred to prefill engines via RDMA over the compute network. DualPath combines this optimized data path -- which inherently avoids network congestion and avoids interference with latency-critical model execution communications -- with a global scheduler that dynamically balances load across prefill and decode engines. Our evaluation on three models with production agentic workloads demonstrates that DualPath improves offline inference throughput by up to 1.87$\times$ on our in-house inference system. It can also improve online serving throughput by an average factor of 1.96$\times$ without violating SLO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑