arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.28095cs.DC
收藏

SiDP:面向离线LLM推理的内存高效数据并行

SiDP: Memory-Efficient Data Parallelism for Offline LLM Inference

scitix

Alan Zhao, Cyril Y. He

AI总结:

提出SiDP,一种将权重视为带宽支持的共享资源的内存高效数据并行范式,通过Weight-as-a-Service和Compute-as-a-Service两种模式,在相同配置下将可用KV容量提升至1.8倍,端到端吞吐量提升至1.5倍。

摘要:

大型语言模型(LLM)的快速普及已将大量推理工作负载转移到面向吞吐量的离线场景中,其中充分利用GPU计算需要大批量。然而,现有部署面临结构性矛盾。数据并行(DP)可良好扩展吞吐量,但会复制模型权重,导致用于键值(KV)缓存的GPU内存有限,从而限制批量大小。模型并行减少每设备权重,但需要细粒度同步,削弱了DP的独立性和调度灵活性。我们提出SiDP,一种用于离线LLM推理的内存高效数据并行范式,将权重视为DP组内带宽支持的共享资源。SiDP并非在每个GPU上存储完整模型,而是将权重组织为分布式池:每层由单个GPU拥有,其他副本通过两种互补执行模式按需访问其权重:Weight-as-a-Service(WaS)模式在大批量场景下通过NVLink将远程权重流式传输到小缓存中,以及Compute-as-a-Service(CaS)模式在少量尾部将激活发送给所有者。在NVIDIA H20、H200和B200 GPU上使用Qwen3-32B、Qwen2.5-72B和Llama-3.1-70B进行评估,SiDP在相同配置下将可用KV容量提升至1.8倍,并将此转化为离线工作负载相比基线(vLLM)高达1.5倍的端到端吞吐量提升。

英文摘要:

The rapid adoption of large language models (LLMs) has shifted a substantial portion of inference workloads into throughput-oriented offline regimes, where fully utilizing GPU compute requires large batch sizes. However, existing deployments face a structural tension. Data parallelism (DP) scales throughput well but replicates model weights, leaving limited GPU memory for key-value (KV) cache and constraining batch size. Model parallelism reduces per-device weights, but requires fine-grained synchronization that erodes DP's independence and scheduling flexibility. We present SiDP, a memory-efficient data-parallel paradigm for offline LLM inference that treats weights as a bandwidth-backed shared resource inside a DP group. Instead of storing the full model on every GPU, SiDP organizes weights as a distributed pool: each layer is owned by a single GPU, and other replicas access its weights on demand via two complementary execution modes: a Weight-as-a-Service (WaS) mode that streams remote weights over NVLink into a small cache in the large-batch regime, and a Compute-as-a-Service (CaS) mode that ships activations to owners in the small-batch tail. Evaluated on NVIDIA H20, H200, and B200 GPUs with Qwen3-32B, Qwen2.5-72B, and Llama-3.1-70B, SiDP increases usable KV capacity by up to 1.8x under the same configurations, and converts this into up to 1.5x higher end-to-end throughput over baselines (vLLM) for offline workloads.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑