arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.00516cs.AI
收藏

基于阈值的独占批处理用于LLM推理

Threshold-Based Exclusive Batching for LLM Inference

The Hong Kong Polytechnic University(香港理工大学);Engineering Center for Space Utilization, Chinese Academy of Sciences(中国科学院空间利用工程技术中心)

Weifang Zhang, Yuzhou Nie, Bowen Pang, Guangrui Ma, Shining Wu

AI总结:

针对混合批处理中预填充与解码干扰导致边际成本上升的问题,提出基于GPU内存带宽、模型大小和工作负载的EB-MB性能交叉条件及最优切换阈值,优化后的独占批处理在带宽受限GPU上吞吐量提升高达41.9%。

摘要:

混合批处理(MB)——将预填充和解码交错在单个批次中——已成为大型语言模型(LLM)推理的标准调度策略,因其在最大化计算和内存利用率方面的效率。然而,通过受控实验,我们发现预填充-解码干扰使MB的每步边际成本高于纯解码。在高带宽H200(4.8 TB/s)上,这仅在解码token超过批次的80%时发生;然而,在带宽受限的RTX PRO 6000(1.792 TB/s)上,该阈值骤降至仅20%。因此,MB与独占批处理(EB)之间的最优选择根本上取决于GPU内存带宽、模型大小和工作负载组成。我们推导了该EB-MB性能交叉的闭式条件,以及渐近最优的相位切换阈值和EB的内存安全批次大小。优化的EB在带宽受限GPU上吞吐量提升高达41.9%,而MB在具有更大模型的高带宽硬件上保持优势。我们的混合调度器EB+在线应用该条件,在无需人工干预的情况下动态切换EB和MB。在分布或并发度变化的非平稳流量下,EB+在每个设置中达到最高或接近最高的吞吐量,比MB高出高达36.4%。

英文摘要:

Mixed batching (MB)--interleaving prefill and decode in a single batch--has become the standard scheduling strategy for large language model (LLM) inference due to its efficiency in maximizing compute and memory utilization. However, through controlled experiments, we find that prefill-decode interference inflates MB's per-step marginal cost above that of pure decode. On the high-bandwidth H200 (4.8 TB/s), this occurs only when decode tokens exceed 80% of the batch; however, on the bandwidth-constrained RTX PRO 6000 (1.792 TB/s), this threshold plummets to just 20%. Consequently, the optimal choice between MB and exclusive batching (EB) fundamentally depends on GPU memory bandwidth, model size, and workload composition. We derive a closed-form condition for this EB-MB performance crossover, along with asymptotically optimal phase-switching thresholds and memory-safe batch sizing for EB. Optimized EB achieves up to 41.9% higher throughput on bandwidth-constrained GPUs, while MB retains its advantage on high-bandwidth hardware with larger models. Our hybrid scheduler EB+ applies this condition online to dynamically switch between EB and MB without manual intervention. Under non-stationary traffic with distribution or concurrency shifts, EB+ attains the highest or near-highest throughput in every setting, outperforming MB by up to 36.4%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑