发表机构
Baseten（巴斯滕）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
WavePP通过异步前缀查找和动态块规划，在TensorRT-LLM上实现高吞吐预填充，显著提升多模型并发下的性能。
AI 中文摘要
流水线并行可以通过在模型的不同阶段并发处理多个请求块来提高预填充吞吐量。然而，保持流水线充分利用需要高效的调度和请求准备。在阶段独立保留和驱逐缓存状态的系统中，本地缓存命中并不能保证同一前缀可以在整个流水线中复用。在这种情况下，协调开销可能阻碍请求接纳节奏，从而降低整体吞吐量。在本文中，我们提出了WavePP，一个构建在TensorRT-LLM之上的预填充运行时，通过将请求接纳与流水线执行重叠来解决这些挑战。WavePP异步地找到可以在所有阶段复用的前缀，保护缓存状态，并在早期请求继续执行时为剩余输入预留空间。随后，它动态规划每个请求的块大小以最大化流水线填充。每个阶段在执行请求之前完成本地准备。在相同的系统和流水线拓扑下，WavePP在GLM 5.2和MiniMax M2.7上的40个测试设置中的37个中提高了TensorRT-LLM的预填充吞吐量。在并发128且高缓存复用的情况下，这些更改分别将吞吐量提高了2.91倍和2.02倍。在28个Kimi K3设置中，与来自TRT-LLM、SGLang和vLLM的张量/专家并行和流水线并行基线相比，WavePP在并发八或更高的所有18个设置中也具有最高的测量吞吐量。
英文摘要
Pipeline parallelism can improve prefill throughput by processing multiple request chunks concurrently across different stages of the model. However, keeping the pipeline fully utilized requires efficient scheduling and request preparation. In systems where stages retain and evict cache state independently, a local cache hit does not guarantee that the same prefix can be reused across the pipeline. Here, coordination overhead can impede request admission cadence and thus reduce overall throughput. In this paper, we present WavePP, a prefill runtime built on top of TensorRT-LLM that addresses these challenges by overlapping request admission with pipeline execution. WavePP asynchronously finds a prefix that can be reused across all stages, protects the cached state, and reserves space for the remaining input while earlier requests continue to execute. It subsequently plans the chunk sizes of each request dynamically to maximize pipeline fill. Each stage then completes the local preparation before executing the request. In the same system and pipeline topology, WavePP improves TensorRT-LLM's prefill throughput in 37 of 40 tested settings on GLM 5.2 and MiniMax M2.7. At concurrency 128 with high cache reuse, these changes increase throughput by factors of 2.91 and 2.02, respectively. Across 28 Kimi K3 settings, WavePP also has the highest measured throughput in all 18 settings at concurrency eight or higher, compared with tensor/expert-parallel and pipeline-parallel baselines from TRT-LLM, SGLang, and vLLM.
Comments33 pages, 14 figures, 11 tables
