arXiv 2511.047912025-11cs.LG

DuetServe: 通过自适应GPU多路复用协调LLM服务的预填充与解码

DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing

University of California, Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）；University of Toronto（多伦多大学）；University of Washington（华盛顿大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）

Lei Gao, Chaoyi Jiang, Hossein Entezari Zarch, Daniel Wong, Mark Hill, Murali Annavaram

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对LLM服务中预填充与解码阶段的干扰问题，提出DuetServe框架，通过自适应SM级GPU空间多路复用实现单GPU内的阶段隔离，在保证低延迟的同时提升吞吐量。

摘要：

现代LLM服务系统必须维持高吞吐量，同时满足两个不同推理阶段（计算密集型预填充阶段和内存受限的解码阶段）的严格延迟SLO。现有方法要么（1）在共享GPU上聚合两个阶段，导致预填充和解码阶段之间的干扰，从而降低令牌间时间（TBT）；要么（2）将两个阶段分离到不同GPU上，改善延迟但通过重复模型和KV缓存传输浪费资源。我们提出DuetServe，一个统一的LLM服务框架，在单个GPU内实现分离级别的隔离。DuetServe默认以聚合模式运行，并在预测到TBT下降时动态激活SM级GPU空间多路复用。其关键思想是仅在需要时通过细粒度、自适应的SM分区解耦预填充和解码执行，仅在争用威胁延迟服务级别目标时提供阶段隔离。DuetServe集成了（1）一个注意力感知的屋顶线模型以预测迭代延迟，（2）一个分区优化器，选择最优SM分割以在TBT约束下最大化吞吐量，以及（3）一个无中断执行引擎，消除CPU-GPU同步开销。评估表明，与最先进框架相比，DuetServe在保持低生成延迟的同时，总吞吐量提升高达1.3倍。

英文摘要：

Modern LLM serving systems must sustain high throughput while meeting strict latency SLOs across two distinct inference phases: compute-intensive prefill and memory-bound decode phases. Existing approaches either (1) aggregate both phases on shared GPUs, leading to interference between prefill and decode phases, which degrades Time-Between-Tokens (TBT); or (2) disaggregate the two phases across GPUs, improving latency but wasting resources through duplicated models and KV cache transfers. We present DuetServe, a unified LLM serving framework that achieves disaggregation-level isolation within a single GPU. DuetServe operates in aggregated mode by default and dynamically activates SM-level GPU spatial multiplexing when TBT degradation is predicted. Its key idea is to decouple prefill and decode execution only when needed through fine-grained, adaptive SM partitioning that provides phase isolation only when contention threatens latency service level objectives. DuetServe integrates (1) an attention-aware roofline model to forecast iteration latency, (2) a partitioning optimizer that selects the optimal SM split to maximize throughput under TBT constraints, and (3) an interruption-free execution engine that eliminates CPU-GPU synchronization overhead. Evaluations show that DuetServe improves total throughput by up to 1.3x while maintaining low generation latency compared to state-of-the-art frameworks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。