DistServe:面向高吞吐优化的大语言模型服务,将预填充与解码分离
DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
AI总结:
DistServe 通过将 LLM 服务的预填充与解码阶段分离到不同 GPU,并协同优化资源分配与并行策略,消除了阶段间干扰,在满足 TTFT 和 TPOT 延迟约束下将可服务请求速率提升 7.4 倍或实现 12.6 倍更严格的 SLO。
摘要:
DistServe 通过将预填充和解码计算分离来提升大语言模型(LLMs)服务的性能。现有 LLM 服务系统将这两个阶段放在一起,并对所有用户和请求的预填充与解码计算进行批处理。我们发现,这种策略不仅会导致严重的预填充-解码相互干扰,还会将两个阶段的资源分配和并行计划耦合在一起。LLM 应用通常强调每个阶段的单独延迟:预填充阶段的首个 token 时间(TTFT),以及解码阶段每个请求的每输出 token 时间(TPOT)。在严格的延迟要求下,现有系统不得不优先满足其中一个延迟指标,或过度配置计算资源以同时满足两者。\nDistServe 将预填充和解码计算分配到不同的 GPU 上,从而消除了预填充-解码相互干扰。在给定应用的 TTFT 和 TPOT 要求下,DistServe 针对每个阶段联合优化资源分配和并行策略。DistServe 还根据服务集群的带宽来放置这两个阶段,以最小化由分离引起的通信开销。因此,DistServe 在每个 GPU 上同时满足 TTFT 和 TPOT 约束的前提下,显著提升了 LLM 服务性能,表现为可服务的最大请求速率。我们的评估表明,在各种流行的 LLM、应用和延迟要求下,与最先进的系统相比,DistServe 可以服务 7.4 倍更多的请求或实现 12.6 倍更严格的 SLO,同时超过 90% 的请求保持在延迟约束内。
英文摘要:
DistServe improves the performance of large language models (LLMs) serving by disaggregating the prefill and decoding computation. Existing LLM serving systems colocate the two phases and batch the computation of prefill and decoding across all users and requests. We find that this strategy not only leads to strong prefill-decoding interferences but also couples the resource allocation and parallelism plans for both phases. LLM applications often emphasize individual latency for each phase: time to first token (TTFT) for the prefill phase and time per output token (TPOT) of each request for the decoding phase. In the presence of stringent latency requirements, existing systems have to prioritize one latency over the other, or over-provision compute resources to meet both. DistServe assigns prefill and decoding computation to different GPUs, hence eliminating prefill-decoding interferences. Given the application's TTFT and TPOT requirements, DistServe co-optimizes the resource allocation and parallelism strategy tailored for each phase. DistServe also places the two phases according to the serving cluster's bandwidth to minimize the communication caused by disaggregation. As a result, DistServe significantly improves LLM serving performance in terms of the maximum rate that can be served within both TTFT and TPOT constraints on each GPU. Our evaluations show that on various popular LLMs, applications, and latency requirements, DistServe can serve 7.4x more requests or 12.6x tighter SLO, compared to state-of-the-art systems, while staying within latency constraints for > 90% of requests.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。