arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2508.01002cs.LGcs.DC
收藏

LLM推理的最优调度算法:理论与实践

Optimal Scheduling Algorithms for LLM Inference: Theory and Practice

The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Agrim Bari, Parikshit Hegde, Gustavo de Veciana

AI总结:

本文提出SLAI调度器,通过动态资源分配和实时测量优化LLM推理的吞吐量和延迟,减少TTFT并提高服务容量。

摘要:

随着大型语言模型(LLM)基础工具如ChatGPT、Perplexity和Gemini在各行业中的广泛应用,对高效LLM推理系统的需求日益增加。这些系统采用独特的两阶段计算结构:预填阶段处理完整的输入提示,解码阶段则逐个自回归生成令牌。这种结构要求新的路由和调度策略。在本文中,我们通过开发一个理论框架来解决这一挑战,该框架模型了LLM推理系统中的路由和调度。我们识别出两个关键设计原则——最优分块和动态资源分配——对于实现高吞吐量至关重要。基于这些原则,我们提出了资源感知动态(RAD)调度器,并证明其在温和条件下能够实现吞吐量最优。为了解决实际的服务级别目标(SLOs)如不同时间令牌间隔(TBT)约束下的请求服务,我们设计了SLO感知的LLM推理(SLAI)调度器。SLAI利用实时测量来优先处理接近错过TBT截止时间的解码请求,并根据已知的提示长度重新排序预填请求以进一步减少时间到第一个令牌(TTFT)延迟。我们使用Mistral-7B模型在NVIDIA RTX ADA 6000 GPU上评估SLAI,基于Openchat ShareGPT4数据集。与Sarathi-Serve相比,SLAI将中位TTFT减少了53%,并提高了最大服务容量26%,使得中位TTFT低于0.5秒,同时满足尾部TBT延迟约束。

英文摘要:

With the growing use of Large Language Model (LLM)-based tools like ChatGPT, Perplexity, and Gemini across industries, there is a rising need for efficient LLM inference systems. These systems handle requests with a unique two-phase computation structure: a prefill-phase that processes the full input prompt and a decode-phase that autoregressively generates tokens one at a time. This structure calls for new strategies for routing and scheduling requests. In this paper, we take a comprehensive approach to this challenge by developing a theoretical framework that models routing and scheduling in LLM inference systems. We identify two key design principles-optimal tiling and dynamic resource allocation-that are essential for achieving high throughput. Guided by these principles, we propose the Resource-Aware Dynamic (RAD) scheduler and prove that it achieves throughput optimality under mild conditions. To address practical Service Level Objectives (SLOs) such as serving requests with different Time Between Token (TBT) constraints, we design the SLO-Aware LLM Inference (SLAI) scheduler. SLAI uses real-time measurements to prioritize decode requests that are close to missing their TBT deadlines and reorders prefill requests based on known prompt lengths to further reduce the Time To First Token (TTFT) delays. We evaluate SLAI on the Openchat ShareGPT4 dataset using the Mistral-7B model on an NVIDIA RTX ADA 6000 GPU. Compared to Sarathi-Serve, SLAI reduces the median TTFT by 53% and increases the maximum serving capacity by 26% such that median TTFT is below 0.5 seconds, while meeting tail TBT latency constraints.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑