随着大型语言模型(LLM)基础工具如ChatGPT、Perplexity和Gemini在各行业中的广泛应用,对高效LLM推理系统的需求日益增加。这些系统采用独特的两阶段计算结构:预填阶段处理完整的输入提示,解码阶段则逐个自回归生成令牌。这种结构要求新的路由和调度策略。在本文中,我们通过开发一个理论框架来解决这一挑战,该框架模型了LLM推理系统中的路由和调度。我们识别出两个关键设计原则——最优分块和动态资源分配——对于实现高吞吐量至关重要。基于这些原则,我们提出了资源感知动态(RAD)调度器,并证明其在温和条件下能够实现吞吐量最优。为了解决实际的服务级别目标(SLOs)如不同时间令牌间隔(TBT)约束下的请求服务,我们设计了SLO感知的LLM推理(SLAI)调度器。SLAI利用实时测量来优先处理接近错过TBT截止时间的解码请求,并根据已知的提示长度重新排序预填请求以进一步减少时间到第一个令牌(TTFT)延迟。我们使用Mistral-7B模型在NVIDIA RTX ADA 6000 GPU上评估SLAI,基于Openchat ShareGPT4数据集。与Sarathi-Serve相比,SLAI将中位TTFT减少了53%,并提高了最大服务容量26%,使得中位TTFT低于0.5秒,同时满足尾部TBT延迟约束。
英文摘要:
With the growing use of Large Language Model (LLM)-based tools like ChatGPT, Perplexity, and Gemini across industries, there is a rising need for efficient LLM inference systems. These systems handle requests with a unique two-phase computation structure: a prefill-phase that processes the full input prompt and a decode-phase that autoregressively generates tokens one at a time. This structure calls for new strategies for routing and scheduling requests.
In this paper, we take a comprehensive approach to this challenge by developing a theoretical framework that models routing and scheduling in LLM inference systems. We identify two key design principles-optimal tiling and dynamic resource allocation-that are essential for achieving high throughput. Guided by these principles, we propose the Resource-Aware Dynamic (RAD) scheduler and prove that it achieves throughput optimality under mild conditions. To address practical Service Level Objectives (SLOs) such as serving requests with different Time Between Token (TBT) constraints, we design the SLO-Aware LLM Inference (SLAI) scheduler. SLAI uses real-time measurements to prioritize decode requests that are close to missing their TBT deadlines and reorders prefill requests based on known prompt lengths to further reduce the Time To First Token (TTFT) delays.
We evaluate SLAI on the Openchat ShareGPT4 dataset using the Mistral-7B model on an NVIDIA RTX ADA 6000 GPU. Compared to Sarathi-Serve, SLAI reduces the median TTFT by 53% and increases the maximum serving capacity by 26% such that median TTFT is below 0.5 seconds, while meeting tail TBT latency constraints.
Beyond Prediction: Tail-Aware Scheduling for LLM Inference
Cornell University, Computer Science Department(康奈尔大学计算机科学系);Cornell University, Electrical and Computer Engineering Department(康奈尔大学电气与计算机工程系);Cornell University, Operations Research and Information Engineering Department(康奈尔大学运筹学与信息工程系);Microsoft Azure System Research(微软Azure系统研究);NVIDIA Corporation(英伟达公司)
Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latency
School of Cyber Science and Engineering, Xi’an Jiaotong University(西安交通大学计算机科学与工程学院);School of Computer Science and Engineering, The University of Aizu(台湾花莲大学计算机科学与工程学院)
Hierarchical Online-Scheduling for Energy-Efficient Split Inference with Progressive Transmission
School of Electronic and Information Engineering, Beijing Jiaotong University(电子与信息工程学院,北京交通大学);Department of Electrical and Electronic Engineering, The University of Hong Kong(电气与电子工程系,香港大学)
University of California, Santa Barbara(加州大学圣芭芭拉分校);University of British Columbia(不列颠哥伦比亚大学);University of Washington, Seattle(华盛顿大学(西雅图));EPFL(苏黎世联邦理工学院);Rutgers University, New Brunswick(罗格斯大学新 Brunswick分校);Microsoft(微软公司)
HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);School of Integrated Circuits, Peking University(北京大学集成电路学院);Institute of Electronic Design Automation, Peking University(北京大学电子设计自动化研究院);Beijing Advanced Innovation Center for Integrated Circuits(北京集成电路高精尖创新中心);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院)
Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
The Hong Kong University of Science and Technology(香港科技大学);Shanghai Jiao Tong University(上海交通大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Online Scheduling for LLM Inference with KV Cache Constraints
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology(麻省理工学院电子工程与计算机科学系);HKUST(香港科技大学);Microsoft Research(微软研究院);Sloan School of Management, Massachusetts Institute of Technology(斯隆管理学院,麻省理工学院);Operations Research Center, Massachusetts Institute of Technology(运营研究中心,麻省理工学院)
Adapter-Augmented Bandits for Online Multi-Constrained Multi-Modal Inference Scheduling
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, 510006, China.(中山大学计算机科学与工程学院);School of Computing, Macquarie University, NSW 2109, Australia(麦考瑞大学计算机学院)
Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints
Institute for Data, Systems, and Society, Massachusetts Institute of Technology(数据、系统与社会研究所,麻省理工学院);School of Mathematical Sciences, Peking University(北京大学数学科学学院);Alibaba Group(阿里巴巴集团)
Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents
School of Operations Research and Information Engineering, Cornell University(Cornell大学运筹学与信息工程学院);Operations Management, Booth School of Business, University of Chicago(芝加哥大学博斯商学院运营管理系);Decision, Risk and Operations, Columbia Business School, Columbia University(哥伦比亚大学哥伦比亚商学院决策、风险与运营系)
Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions
School of Computer Science, Wuhan University, Wuhan, China(武汉大学计算机学院);School of Computer Science, Central China Normal University, Wuhan, China(中央财经大学计算机学院);Dameng Database Co., Ltd., Wuhan, China(达梦数据库有限公司);School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China(上海交通大学人工智能学院);Institute for Math and AI, Wuhan University, Wuhan, China(武汉大学数学与人工智能研究院)