In Large Language Model (LLM) inference, the output length of an LLM request is typically regarded as not known a priori. Consequently, most LLM serving systems employ a simple First-come-first-serve (FCFS) scheduling strategy, leading to Head-Of-Line (HOL) blocking and reduced throughput and service quality. In this paper, we reexamine this assumption -- we show that, although predicting the exact generation length of each request is infeasible, it is possible to predict the relative ranks of output lengths in a batch of requests, using learning to rank. The ranking information offers valuable guidance for scheduling requests. Building on this insight, we develop a novel scheduler for LLM inference and serving that can approximate the shortest-job-first (SJF) schedule better than existing approaches. We integrate this scheduler with the state-of-the-art LLM serving system and show significant performance improvement in several important applications: 2.8x lower latency in chatbot serving and 6.5x higher throughput in synthetic data generation. Our code is available at https://github.com/hao-ai-lab/vllm-ltr.git
Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
The Hong Kong University of Science and Technology(香港科技大学);Shanghai Jiao Tong University(上海交通大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
University of California, Santa Barbara(加州大学圣芭芭拉分校);University of British Columbia(不列颠哥伦比亚大学);University of Washington, Seattle(华盛顿大学(西雅图));EPFL(苏黎世联邦理工学院);Rutgers University, New Brunswick(罗格斯大学新 Brunswick分校);Microsoft(微软公司)
Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions
School of Computer Science, Wuhan University, Wuhan, China(武汉大学计算机学院);School of Computer Science, Central China Normal University, Wuhan, China(中央财经大学计算机学院);Dameng Database Co., Ltd., Wuhan, China(达梦数据库有限公司);School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China(上海交通大学人工智能学院);Institute for Math and AI, Wuhan University, Wuhan, China(武汉大学数学与人工智能研究院)
Beyond Prediction: Tail-Aware Scheduling for LLM Inference
Cornell University, Computer Science Department(康奈尔大学计算机科学系);Cornell University, Electrical and Computer Engineering Department(康奈尔大学电气与计算机工程系);Cornell University, Operations Research and Information Engineering Department(康奈尔大学运筹学与信息工程系);Microsoft Azure System Research(微软Azure系统研究);NVIDIA Corporation(英伟达公司)
Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latency
School of Cyber Science and Engineering, Xi’an Jiaotong University(西安交通大学计算机科学与工程学院);School of Computer Science and Engineering, The University of Aizu(台湾花莲大学计算机科学与工程学院)
Online Scheduling for LLM Inference with KV Cache Constraints
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology(麻省理工学院电子工程与计算机科学系);HKUST(香港科技大学);Microsoft Research(微软研究院);Sloan School of Management, Massachusetts Institute of Technology(斯隆管理学院,麻省理工学院);Operations Research Center, Massachusetts Institute of Technology(运营研究中心,麻省理工学院)
Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents
School of Operations Research and Information Engineering, Cornell University(Cornell大学运筹学与信息工程学院);Operations Management, Booth School of Business, University of Chicago(芝加哥大学博斯商学院运营管理系);Decision, Risk and Operations, Columbia Business School, Columbia University(哥伦比亚大学哥伦比亚商学院决策、风险与运营系)
LLM-Powered Predictive Decision-Making for Sustainable Data Center Operations
The University of Sydney Business School, The University of Sydney(悉尼大学商学院,悉尼大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Purdue University(普渡大学)
LLM-Enabled In-Context Learning for Data Collection Scheduling in UAV-assisted Sensor Networks
Real-Time and Embedded Computing Systems Research Centre (CISTER)(实时与嵌入式计算系统研究中心(CISTER));Faculdade de Engenharia da Universidade do Porto(葡萄牙波尔图大学工程学院);School of Computer Science, McGill University(麦吉尔大学计算机科学学院);Knight Foundation School of Computing and Information Sciences, Florida International University(国际大学计算与信息科学学院(Knight基金会))
Decentralized Rank Scheduling for Energy-Constrained Multi-Task Federated Fine-Tuning in Edge-Assisted IoV Networks
School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);School of Cyber Science and Technology, Sun Yat-sen University(中山大学网络安全科学与技术学院)
Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoC
School of Computer Science, Peking University(北京大学计算机科学学院);The University of Hong Kong(香港大学);National Key Laboratory for Multimedia Information Processing, Peking University(北京大学多媒体信息处理国家重点实验室)
Discovering Heuristics with Large Language Models (LLMs) for Mixed-Integer Programs: Single-Machine Scheduling
Grado Department of Industrial and Systems Engineering, Virginia Tech, Blacksburg, VA(弗吉尼亚理工大学工业与系统工程系);Department of Computer Science, Virginia Tech, Arlington, VA(弗吉尼亚理工大学计算机科学系)
Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints
Institute for Data, Systems, and Society, Massachusetts Institute of Technology(数据、系统与社会研究所,麻省理工学院);School of Mathematical Sciences, Peking University(北京大学数学科学学院);Alibaba Group(阿里巴巴集团)
Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels
Huawei(华为);Tsinghua University(清华大学);Shanghai AI Lab(上海人工智能实验室)