High-demand LLM inference services (e.g., ChatGPT and BARD) support a wide range of requests from short chat conversations to long document reading. To ensure that all client requests are processed fairly, most major LLM inference services have request rate limits, to ensure that no client can dominate the request queue. However, this rudimentary notion of fairness also results in under-utilization of the resources and poor client experience when there is spare capacity. While there is a rich literature on fair scheduling, serving LLMs presents new challenges due to their unpredictable request lengths and their unique batching characteristics on parallel accelerators. This paper introduces the definition of LLM serving fairness based on a cost function that accounts for the number of input and output tokens processed. To achieve fairness in serving, we propose a novel scheduling algorithm, the Virtual Token Counter (VTC), a fair scheduler based on the continuous batching mechanism. We prove a 2x tight upper bound on the service difference between two backlogged clients, adhering to the requirement of work-conserving. Through extensive experiments, we demonstrate the superior performance of VTC in ensuring fairness, especially in contrast to other baseline methods, which exhibit shortcomings under various conditions. The reproducible code is available at https://github.com/Ying1123/VTC-artifact
Reasoning Language Model Inference Serving Unveiled: An Empirical Study
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);HKBU(香港 Baptist University);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));The Hong Kong University of Science and Technology(香港科学与技术大学)
Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
School of Computing and Information Systems, The University of Melbourne(墨尔本大学计算与信息系统学院);School of Computer Science and Technology, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院)
FairCoT: Enhancing Fairness in Text-to-Image Generation via Chain of Thought Reasoning with Multimodal Large Language Models
School of Electronic Engineering and Computer Science, Queen Mary University of London(伦敦女王学院电子工程与计算机科学学院);Department of Knowledge Technologies, Jožef Stefan Institute(Jožef Stefan研究所知识技术系)
SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget
Shanghai Jiao Tong University(上海交通大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);National University of Singapore(新加坡国立大学);AsiaInfo Technologies (China), Inc.(亚信科技(中国)有限公司)
A Normative Framework for Benchmarking Consumer Fairness in Large Language Model Recommender System
Polytechnic University of Bari(巴里理工大学);Peoples' Friendship University of Russia (RUDN University)(俄罗斯人民友谊大学(RUDN大学));Joint Institute for Nuclear Research(联合核子研究所);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);University of Skövde(舍夫德大学)
MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);The International Joint Institute of Tianjin University, Tianjin University(天津大学国际联合学院);Faculty of Digital Economics and Managements, Tianjin University of Finance and Economics(天津财经大学数字经济与管理学院)
Your Large Language Model is Secretly a Fairness Proponent and You Should Prompt it Like One
Nanyang Technological University(南洋理工大学);Sea AI Lab(Sea AI实验室);Xi’an Jiaotong University(西安交通大学);Centre for Frontier AI Research (CFAR), A*STAR(新加坡科技研究局前沿人工智能研究中心)
An Analysis of the Effects of Decoding Algorithms on Fairness in Open-Ended Language Generation
Amazon Alexa AI(亚马逊Alexa人工智能部门);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Information Sciences Institute, University of Southern California(南加利福尼亚大学信息科学研究所)