arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.00588cs.AIcs.LGcs.PF
收藏

大型语言模型服务中的公平性

Fairness in Serving Large Language Models

UC Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学);Duke University(杜克大学)

Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, Ion Stoica

AI总结:

本文针对LLM服务因请求长度不可预测和并行加速器批处理特性带来的公平性挑战,基于处理token数量的成本函数定义了公平性,并提出基于连续批处理的新颖调度算法VTC,证明了2倍紧致上界且实验验证其性能卓越。

摘要:

高需求的LLM推理服务(例如ChatGPT和BARD)支持从简短的聊天对话到长文档阅读的广泛请求。为了确保所有客户端请求得到公平处理,大多数主要的LLM推理服务都设有请求速率限制,以确保没有客户端能垄断请求队列。然而,这种初级的公平性概念也会导致资源利用不足,并在有剩余容量时导致客户端体验不佳。尽管关于公平调度已有丰富的文献,但由于请求长度不可预测以及在并行加速器上具有独特的批处理特性,服务LLM带来了新的挑战。本文引入了基于成本函数的LLM服务公平性定义,该函数考虑了已处理的输入和输出token数量。为了在服务中实现公平,我们提出了一种新颖的调度算法,即Virtual Token Counter(VTC),这是一种基于连续批处理机制的公平调度器。我们证明了两个积压客户端之间服务差异的2倍紧致上界,符合工作保持的要求。通过大量实验,我们证明了VTC在确保公平性方面的卓越性能,特别是与其他基线方法相比,后者在各种条件下表现出不足。可复现代码可在https://github.com/Ying1123/VTC-artifact获取

英文摘要:

High-demand LLM inference services (e.g., ChatGPT and BARD) support a wide range of requests from short chat conversations to long document reading. To ensure that all client requests are processed fairly, most major LLM inference services have request rate limits, to ensure that no client can dominate the request queue. However, this rudimentary notion of fairness also results in under-utilization of the resources and poor client experience when there is spare capacity. While there is a rich literature on fair scheduling, serving LLMs presents new challenges due to their unpredictable request lengths and their unique batching characteristics on parallel accelerators. This paper introduces the definition of LLM serving fairness based on a cost function that accounts for the number of input and output tokens processed. To achieve fairness in serving, we propose a novel scheduling algorithm, the Virtual Token Counter (VTC), a fair scheduler based on the continuous batching mechanism. We prove a 2x tight upper bound on the service difference between two backlogged clients, adhering to the requirement of work-conserving. Through extensive experiments, we demonstrate the superior performance of VTC in ensuring fairness, especially in contrast to other baseline methods, which exhibit shortcomings under various conditions. The reproducible code is available at https://github.com/Ying1123/VTC-artifact

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑