arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.15792cs.LG
收藏

基于 Learning to Rank 的高效 LLM 调度

Efficient LLM Scheduling by Learning to Rank

UCSD(加州大学圣迭戈分校);Tsinghua University(清华大学);Snowflake(雪花公司);UC Berkeley(加州大学伯克利分校)

Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, Hao Zhang

AI总结:

针对LLM推理中因输出长度未知导致的FCFS调度队头阻塞问题,提出基于learning to rank预测输出长度相对排名的新颖调度器,近似SJF调度,在聊天机器人等服务中显著降低延迟并提升吞吐量。

摘要:

在大型语言模型(LLM)推理中,LLM 请求的输出长度通常被认为是先验未知的。因此,大多数 LLM 服务系统采用简单的先来先服务(FCFS)调度策略,导致队头(HOL)阻塞,并降低吞吐量和服务质量。在本文中,我们重新审视了这一假设——我们表明,尽管预测每个请求的确切生成长度是不可行的,但使用 learning to rank 可以预测一批请求中输出长度的相对排名。排名信息为调度请求提供了有价值的指导。基于这一洞察,我们为 LLM 推理和服务开发了一种新颖的调度器,与现有方法相比,它可以更好地近似最短作业优先(SJF)调度。我们将此调度器与最先进的 LLM 服务系统集成,并展示了在几个重要应用中的显著性能提升:聊天机器人服务中的延迟降低 2.8 倍,合成数据生成中的吞吐量提高 6.5 倍。我们的代码可在 https://github.com/hao-ai-lab/vllm-ltr.git 获取

英文摘要:

In Large Language Model (LLM) inference, the output length of an LLM request is typically regarded as not known a priori. Consequently, most LLM serving systems employ a simple First-come-first-serve (FCFS) scheduling strategy, leading to Head-Of-Line (HOL) blocking and reduced throughput and service quality. In this paper, we reexamine this assumption -- we show that, although predicting the exact generation length of each request is infeasible, it is possible to predict the relative ranks of output lengths in a batch of requests, using learning to rank. The ranking information offers valuable guidance for scheduling requests. Building on this insight, we develop a novel scheduler for LLM inference and serving that can approximate the shortest-job-first (SJF) schedule better than existing approaches. We integrate this scheduler with the state-of-the-art LLM serving system and show significant performance improvement in several important applications: 2.8x lower latency in chatbot serving and 6.5x higher throughput in synthetic data generation. Our code is available at https://github.com/hao-ai-lab/vllm-ltr.git

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑