arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.03243cs.LGcs.AIcs.DCcs.PF
收藏

先排序后服务:通过成对学习排序实现低延迟LLM服务

Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank

University of Illinois Chicago(伊利诺伊大学芝加哥分校);Argonne National Laboratory(阿贡国家实验室)

Yiheng Tao, Yihe Zhang, Matthew Dearing, Xin Wang, Yuping Fan, Michael E. Papka, Zhiling Lan

AI总结:

提出PARS调度器,通过成对排序预测任务响应长度,近似最短作业优先调度,减少队头阻塞,在vLLM上实现高达15.7倍延迟降低。

摘要:

高效调度大语言模型(LLM)推理任务对于实现低延迟和高吞吐量至关重要,这一挑战随着具有高度可变生成长度的推理型LLM的兴起而日益严峻。传统的先来先服务(FCFS)策略常常遭受队头阻塞(HOL)问题,即长时间运行的任务会延迟其后排队的短任务。在本文中,我们引入了PARS,一种提示感知的LLM任务调度器,它通过使用边际排序损失的成对排序来近似最短作业优先(SJF)调度,从而缓解HOL阻塞。PARS能够直接从提示中有效预测基于响应长度的任务排序,从而以最小开销优化调度决策。此外,它与最先进的LLM服务系统vLLM无缝集成,供研究社区使用。在多个LLM模型和实际推理用例(包括聊天、数学和代码生成)上的大量实验表明,与vLLM默认调度器相比,PARS将延迟显著降低高达15.7倍。跨模型评估表明,我们的设计具有良好的泛化能力,能够在不同LLM之间实现有效调度,而无需针对特定模型进行重新训练。

英文摘要:

Efficient scheduling of large language model (LLM) inference tasks is critical for achieving low latency and high throughput, a challenge that is becoming increasingly acute with the rise of reasoning-capable LLMs whose generation lengths are highly variable. Traditional strategies like First Come, First-Serve (FCFS) often suffer from Head-of-Line (HOL) blocking, where long-running tasks delay shorter ones queued behind them. In this paper, we introduce PARS, a prompt-aware LLM task scheduler that mitigates HOL blocking by approximating shortest-job-first (SJF) scheduling through pairwise ranking with a margin ranking loss. PARS effectively predicts response-length-based task ordering directly from prompts, thereby optimizing scheduling decisions with minimal overhead. In addition, it integrates seamlessly with vLLM, a state-of-the-art LLM serving system, for the research community. Extensive experiments across multiple LLM models and real-world inference use cases, including chat, math, and code generation, demonstrate that PARS significantly reduces latency by up to 15.7x compared to the vLLM default scheduler. Cross-model evaluations demonstrate that our design generalizes effectively, allowing effective scheduling across diverse LLMs without requiring model-specific retraining.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑