arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.21626cs.DC
收藏

多层调度用于基于MoE的LLM推理

Multi-Layer Scheduling for MoE-Based LLM Reasoning

The University of Melbourne(墨尔本大学);Monash University(莫纳什大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)

Yifan Sun, Gholamreza Haffari, Minxian Xu, Rajkumar Buyya, Adel N. Toosi

AI总结:

本研究提出多层调度框架,针对基于MoE的LLM推理优化,通过请求、引擎和专家层面调度策略,提升吞吐量和效率,实验表明在TTFT和TPOT延迟上分别减少17.8%和13.3%。

摘要:

大型语言模型(LLMs)在广泛的任务中取得了显著的成功,但大规模高效服务仍然是一个关键挑战,因为它们有巨大的计算和延迟需求。虽然现有的大多数推理框架依赖于简单的调度策略,如在引擎级别使用先进先出(FCFS)和在调度器或协调器级别使用循环轮询(RR),但它们往往无法充分利用系统资源,并可能面临头部阻塞和负载不平衡等问题。最近在混合专家(MoE)模型中的进展也引入了新的调度挑战,源于专家并行性和路由复杂性。本研究提出了一种针对基于MoE的LLM服务的多层调度框架。它针对三个层面进行调度:请求层面、引擎层面和专家层面。在请求层面,我们探索了最短作业优先(SJF)和优先级感知老化等算法,以提高吞吐量并减少延迟。在引擎层面,我们设计了负载感知的调度策略,考虑当前前缀标记负载、KV缓存利用率和用户粘性,以实现更好的资源匹配。在专家层面,我们专注于缓解专家热点问题,并战略性地放置跨层专家依赖关系,以平衡负载并提高路由效率。超过100次实验的结果表明,我们的方法在多样化的负载分布下始终优于最先进的推理框架vLLM,实现了高达17.8%的时延减少(Time To First Token, TTFT)和13.3%的时延减少(Time-Per-Output-Token, TPOT)。\n

英文摘要:

Large Language Models (LLMs) have achieved remarkable success across a wide range of tasks, but serving them efficiently at scale remains a critical challenge due to their substantial computational and latency demands. While most existing inference frameworks rely on simple scheduling strategies such as First-Come-First-Serve (FCFS) at the engine level and Round-Robin (RR) at the scheduler or coordinator level, they often fail to fully utilize system resources and may suffer from issues such as head-of-line blocking and load imbalance. Recent advances in Mixture-of-Experts (MoE) models have also introduced new challenges in scheduling arising from expert parallelism and routing complexity. This research proposes a multi-layer scheduling framework tailored for MoE-based LLM serving. It targets scheduling at three levels: request-level, enginelevel, and expert-level. At the request level, we explore algorithms such as Shortest-Job-First (SJF) and priority-aware aging to improve throughput and reduce latency. At the engine level, we design load-aware dispatching strategies that account for the current prefix token load, KV cache utilization, and user stickiness to achieve better resource matching. At the expert level, we focus on alleviating expert hotspots and strategically placing inter-layer expert dependencies to balance load and improve routing efficiency. Extensive experimental results from more than 100 experiments conducted under diverse workload distributions show that our approach consistently outperforms the state-of-theart inference framework vLLM, achieving up to 17.8% reduction in Time To First Token (TTFT) latency and 13.3% reduction in Time-Per-Output-Token (TPOT) latency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑