大型语言模型(LLMs)在广泛的任务中取得了显著的成功,但大规模高效服务仍然是一个关键挑战,因为它们有巨大的计算和延迟需求。虽然现有的大多数推理框架依赖于简单的调度策略,如在引擎级别使用先进先出(FCFS)和在调度器或协调器级别使用循环轮询(RR),但它们往往无法充分利用系统资源,并可能面临头部阻塞和负载不平衡等问题。最近在混合专家(MoE)模型中的进展也引入了新的调度挑战,源于专家并行性和路由复杂性。本研究提出了一种针对基于MoE的LLM服务的多层调度框架。它针对三个层面进行调度:请求层面、引擎层面和专家层面。在请求层面,我们探索了最短作业优先(SJF)和优先级感知老化等算法,以提高吞吐量并减少延迟。在引擎层面,我们设计了负载感知的调度策略,考虑当前前缀标记负载、KV缓存利用率和用户粘性,以实现更好的资源匹配。在专家层面,我们专注于缓解专家热点问题,并战略性地放置跨层专家依赖关系,以平衡负载并提高路由效率。超过100次实验的结果表明,我们的方法在多样化的负载分布下始终优于最先进的推理框架vLLM,实现了高达17.8%的时延减少(Time To First Token, TTFT)和13.3%的时延减少(Time-Per-Output-Token, TPOT)。\n
英文摘要:
Large Language Models (LLMs) have achieved remarkable success across a wide range of tasks, but serving them efficiently at scale remains a critical challenge due to their substantial computational and latency demands. While most existing inference frameworks rely on simple scheduling strategies such as First-Come-First-Serve (FCFS) at the engine level and Round-Robin (RR) at the scheduler or coordinator level, they often fail to fully utilize system resources and may suffer from issues such as head-of-line blocking and load imbalance. Recent advances in Mixture-of-Experts (MoE) models have also introduced new challenges in scheduling arising from expert parallelism and routing complexity. This research proposes a multi-layer scheduling framework tailored for MoE-based LLM serving. It targets scheduling at three levels: request-level, enginelevel, and expert-level. At the request level, we explore algorithms such as Shortest-Job-First (SJF) and priority-aware aging to improve throughput and reduce latency. At the engine level, we design load-aware dispatching strategies that account for the current prefix token load, KV cache utilization, and user stickiness to achieve better resource matching. At the expert level, we focus on alleviating expert hotspots and strategically placing inter-layer expert dependencies to balance load and improve routing efficiency. Extensive experimental results from more than 100 experiments conducted under diverse workload distributions show that our approach consistently outperforms the state-of-theart inference framework vLLM, achieving up to 17.8% reduction in Time To First Token (TTFT) latency and 13.3% reduction in Time-Per-Output-Token (TPOT) latency.
HyperParallel-MoE: Multi-Core Interleaved Scheduling for Fast MoE Training on Ascend NPUs
University of Science and Technology of China(中国科学技术大学);Huawei Technologies Co., Ltd(华为技术有限公司);Peking University(北京大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training
Zhejiang University(浙江大学);Institute for Communication Systems (ICS), University of Surrey(萨里大学通信系统研究所);Xi’an Jiaotong-Liverpool University(西交利物浦大学);Northeastern University(东北大学);Khalifa University(哈利法大学)
Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Hong Kong Baptist University(香港 Baptist 大学);The Hong Kong University of Science(香港科学与技术大学)
ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling
School of Electronic Science and Engineering, Nanjing University, China.(南京大学电子科学与工程学院);National Key Laboratory for Novel Software Technology, Nanjing University, China.(南京大学新型软件技术国家重点实验室)
HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);School of Integrated Circuits, Peking University(北京大学集成电路学院);Institute of Electronic Design Automation, Peking University(北京大学电子设计自动化研究院);Beijing Advanced Innovation Center for Integrated Circuits(北京集成电路高精尖创新中心);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院)
MoE-CORE: Coordinated Expert Offloading and Residency for Memory-Constrained MoE Inference
University of Chinese Academy of Sciences(中国科学院大学);Huawei Technologies Co. Ltd(华为技术有限公司);Huawei Ascend Computing Technology Development(华为昇腾计算技术发展中心);Shanghai Jiao Tong University(上海交通大学)
Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUs
Hong Kong University of Science and Technology(香港科技大学);Shanghai Jiao Tong University(上海交通大学);Huawei Technologies Co. Ltd.(华为技术有限公司);Peking University(北京大学);Shanghai Qi Zhi Institute(上海人工智能实验室)
MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?
Department of Computing(计算系);Hong Kong Polytechnic University(香港理工大学);Mathematics and Computer Science Division(数学与计算机科学 division);Argonne National Laboratory(阿贡国家实验室);The University of Hong Kong(香港大学);Stevens Institute of Technology(史蒂文斯理工学院);Department of Computer Science and Engineering(计算机科学与工程系);University of California, Merced(加州大学默塞德分校)
RailWave: Adaptive Spatial and Temporal Scheduling for Expert-Parallel Communication
Sun Yat-sen University(中山大学);Southeast University(东南大学);Monash University(蒙纳士大学);National University of Singapore(新加坡国立大学);Shenzhen University of Advanced Technology(深圳先进技术研究院);Renmin University of China(中国人民大学)
Evaluating the Efficacy of LLM-Based Reasoning for Multiobjective HPC Job Scheduling
University of Tennessee, Knoxville\ Ridge National Laboratory Oak Ridge, TN USA;Argonne National Laboratory Lemont, IL USA;University of Southern California Los Angeles, CA USA;Oak Ridge National Laboratory Oak Ridge, TN USA;University of Tennessee, Knoxville\ Ridge National Laboratory;Argonne National Laboratory;University of Southern California;Oak Ridge National Laboratory
BrownoutMoE: Learning Structure-Aware Expert Grouping for Adaptive Mixture-of-Experts Model Serving
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Victoria University of Wellington(惠灵顿维多利亚大学);State Key Lab of IoTSC, University of Macau(澳门大学物联网与智能计算国家重点实验室)
TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading
SKLP, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China(SKLP,计算技术研究所,中国科学院,北京);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京)
RailS: Load Balancing for All-to-All Communication in Distributed Mixture-of-Experts Training
Faculty of Computer Science and Control Engineering, Shenzhen University of Advanced Technology(深圳先进技术学院计算机科学与控制工程系);Institute for Network Sciences and Cyberspace, Tsinghua University(清华大学网络科学与网络空间研究院);Department of Information Engineering, Chinese University of Hong Kong(香港中文大学信息工程系);China Mobile(中国移动)