专家并行性(EP)使混合专家(MoE)模型能够超越单个GPU。为了解决EP中GPU之间的负载不均衡问题,现有方法旨在平衡每个GPU处理的token数量。令人惊讶的是,当我们处理时受限于内存——在MoE服务中常见,尤其是在解码阶段——这种目标反而会降低性能而非提高。我们的分析表明,平衡每个GPU处理的token数量会增加激活的专家数量,加剧内存压力。我们提出最小专家token路由(METRO),这是一种新颖的token路由算法,用于在内存受限环境下实现高性能的专家并行MoE服务,通过平衡每个GPU激活的专家数量而非token数量。METRO通过联合优化算法效率并利用GPU的并行处理能力,以最小的计算开销实现接近最优的路由质量。为了保证路由质量,METRO还采用了一种新的allGather方案来收集全局top-k知识,其开销比传统allToAll要小。我们在真实系统(vLLM over 8 A100 GPUs)和专有模拟器(8-16 B200 GPUs)上评估METRO与EPLB的性能,结果显示,METRO在Qwen3和DeepSeek-V3服务中,解码延迟减少了11-22%,总token吞吐量减少了3-21%。此外,通过用延迟头寸换取吞吐量,METRO在固定解码SLO下,将解码吞吐量提高了4.11倍以上EPLB。
英文摘要:
Expert Parallelism (EP) permits Mixture of Experts (MoE) models to scale beyond a single GPU. To address load imbalance across GPUs in EP, existing approaches aim to balance the number of tokens each GPU processes. Surprisingly, we find that this objective degrades performance rather than improving it when processing is memory-bound - a common occurrence in MoE serving, especially in the decode phase. Our analysis reveals that balancing the number of tokens processed per GPU increases the number of activated experts, exacerbating memory pressure in the memory-bound regime.
We propose Minimum Expert Token ROuting, a novel token-routing algorithm for high-performance expert-parallel MoE serving in the memory-bound regime that balances the number of activated experts per GPU rather than token counts. METRO achieves near-optimal routing quality with minimal computational overhead by jointly optimizing algorithmic efficiency and leveraging the GPU's parallel processing power. To guarantee routing quality, METRO also employs a novel allGather scheme to gather global top-k knowledge, which has minimal overhead compared to conventional allToAll. Our evaluation of METRO against EPLB on both real systems (vLLM over 8 A100 GPUs) and a proprietary simulator (8-16 B200 GPUs) shows that METRO reduces decode latency by 11 - 22%, and total token throughput by 3 - 21% for Qwen3 and DeepSeek-V3 serving, where prefill and decode phases are co-deployed. In addition, by trading latency headroom for throughput, METRO improves decode throughput by up to 4.11x over EPLB at a fixed decode SLO.
Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving
National University of Singapore(新加坡国立大学);Shanghai Qiji Zhifeng Co., Ltd.(上海奇积智峰有限公司);Nanyang Technology University(南洋理工大学);Tsinghua University(清华大学);Shanghai Innovation Institute(上海创新研究院)
MixServe: An Automatic Distributed Serving System for MoE Models with Hybrid Parallelism Based on Fused Communication Algorithm
School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程系)
ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling
School of Electronic Science and Engineering, Nanjing University, China.(南京大学电子科学与工程学院);National Key Laboratory for Novel Software Technology, Nanjing University, China.(南京大学新型软件技术国家重点实验室)
BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMs
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Southern University of Science and Technology(南方科技大学);State Key Lab of IOTSC, University of Macau(澳门大学物联网智能计算国家重点实验室)
ProMoE: Fast MoE-based LLM Serving using Proactive Caching
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University(上海交通大学平行与分布式系统研究所);Key Laboratory of System Software (Chinese Academy of Sciences)(中国科学院系统软件重点实验室);Zhejiang University(浙江大学)
Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules
Data Science and Analytics Thrust, The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)数据科学与分析学域);Hong Kong Baptist University(香港浸会大学);School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)计算机科学与技术学院);Department of Computer Science and Engineering, The Hong Kong University of Science and Technology(香港科技大学计算机科学与工程学系)
Janus: Disaggregating Attention and Experts for Scalable MoE Inference
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Institute of Artificial Intelligence (TeleAI), China Telecom(中国电信人工智能研究院(TeleAI));Shenzhen Loop Area Institute Nokia Bell Labs Harbin Institute of Technology, Shenzhen(深圳_loop_area_研究所 诺基亚贝尔实验室 哈尔滨工业大学(深圳))
MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?
Department of Computing(计算系);Hong Kong Polytechnic University(香港理工大学);Mathematics and Computer Science Division(数学与计算机科学 division);Argonne National Laboratory(阿贡国家实验室);The University of Hong Kong(香港大学);Stevens Institute of Technology(史蒂文斯理工学院);Department of Computer Science and Engineering(计算机科学与工程系);University of California, Merced(加州大学默塞德分校)
HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap
School of Computer Science and Technology(计算机科学与技术学院);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Data Science and Analytics Thrust, The Hong Kong University of Science and Technology (Guangzhou)(数据科学与分析方向,香港科技大学(广州));Department of Computer Science and Engineering, The Hong Kong University of Science and Technology(计算机科学与工程系,香港科技大学)