arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.09277cs.DCcs.AR
收藏

在内存受限环境下高效实现MoE服务:平衡激活专家而非token

Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens

Yale University(耶鲁大学);Princeton University(普林斯顿大学);Carnegie Mellon University(卡内基梅隆大学);NVIDIA(英伟达)
Yanpeng Yu, Haiyue Ma, Krish Agarwal, Nicolai Oswald, Qijing Huang, Hugo Linsenmaier, Chunhui Mei, Ritchie Zhao, Ritika Borkar, Bita Darvish Rouhani, David Nell… 展开作者

Yanpeng Yu, Haiyue Ma, Krish Agarwal, Nicolai Oswald, Qijing Huang, Hugo Linsenmaier, Chunhui Mei, Ritchie Zhao, Ritika Borkar, Bita Darvish Rouhani, David Nellans, Ronny Krashinsky, Anurag Khandelwal

AI总结:

METRO通过平衡每个GPU激活的专家数量而非token数量,在内存受限环境下实现高效MoE服务,减少解码延迟并提升吞吐量。

摘要:

专家并行性(EP)使混合专家(MoE)模型能够超越单个GPU。为了解决EP中GPU之间的负载不均衡问题,现有方法旨在平衡每个GPU处理的token数量。令人惊讶的是,当我们处理时受限于内存——在MoE服务中常见,尤其是在解码阶段——这种目标反而会降低性能而非提高。我们的分析表明,平衡每个GPU处理的token数量会增加激活的专家数量,加剧内存压力。我们提出最小专家token路由(METRO),这是一种新颖的token路由算法,用于在内存受限环境下实现高性能的专家并行MoE服务,通过平衡每个GPU激活的专家数量而非token数量。METRO通过联合优化算法效率并利用GPU的并行处理能力,以最小的计算开销实现接近最优的路由质量。为了保证路由质量,METRO还采用了一种新的allGather方案来收集全局top-k知识,其开销比传统allToAll要小。我们在真实系统(vLLM over 8 A100 GPUs)和专有模拟器(8-16 B200 GPUs)上评估METRO与EPLB的性能,结果显示,METRO在Qwen3和DeepSeek-V3服务中,解码延迟减少了11-22%,总token吞吐量减少了3-21%。此外,通过用延迟头寸换取吞吐量,METRO在固定解码SLO下,将解码吞吐量提高了4.11倍以上EPLB。

英文摘要:

Expert Parallelism (EP) permits Mixture of Experts (MoE) models to scale beyond a single GPU. To address load imbalance across GPUs in EP, existing approaches aim to balance the number of tokens each GPU processes. Surprisingly, we find that this objective degrades performance rather than improving it when processing is memory-bound - a common occurrence in MoE serving, especially in the decode phase. Our analysis reveals that balancing the number of tokens processed per GPU increases the number of activated experts, exacerbating memory pressure in the memory-bound regime. We propose Minimum Expert Token ROuting, a novel token-routing algorithm for high-performance expert-parallel MoE serving in the memory-bound regime that balances the number of activated experts per GPU rather than token counts. METRO achieves near-optimal routing quality with minimal computational overhead by jointly optimizing algorithmic efficiency and leveraging the GPU's parallel processing power. To guarantee routing quality, METRO also employs a novel allGather scheme to gather global top-k knowledge, which has minimal overhead compared to conventional allToAll. Our evaluation of METRO against EPLB on both real systems (vLLM over 8 A100 GPUs) and a proprietary simulator (8-16 B200 GPUs) shows that METRO reduces decode latency by 11 - 22%, and total token throughput by 3 - 21% for Qwen3 and DeepSeek-V3 serving, where prefill and decode phases are co-deployed. In addition, by trading latency headroom for throughput, METRO improves decode throughput by up to 4.11x over EPLB at a fixed decode SLO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑