UBEP:为生产超级集群重新架构专家并行通信库
UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods
AI总结:
研究针对生产超级集群部署MoE模型面临的挑战,提出UBEP通信库,重新思考MoE全对全原语,经大规模实验,有效降低了全对全延迟和MoE推理每输出令牌时间。
摘要:
在生产型高带宽超级集群(如英伟达的NVL72/576和华为的CloudMatrix384)上部署专家混合(MoE)模型,带来了超出原始互连带宽的关键挑战。这些系统虽提供统一全局地址空间和高带宽架构,但稀疏MoE通信的潜力受三个瓶颈阻碍:粗粒度批量同步并行(BSP)编排导致的严格执行序列化、同步开销过高以及不规则令牌流量的距离无关调度导致的严重负载不平衡。为消除这些瓶颈,我们引入了UBEP(统一总线专家并行),一个为现代超级集群架构重新思考MoE全对全原语的生产就绪通信库。通过大规模实验,UBEP将全对全延迟降低了52.4%,将MoE推理每输出令牌时间(TPOT)降低了11.1%。
英文摘要:
The deployment of Mixture-of-Experts (MoE) models on production high-bandwidth superpods, such as NVIDIA's NVL72/576 and Huawei's CloudMatrix384, introduces critical challenges beyond raw interconnect bandwidth. While these systems provide unified global address spaces and high-bandwidth fabrics, their full potential for sparse MoE communication is hindered by three fundamental bottlenecks: (1) Strict execution serialization imposed by coarse-grained Bulk Synchronous Parallel (BSP) orchestration of interdependent communication phases; (2) Prohibitive synchronization overhead that fails to scale alongside high interconnect bandwidth; and (3) Severe load imbalance resulting from distance-agnostic scheduling of irregular token traffic. To eliminate these bottlenecks, we introduce UBEP (Unified-Bus Expert Parallelism), a production-ready communication library that rethinks MoE's All-to-All primitives for modern superpod architectures. Through large scale experiments, UBEP reduces All-to-All latency by up to 52.4% and MoE inference Time Per Output Token (TPOT) by up to 11.1%.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。