MoETuner：通过平衡专家放置与令牌路由优化专家混合模型服务
MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing
AI总结：
针对MoE模型分布式部署中令牌路由与专家激活不均衡、通信尾延迟等问题，提出MoETuner方案，通过ILP优化专家放置并利用层间令牌路由依赖，实现负载均衡与通信优化，单节点和多节点推理速度分别提升9.3%和17.5%。
摘要：
混合专家（Mixture-of-Experts，MoE）模型架构已成为高效扩展Transformer模型的有前景方案，其稀疏激活特性在降低计算成本的同时提升了模型容量。然而，随着MoE模型规模扩大，需跨GPU设备分布式部署，因内存占用大面临关键性能瓶颈。专家并行将专家分布于GPU，但存在令牌路由与专家激活不均衡的核心挑战，导致通信尾延迟与处理低效。现有方案虽解决部分问题，但未能化解负载不均衡与通信偏移的双重挑战：专家间令牌处理负载不均导致不同GPU处理时间差异，GPU间通信偏移引发跨GPU数据传输失衡，这些因素通过增加尾延迟与降低整体吞吐量劣化MoE模型性能。为此，我们提出整数线性规划（Integer Linear Programming，ILP）公式优化专家放置，联合考虑令牌负载、通信与计算成本；并利用层间令牌路由依赖特性——某层路由至特定专家的令牌，后续层可能仅路由至有限专家集。我们的方案MoETuner实现专家到GPU的最优分配，最小化跨GPU令牌路由成本并平衡设备间令牌处理，从而减少尾延迟与端到端执行时间。实验结果显示，单节点与多节点推理的端到端速度分别提升9.3%和17.5%，验证了基于ILP的优化在为下一代MoE提供专家并行解决方案方面的潜力。
英文摘要：
Mixture-of-Experts (MoE) model architecture has emerged as a promising solution for scaling transformer models efficiently, offering sparse activation that reduces computational costs while increasing model capacity. However, as MoE models scale, they need to be distributed across GPU devices, thus face critical performance bottlenecks due to their large memory footprint. Expert parallelism distributes experts across GPUs, however, faces key challenges including an unbalanced token routing and expert activation, resulting in communication tail latency and processing inefficiencies. While existing solutions address some of these issues, they fail to resolve the dual challenges of load imbalance and communication skew. The imbalance in token processing load across experts causes uneven processing times on different GPUs, while communication skew between GPUs leads to unbalanced inter-GPU data transfers. These factors degrade the performance of MoE models by increasing tail latency and reducing overall throughput. To address these limitations, we propose an Integer Linear Programming (ILP) formulation to optimize expert placement by jointly considering token load, communication, and computation costs. We exploit the property that there is a token routing dependency across layers, where tokens routed to a specific expert in one layer are likely to be routed to a limited set of experts in the subsequent layer. Our solution, MoETuner, offers an optimal expert-to-GPU assignment that minimizes inter-GPU token routing costs and balances token processing across devices, thereby reducing tail latency and end-to-end execution time. Experimental results demonstrate 9.3% and 17.5% of end-to-end speedups for single-node and multi-node inference respectively, showcasing the potential of our ILP-based optimization for offering expert parallel solutions for next-generation MoEs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
