Latent Prototype Routing：在 Mixture-of-Experts 中实现近乎完美的负载均衡
Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts
AI总结：
针对MoE架构的负载不平衡问题，提出Latent Prototype Routing (LPR)路由框架，通过聚类视角泛化现有方法以促进均衡的专家利用，在不影响下游性能的前提下，将专家负载基尼系数降至0.035，实现近乎完美的负载均衡。
摘要：
Mixture-of-Experts (MoE) 架构已成为高效扩展大型语言模型 (LLMs) 的关键策略。然而，当前的 MoE 系统存在严重的负载不平衡问题，在训练和推理期间仅持续激活一小部分专家，导致模型容量和计算资源的严重未充分利用。在这项工作中，我们通过聚类视角重新审视专家路由，并提出了 Latent Prototype Routing (LPR)，这是一种新颖的路由框架，它泛化了现有方法，同时在不影响下游性能的情况下促进了均衡的专家利用。在多个开源 MoE 模型（包括 DeepSeek-V3、Qwen3-MoE 和 Mixtral）上进行的广泛实验表明，LPR 将专家负载的基尼系数平均从 0.70 降低至 0.035，将最小-最大专家负载比从 1e-6 提高到 0.70，实现了近乎完美的负载均衡。
英文摘要：
Mixture-of-Experts (MoE) architectures have emerged as a key strategy for scaling large language models (LLMs) efficiently. However, current MoE systems suffer from severe load imbalance, where only a small subset of experts is consistently activated during training and inference, leading to significant underutilization of model capacity and computational resources. In this work, we revisit expert routing through a clustering perspective and propose Latent Prototype Routing (LPR), a novel routing framework that generalizes existing approaches while promoting balanced expert utilization without compromising downstream performance. Extensive experiments across multiple open-source MoE models -- including DeepSeek-V3, Qwen3-MoE, and Mixtral -- demonstrate that LPR reduces the Gini coefficient of expert load from 0.70 to 0.035 on average, improves the min-max expert load ratio from 1e-6 to 0.70, achieving near-perfect load balancing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
