$ϕ$-平衡用于专家混合训练
$ϕ$-Balancing for Mixture-of-Experts Training
AI总结：
本文提出$ϕ$-平衡框架，通过最小化严格凸、对称且可导的潜在函数，实现专家资源的群体平衡，优于现有启发式方法，在大规模预训练和下游微调中表现更稳定有效。
摘要：
混合专家（MoE）模型依赖于平衡的专家利用以充分发挥其可扩展性。然而，现有负载平衡方法大多是启发式的，并基于嘈杂的小批量分配统计，引入了相对于群体目标的偏差。我们提出$ϕ$-平衡，一个原则性的框架，通过最小化预期路由分布的严格凸、对称且可导的潜在函数，直接针对群体层面的专家平衡。利用凸对偶性，我们推导出等价的min-max形式，并通过镜像下降法获得一个简单的在线算法，得到一个高效的EMA基于路由调整，具有可忽略的开销。在大规模预训练和下游微调中，$ϕ$-平衡一致优于先前的Switch式和无损失基线，展示了更稳定和有效的专家利用。
英文摘要：
Mixture-of-Experts (MoE) models rely on balanced expert utilization to fully realize their scalability. However, existing load-balancing methods are largely heuristic and operate on noisy mini-batch assignment statistics, introducing bias relative to population-level objectives. We propose $ϕ$-balancing, a principled framework that directly targets population-level expert balance by minimizing a strictly convex, symmetric, and differentiable potential of the expected routing distribution. Using convex duality, we derive an equivalent min-max formulation and obtain a simple online algorithm via mirror descent, yielding an efficient EMA-based routing adjustment with negligible overhead. Across large-scale pretraining and downstream fine-tuning, $ϕ$-balancing consistently outperforms prior Switch-style and loss-free baselines, demonstrating more stable and effective expert utilization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
