面向混合专家模型的最大分数路由
Maximum Score Routing For Mixture-of-Experts
AI总结：
本文提出MaxScore路由范式，将MoE路由建模为最小成本最大流问题并结合SoftTopk算子，在相同FLOPs下实现更低训练损失和更高评估分数。
摘要：
在稀疏激活的混合专家（MoE）模型中，路由网络通过可微稀疏变换将输入token动态分配给top-k专家，从而在保持计算效率的同时实现可扩展的模型容量。传统MoE网络施加专家容量约束以确保GPU友好的计算，但这会导致容量饱和时token被丢弃，并因未充分利用专家中的填充而导致硬件效率低下。移除容量约束又会损害负载均衡和计算效率。为解决这些问题，我们提出了最大分数路由（MaxScore），这是一种新颖的MoE路由范式，将路由建模为最小成本最大流问题，并集成了SoftTopk算子。MaxScore解决了迭代重路由和最优传输公式的根本局限，在相同FLOPs下，与有约束和无约束基线相比，实现了更低的训练损失和更高的评估分数。实现细节和实验配置可从https://github.com/dongbw18/MaxScore.git获取。
英文摘要：
Routing networks in sparsely activated mixture-of-experts (MoE) dynamically allocate input tokens to top-k experts through differentiable sparse transformations, enabling scalable model capacity while preserving computational efficiency. Traditional MoE networks impose an expert capacity constraint to ensure GPU-friendly computation. However, this leads to token dropping when capacity is saturated and results in low hardware efficiency due to padding in underutilized experts. Removing the capacity constraint, in turn, compromises load balancing and computational efficiency. To address these issues, we propose Maximum Score Routing ($\mathbf{MaxScore}$), a novel MoE routing paradigm that models routing as a minimum-cost maximum-flow problem and integrates a SoftTopk operator. MaxScore resolves the fundamental limitations of iterative rerouting and optimal transport formulations, achieving lower training losses and higher evaluation scores at equivalent FLOPs compared to both constrained and unconstrained baselines. Implementation details and experimental configurations can be obtained from $\href{https://github.com/dongbw18/MaxScore.git}{MaxScore}$.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
