使用保持相似性的路由器实现专家混合的负载均衡
Load Balancing Mixture of Experts with Similarity Preserving Routers
AI总结：
针对稀疏MoE模型路由器因现有负载均衡机制导致不一致路由和冗余知识的问题，提出一种保留token关系结构的新颖负载均衡损失，促使相似输入选择一致专家，实现36%更快收敛和更低冗余。
摘要：
稀疏专家混合（MoE）模型通过为每个输入仅激活一部分参数（“专家”），为训练大型神经网络提供了一种可扩展且高效的架构。一个学习到的路由器计算这些专家上的分布，并将输入 token 分配给一小部分专家。然而，如果没有辅助的均衡机制，路由器通常会收敛到仅使用少数专家，严重限制了模型容量并降低了性能。当前大多数负载均衡机制鼓励专家上的分布类似于每个 token 大致均匀的专家分布。在训练期间，这可能导致不一致的路由行为，导致模型将其容量用于学习冗余知识。我们通过引入一种新颖的负载均衡损失来解决这一问题，该损失保留了 token 级别的关系结构，在训练期间鼓励对相似输入做出一致的专家选择。我们的实验结果表明，与一种流行的负载均衡损失相比，将我们的损失应用于路由器可实现 36% 更快的收敛速度和更低的冗余度。
英文摘要：
Sparse Mixture of Experts (MoE) models offer a scalable and efficient architecture for training large neural networks by activating only a subset of parameters ("experts") for each input. A learned router computes a distribution over these experts, and assigns input tokens to a small subset. However, without auxiliary balancing mechanisms, routers often converge to using only a few experts, severely limiting model capacity and degrading performance. Most current load balancing mechanisms encourage a distribution over experts that resembles a roughly uniform distribution of experts per token. During training, this can result in inconsistent routing behavior, resulting in the model spending its capacity to learn redundant knowledge. We address this by introducing a novel load balancing loss that preserves token-wise relational structure, encouraging consistent expert choices for similar inputs during training. Our experimental results show that applying our loss to the router results in 36% faster convergence and lower redundancy compared to a popular load balancing loss.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
