DirMoE：基于Dirichlet的混合专家模型
DirMoE: Dirichlet-routed Mixture of Experts
AI总结：
DirMoE通过基于Dirichlet变分自动编码器的端到端可微路由机制，解耦专家选择与贡献分配，提升混合专家模型的性能和可扩展性。
摘要：
混合专家（MoE）模型在大规模语言模型中展现了卓越的性能。现有的路由机制通常依赖于非可微的Top-k+Softmax，限制了其性能和可扩展性。我们认为，在标准Top-k+Softmax中，专家选择和专家贡献分配两个不同的决策被混淆了。我们引入了Dirichlet-Routed MoE（DirMoE），一种基于Dirichlet变分自动编码器框架的新型端到端可微路由机制。该设计从根本上解耦了核心路由问题：专家选择，由伯努利组件建模，以及所选专家之间的贡献分配，由Dirichlet组件处理。通过使用Gumbel-Sigmoid放松专家选择和隐式重参数化Dirichlet分布，整个前向传递保持完全可微。我们的训练目标，一个变分ELBO，包括一个直接的稀疏性惩罚，精确控制期望中的活跃专家数量，以及一个关键超参数的调度，引导模型从探索状态到确定性路由状态。此外，我们的DirMoE路由器在匹配或超过其他方法的同时，提高了专家的专业化程度。
英文摘要：
Mixture-of-Experts (MoE) models have demonstrated exceptional performance in large-scale language models. Existing routers typically rely on non-differentiable Top-$k$+Softmax, limiting their performance and scalability. We argue that two distinct decisions, which experts to activate and how to distribute expert contributions among them, are conflated in standard Top-$k$+Softmax. We introduce Dirichlet-Routed MoE (DirMoE), a novel end-to-end differentiable routing mechanism built on a Dirichlet variational autoencoder framework. This design fundamentally disentangles the core routing problems: expert selection, modeled by a Bernoulli component, and expert contribution among chosen experts, handled by a Dirichlet component. The entire forward pass remains fully differentiable through the use of Gumbel-Sigmoid relaxation for the expert selection and implicit reparameterization for the Dirichlet distribution. Our training objective, a variational ELBO, includes a direct sparsity penalty that precisely controls the number of active experts in expectation, alongside a schedule for key hyperparameters that guides the model from an exploratory to a definitive routing state. Moreover, our DirMoE router matches or exceeds other methods while improving expert specialization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
