无路由专家混合模型
Routing-Free Mixture-of-Experts
AI总结：
本文提出无路由专家混合模型，通过消除传统集中路由机制，实现专家自主激活，改进资源分配方法，实验表明其在可扩展性和鲁棒性上优于基线模型。
摘要：
标准的专家混合（MoE）模型依赖于集中路由机制，引入了刚性的归纳偏置。我们提出了无路由MoE，消除了所有硬编码的集中设计，包括外部路由器、Softmax、Top-K和负载平衡，将所有激活功能封装在个体专家中，并通过连续梯度流直接优化，使每个专家能够自主决定其激活。我们引入了一个统一的自适应负载平衡框架，通过可配置的插值同时优化专家平衡和令牌平衡目标，允许灵活且可定制的资源分配。大量实验表明，无路由MoE能够一致优于基线模型，在可扩展性和鲁棒性方面表现更佳。我们详细分析了其行为，并提供了可能促进未来MoE设计和优化的见解。
英文摘要：
Standard Mixture-of-Experts (MoE) models rely on centralized routing mechanisms that introduce rigid inductive biases. We propose Routing-Free MoE which eliminates any hard-coded centralized designs including external routers, Softmax, Top-K and load balancing, instead encapsulating all activation functionalities within individual experts and directly optimized through continuous gradient flow, enabling each expert to determine its activation entirely on its own. We introduce a unified adaptive load-balancing framework to simultaneously optimize both expert-balancing and token-balancing objectives through a configurable interpolation, allowing flexible and customizable resource allocation. Extensive experiments show that Routing-Free MoE can consistently outperform baselines with better scalability and robustness. We analyze its behavior in detail and offer insights that may facilitate future MoE design ad optimization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
