Mixture-of-Experts (MoE) models scale by activating only a small subset of experts per token. However, training such models remains challenging because top-$k$ routing is discrete and non-differentiable, requiring gradient estimators for expert selection whose design remains a central open problem. We introduce ProbMoE, a probabilistic routing framework that models expert selection as a distribution over cardinality-constrained expert subsets and formulates routing as probabilistic inference in this discrete subset space. We first propose ProbMoE Exact-$k$ routing, which samples $k$-expert subsets in the forward pass, and the backward pass uses gradients through each expert's exact marginal probability as a tractable surrogate for the true gradient. ProbMoE naturally generalizes to a dynamic-$k$ routing setting, where both training and inference constrain the routing cardinality to the same predefined range, allowing adaptive expert allocation per token. Across benchmarks and model backbones, ProbMoE Exact-$k$ achieves strong performance compared to competitive baselines, with improved expert utilization and routing diversity; ProbMoE Dynamic-$k$ achieves comparable performance with fewer activated experts.
SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
AGH University of Krakow, Poland（克拉科夫AGH大学）；Faculty of Mathematics（数学系）；Computer Science, Jagiellonian University, Poland（计算机科学系，杰哥利安大学，波兰）；Centre for Credible Artificial Intelligence, Warsaw University of Technology（可信人工智能中心，华沙技术大学）
Massachusetts Institute of Technology（麻省理工学院）；Google Research（谷歌研究院）；Google Brain（谷歌大脑）
Neural networks and tree ensembles are state-of-the-art learners, each with its unique statistical and computational advantages. We aim to combine these advantages by introducing a new layer for neural networks, composed of an ensemble of differentiable decision trees (a.k.a. soft trees). While differentiable trees demonstrate promising results in the literature, they are typically slow in training and inference as they do not support conditional computation. We mitigate this issue by introducing a new sparse activation function for sample routing, and implement true conditional computation by…
Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
Queen Mary University of London（伦敦玛丽女王大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；City University of London（伦敦城市大学）；National and Kapodistrian University of Athens（雅典国立卡波迪斯特里亚大学）；Archimedes AI, Athena RC（阿基米德人工智能公司，雅典娜研究中心）；The Cyprus Institute（塞浦路斯研究所）；Imperial College London（帝国理工学院）
