SoftMoE: 用于大语言模型混合专家网络的软可微路由
SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
AI总结：
提出SoftMoE，通过软top-k LapSum松弛替代离散路由，实现专家路由的梯度优化，并学习每层专家激活数量，在语言建模中激活更少专家达到相当或更优性能。
摘要：
稀疏混合专家（MoE）架构通过仅激活一小部分专家（通过top-$k$路由）在固定推理预算下扩展LLM参数。虽然这保持了因果性并适用于自回归语言模型，但离散的top-$k$算子不可微，强制每个输入激活固定数量的专家，导致计算利用效率低下。我们提出SoftMoE，用截断的软top-$k$ LapSum松弛替代离散路由，允许基于梯度的专家路由优化。我们进一步参数化每层平均激活专家数，并施加全局预算约束，使模型能够学习跨层分配专家容量。SoftMoE完全兼容自回归建模，在语言建模和下游任务上达到与稀疏MoE相当或更优的性能，同时激活显著更少的专家。值得注意的是，学习到的分配高度非均匀，后层激活更多专家。源代码已公开$^\dagger$。
英文摘要：
Sparse Mixture-of-Experts (MoE) architectures enable scaling LLM parameters under a fixed inference budget by activating only a small subset of experts via top-$k$ routing. While this preserves causality and suits autoregressive language models, the discrete top-$k$ operator is not differentiable, forcing a fixed number of active experts per input and resulting in inefficient use of computation. We propose SoftMoE, which replaces discrete routing with a truncated soft top-$k$ LapSum relaxation, allowing gradient-based optimization of expert routing. We further parameterize the mean number of active experts per layer and impose a global budget constraint, enabling the model to learn how to allocate expert capacity across layers. SoftMoE remains fully compatible with autoregressive modeling and achieves performance comparable to or better than sparse MoE on language modeling and downstream tasks, while activating significantly fewer experts. Notably, the learned allocation is highly non-uniform, with later layers activating more experts. The source code is publicly available$^\dagger$.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
