细粒度专家的力量：粒度提升 Mixture of Experts 的表达能力
The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts
AI总结：
本文研究 MoE 层活跃专家数量即粒度对表达能力的影响，证明高、低粒度架构间存在指数级表达能力分离，并用实验验证高粒度更具优势。
摘要：
Mixture-of-Experts（MoE，混合专家）层日益成为前沿模型架构的核心。通过选择性地激活参数，它们在扩展总参数量的同时降低计算成本。本文研究活跃专家数量（称为粒度）的影响，将具有较多专家的架构（例如 DeepSeek 中每层 8 个）与专家较少的架构（例如 Llama-4 模型中每层 1 个）进行比较。我们证明，基于这一设计参数，网络表达能力存在指数级分离，表明模型可从更高粒度中受益。实验结果证实了我们的理论发现，并展示了这种分离。
英文摘要：
Mixture-of-Experts (MoE) layers are increasingly central to frontier model architectures. By selectively activating parameters, they reduce computational cost while scaling total parameter count. This paper investigates the impact of the number of active experts, termed granularity, comparing architectures with many (e.g., 8 per layer in DeepSeek) to those with fewer (e.g., 1 per layer in Llama-4 models). We prove an exponential separation in network expressivity based on this design parameter, suggesting that models benefit from higher granularity. Experimental results corroborate our theoretical findings and illustrate this separation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
