联合MoE缩放定律：混合专家模型可实现内存高效
Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
AI总结：
针对MoE模型在内存约束下的可扩展性与效率未被充分探索的问题，本研究提出纳入活跃参数数量等关键因素的联合缩放定律，通过超280次实验验证其理论预测，发现MoE模型可比密集模型更内存高效，为最优配置选择提供框架与可操作见解。
摘要：
混合专家（MoE）架构在大规模机器学习模型的研究与实际应用中显著提升了计算效率。然而，其在内存约束下的可扩展性与效率仍未得到充分探索。本研究提出了密集模型与MoE模型的联合缩放定律，纳入了活跃参数数量、数据集规模和专家数量等关键因素。研究结果为固定内存与计算预算下选择最优MoE配置提供了原则性框架。令人惊讶的是，我们发现MoE模型可比密集模型更内存高效，这与传统认知相悖。为推导并验证缩放定律的理论预测，我们开展了超280次实验，涉及最多27亿活跃参数与50亿总参数。这些结果为实际大规模训练场景中MoE模型的设计与部署提供了可操作的见解。
英文摘要：
Mixture of Experts (MoE) architectures have significantly increased computational efficiency in both research and real-world applications of large-scale machine learning models. However, their scalability and efficiency under memory constraints remain relatively underexplored. In this work, we present joint scaling laws for dense and MoE models, incorporating key factors such as the number of active parameters, dataset size, and the number of experts. Our findings provide a principled framework for selecting the optimal MoE configuration under fixed memory and compute budgets. Surprisingly, we show that MoE models can be more memory-efficient than dense models, contradicting conventional wisdom. To derive and validate the theoretical predictions of our scaling laws, we conduct over 280 experiments with up to 2.7B active parameters and up to 5B total parameters. These results offer actionable insights for designing and deploying MoE models in practical large-scale training scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
