MoE路由测试平台：研究专家专业化和路由行为的小规模情况
MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
AI总结：
本文提出MoE路由测试平台，通过对比不同路由方法，验证了在小规模下平衡范围是实现专家专业化和高利用的关键因素。
摘要：
稀疏混合专家（MoE）架构在前沿大语言模型（LLM）中日益流行，但路由复杂性带来了训练挑战。充分利用MoE模型参数需要所有专家都得到良好训练并以非冗余方式专业化。然而，由于缺乏标准指标，评估这一要求颇具挑战性，且许多路由技术在较小规模上表现相似，这通常不能反映其在大规模上的行为。为此，我们提出MoE路由测试平台，该平台通过使用真实数据提供更清晰的路由动态视图。该平台将具有明显可区分领域的数据混合与一个参考路由器结合，该路由器根据这些领域指定理想的路由，从而提供一个明确的上界用于比较。这使专家专业化能够进行可量化测量。为了展示测试平台的价值，我们比较了各种MoE路由方法，并证明平衡范围是实现专业化同时保持高专家利用率的关键因素。我们确认这一观察结果在35倍更大的模型中也成立。
英文摘要：
Sparse Mixture-of-Experts (MoE) architectures are increasingly popular for frontier large language models (LLM) but they introduce training challenges due to routing complexity. Fully leveraging parameters of an MoE model requires all experts to be well-trained and to specialize in non-redundant ways. Assessing this, however, is complicated due to lack of established metrics and, importantly, many routing techniques exhibit similar performance at smaller sizes, which is often not reflective of their behavior at large scale. To address this challenge, we propose the MoE Routing Testbed, a setup that gives clearer visibility into routing dynamics at small scale while using realistic data. The testbed pairs a data mix with clearly distinguishable domains with a reference router that prescribes ideal routing based on these domains, providing a well-defined upper bound for comparison. This enables quantifiable measurement of expert specialization. To demonstrate the value of the testbed, we compare various MoE routing approaches and show that balancing scope is the crucial factor that allows specialization while maintaining high expert utilization. We confirm that this observation generalizes to models 35x larger.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
