领域专精的幻觉：揭示混合专家模型中的领域不变‘ standing committee ’
The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models
AI总结：
本研究质疑混合专家模型通过稀疏路由实现领域专精的假设，提出COMMITTEEAUDIT框架分析专家组而非个体专家的路由行为，发现领域不变的standing committee，揭示模型存在向集中计算偏倚的结构倾向，表明混合专家模型中的专精程度远低于预期。
摘要：
混合专家模型被广泛假设通过稀疏路由实现领域专精。在本工作中，我们通过引入COMMITTEEAUDIT框架，质疑这一假设，该框架在专家组层面而非个体专家层面分析路由行为。在三个代表性模型和MMLU基准测试中，我们揭示了一个领域不变的standing committee。这是一个紧凑的路由专家联盟，能够跨领域、层和路由预算持续捕获大多数路由质量，即使在架构已包含共享专家的情况下。定性分析进一步显示，standing committee锚定推理结构和语法，而外围专家处理领域特定知识。这些发现揭示了模型对集中计算的强结构偏倚，表明混合专家模型中的专精程度远低于人们普遍认为的水平。这种固有偏倚也表明，当前的训练目标，如强制均匀专家利用的负载平衡损失，可能与模型的自然优化路径相悖，从而限制了训练效率和性能。
英文摘要：
Mixture of Experts models are widely assumed to achieve domain specialization through sparse routing. In this work, we question this assumption by introducing COMMITTEEAUDIT, a post hoc framework that analyzes routing behavior at the level of expert groups rather than individual experts. Across three representative models and the MMLU benchmark, we uncover a domain-invariant Standing Committee. This is a compact coalition of routed experts that consistently captures the majority of routing mass across domains, layers, and routing budgets, even when architectures already include shared experts. Qualitative analysis further shows that Standing Committees anchor reasoning structure and syntax, while peripheral experts handle domain-specific knowledge. These findings reveal a strong structural bias toward centralized computation, suggesting that specialization in Mixture of Experts models is far less pervasive than commonly believed. This inherent bias also indicates that current training objectives, such as load-balancing losses that enforce uniform expert utilization, may be working against the model's natural optimization path, thereby limiting training efficiency and performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
