分层专家混合模型与两阶段优化
Hierarchical Mixture-of-Experts with Two-Stage Optimization
AI总结：
本文提出Hi-MoE分层专家混合框架，通过双阶段优化解决路由负载均衡与专家专业化之间的矛盾，提升模型稳定性和表现。
摘要：
稀疏混合专家（MoE）模型通过路由每个token到少量专家来扩展容量。然而，其路由器存在根本性权衡：强负载均衡会抑制专家专业化，而激进的多样性常导致路由崩溃。我们提出Hi-MoE，一种分组MoE框架，将路由控制分解为两个耦合层次：(i) 组间负载均衡强制专家组间流量公平，(ii) 组内专业化促进互补专家行为并防止组内崩溃。我们的分析提供了原理性解释，说明分层目标如何重塑路由器，从而促进稳定专业化并缓解崩溃。我们观察到在NLP和视觉基准上，Hi-MoE在稀疏路由和分组MoE基线方面有持续改进，并通过扩展研究（模型大小、专家数量）和针对性消融验证了鲁棒性。在58B token大规模预训练中，Hi-MoE-7B在多样评估领域实现了5.6%的困惑度降低和OLMoE-7B的40%专家平衡提升。
英文摘要：
Sparse Mixture-of-Experts (MoE) models scale capacity by routing each token to a small subset of experts. However, their routers exhibit a fundamental trade-off: strong load balancing can suppress expert specialization, while aggressive diversity often causes routing collapse. We propose Hi-MoE, a grouped MoE framework that decomposes routing control into two coupled levels: (i) inter-group balancing that enforces fair traffic across expert groups, and (ii) intra-group specialization that promotes complementary expert behaviors while preventing within-group collapse. Our analysis provides a principled explanation of how our hierarchical objectives reshape the router, thereby promoting stable specialization and mitigating collapse. We observe consistent improvements over recent sparse-routing and grouped-MoE baselines across NLP and vision benchmarks, and confirm robustness via scaling studies (model size, expert count) and targeted ablations. In large-scale pre-training on 58B tokens, Hi-MoE-7B achieves a 5.6% perplexity reduction and a 40% improvement in expert balance over OLMoE-7B across diverse evaluation domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
