专家混合用于低资源大语言模型
Mixture of Experts for Low-Resource LLMs
AI总结：
本文研究了专家混合架构在低资源语言中的专家路由行为，发现持续预训练能有效缓解路由不平衡问题，且路由改进与下游任务表现提升相关。
摘要：
混合专家（MoE）架构能够实现高效的模型扩展，但跨低资源语言的专家路由行为仍不明确。我们通过希伯来语这一形态丰富且低资源的测试环境，分析了两种架构不同的MoE模型——纯Transformer（Qwen3-30B-A3B）和混合Mamba-Transformer（Nemotron-3-Nano-30B-A3B）的路由动态。两种预训练模型均表现出深度层路由崩溃：在最终层使用熵急剧下降，令牌集中在狭窄的专家子集，这种模式在英语中很少见。持续预训练（CPT）在平衡双语数据上显著纠正了这种不平衡，提高了熵并使路由转向共享、语言无关的专家；监督微调（SFT）单独实现的纠正程度较低。将分析扩展到日语，发现定量一致的崩溃特征，提供了跨语言证据，表明该现象是预训练下代表性不足的系统性结果，而非任何语言固有属性。路由改进与一致的下游基准表现提升相关，将路由熵和专家专业化定位为MoE系统多语言能力的原理性诊断。
英文摘要：
Mixture-of-Experts (MoE) architectures enable efficient model scaling, yet expert routing behavior across underrepresented languages remains poorly understood. We analyze routing dynamics in two architecturally distinct MoE models -- a pure Transformer (Qwen3-30B-A3B) and a hybrid Mamba-Transformer (Nemotron-3-Nano-30B-A3B) -- using Hebrew as a morphologically rich, low-resource testbed. Both pre-trained models exhibit \emph{deep-layer routing collapse}: usage entropy drops sharply in final layers and tokens concentrate on a narrow expert subset, a pattern largely absent for English. Continual pre-training (CPT) on balanced bilingual data substantially corrects this imbalance, increasing entropy and shifting routing toward shared, language-agnostic experts; supervised fine-tuning (SFT) alone achieves less complete correction. Extending the analysis to Japanese reveals quantitatively consistent collapse signatures, providing cross-linguistic evidence that the phenomenon is a systematic consequence of pre-training underrepresentation rather than any language-intrinsic property. Routing improvements correlate with consistent downstream benchmark gains, positioning routing entropy and expert specialization as principled diagnostics for multilingual capacity in MoE systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
