双语混合专家语言模型中专家路由的声明式-过程式视角
A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models
浏览论文内容
中文总结 AI 辅助
该研究探究双语MoE语言模型的专家路由是否形成语言结构，对比课程学习与无课程训练的模型，发现无课程模型专业化更强但依赖随机种子，课程模型路由更均衡，揭示MoE路由可出现可解释语言组织。
中文摘要 AI 辅助
我们探究混合专家（Mixture-of-Experts, MoE）语言模型在双语语言习得过程中是否会形成具有语言结构的专家路由。受声明式-过程式框架启发，我们分析了仅解码器结构的英-德MoE Transformer模型的词汇、语法及句法处理，该模型在顺序语言暴露的设置下进行训练。我们构建了基于探测法的验证集，并提取词元级别的路由分布，通过互信息、路由熵和Jensen-Shannon距离来量化类别依赖的专业化程度。经课程学习训练的模型在第5层达到0.1148的峰值互信息，表明不同语言类别间的路由分布存在类别依赖差异。令人惊讶的是，在混合英-德数据上训练的无课程基线模型展现出更强的整体专业化程度，在同一层达到0.2599的峰值互信息。这些结果表明，即使没有顺序语言暴露，MoE的路由模式中也会出现可解释的语言组织。在第二个训练随机种子下的重复实验显示，无课程条件的专业化集中于单一语言，其身份依赖于随机种子，而课程学习则始终产生稳定的、语言均衡的路由分布；分阶段的双语暴露并未均匀提升专业化程度，而是降低了单一语言的主导地位。官方GitHub仓库：this https URL
英文摘要
We investigate whether Mixture-of-Experts (MoE) language models develop linguistically structured expert routing during bilingual language acquisition. Inspired by the Declarative-Procedural framework, we analyze lexical, grammatical, and syntactic processing in a decoder-only English-German MoE Transformer trained under sequential language exposure. We construct a probe-based validation set and extract token-level routing distributions to quantify category-dependent specialisation using mutual information, routing entropy, and Jensen-Shannon distance. The curriculum-trained model exhibits a peak mutual information of 0.1148 at layer 5, indicating category-dependent differences in routing distributions across linguistic categories. Surprisingly, a no-curriculum baseline trained on mixed English-German data shows stronger aggregate specialisation, reaching a peak mutual information of 0.2599 at the same layer. These results suggest that interpretable linguistic organization emerges within MoE routing patterns even without sequential language exposure. A replication at a second training seed shows that the no-curriculum condition's specialisation concentrates on a single language whose identity is seed-dependent, whereas the curriculum consistently yields a stable, language-balanced routing profile; rather than uniformly increasing specialisation, staged bilingual exposure reduces single-language dominance. The official Github repository: https://github.com/Amrit828/DP-Theory-MOE-Interpretability-Research
发表机构
- Sri Sivasubramaniya Nadar College of Engineering（斯里·西瓦苏布拉马尼亚·纳达尔工程学院）
- Shiv Nadar University Chennai（钦奈希夫·纳达尔大学）
机构由 AI 辅助整理，请以论文原文为准。
