探究大型混合专家模型中的语义路由
Probing Semantic Routing in Large Mixture-of-Expert Models
AI总结：
本研究针对大型混合专家（MoE）模型的路由机制，通过两项对照实验验证输入语义对专家路由的影响，证实了大型MoE模型中语义路由的存在，为理解其功能分化提供了依据。
摘要：
过去一年，参数量超过100B的大型混合专家（Mixture-of-Expert, MoE）模型在开放领域日益普及。尽管其优势常从效率角度被阐释，但已有研究也通过路由行为探索了功能分化现象。我们研究大型MoE模型中的专家路由是否受输入语义影响。为此我们设计了两项对照实验：其一，对比包含同一目标词但词义相同或不同的句子对的激活情况；其二，固定上下文，将目标词替换为语义相似或不相似的备选词。对比不同条件下的专家重叠度，发现了大型MoE模型中存在语义路由的清晰且具统计显著性的证据。
英文摘要：
In the past year, large (>100B parameter) mixture-of-expert (MoE) models have become increasingly common in the open domain. While their advantages are often framed in terms of efficiency, prior work has also explored functional differentiation through routing behavior. We investigate whether expert routing in large MoE models is influenced by the semantics of the inputs. To test this, we design two controlled experiments. First, we compare activations on sentence pairs with a shared target word used in the same or different senses. Second, we fix context and substitute the target word with semantically similar or dissimilar alternatives. Comparing expert overlap across these conditions reveals clear, statistically significant evidence of semantic routing in large MoE models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
