稀疏混合专家变压器中的任务条件路由签名
Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers
AI总结：
研究发现稀疏混合专家变压器中的路由机制具有任务条件结构，通过路由签名分析显示同类任务提示具有高相似性，不同任务提示相似性较低，且基于路由签名的分类器在任务分类中表现优异。
摘要：
稀疏混合专家（MoE）架构通过条件计算实现了大型语言模型的高效扩展，但负责专家选择的路由机制仍不为人所理解。在本工作中，我们引入了路由签名，这是一种总结给定提示下各层专家激活模式的向量表示，并利用它们研究MoE路由是否具有任务条件结构。使用OLMoE-1B-7B-0125-Instruct作为经验测试平台，我们显示同一任务类别提示诱导出高度相似的路由签名，而不同类别提示则表现出显著较低的相似性。类内路由相似性（0.8435±0.0879）显著高于类间相似性（0.6225±0.1687），对应Cohen's d = 1.44。仅使用路由签名训练的逻辑回归分类器在四类任务分类上的交叉验证准确率为92.5%±6.1%。为确保统计有效性，我们引入了排列和负载平衡基线，并表明观察到的分离不被稀疏性或平衡约束单独解释。我们进一步分析了层间信号强度和路由签名的低维投影，发现任务结构在更深层中变得更加明显。这些结果表明，稀疏变压器中的路由不仅仅是一个平衡机制，而是条件计算中可测量的任务敏感组件。我们发布了MOE-XRAY，一个轻量级的路由 telemetry 和分析工具包。
英文摘要：
Sparse Mixture-of-Experts (MoE) architectures enable efficient scaling of large language models through conditional computation, yet the routing mechanisms responsible for expert selection remain poorly understood. In this work, we introduce routing signatures, a vector representation summarizing expert activation patterns across layers for a given prompt, and use them to study whether MoE routing exhibits task-conditioned structure. Using OLMoE-1B-7B-0125-Instruct as an empirical testbed, we show that prompts from the same task category induce highly similar routing signatures, while prompts from different categories exhibit substantially lower similarity. Within-category routing similarity (0.8435 +/- 0.0879) significantly exceeds across-category similarity (0.6225 +/- 0.1687), corresponding to Cohen's d = 1.44. A logistic regression classifier trained solely on routing signatures achieves 92.5% +/- 6.1% cross-validated accuracy on four-way task classification. To ensure statistical validity, we introduce permutation and load-balancing baselines and show that the observed separation is not explained by sparsity or balancing constraints alone. We further analyze layer-wise signal strength and low-dimensional projections of routing signatures, finding that task structure becomes increasingly apparent in deeper layers. These results suggest that routing in sparse transformers is not merely a balancing mechanism, but a measurable task-sensitive component of conditional computation. We release MOE-XRAY, a lightweight toolkit for routing telemetry and analysis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
