几何路由实现混合专家中的因果专家控制
Geometric Routing Enables Causal Expert Control in Mixture of Experts
AI总结：
本文通过几何路由揭示混合专家模型中专家的专业化特性，展示专家身份的因果意义，并通过实验验证专家专业化的可解释性。
摘要：
稀疏混合专家（MoE）模型在增加参数规模的同时保持每个token的活跃计算量不变，但单个专家的专业化仍不清晰。本文通过几何路由在低维度量空间中使专家专业化直接可观察。我们提供了四条证据：首先，通过解嵌入矩阵投影专家输出向量得到语义词典，15%的专家是单语义专家；其次，路由表现出频率到语法的梯度，早期层按词频分离，深层按语法类别分离；第三，因果干预验证这些标签，引导至时间专家的质心使P(temporal)增加321%；第四，干预不专属于余弦路由，线性路由支持相似的引导，但只有余弦路由提供几何透明性。MoE专家级别专业化是可解释性基础要素：架构单语义，因果验证，且在推理中可控且无开销。
英文摘要：
Sparse Mixture-of-Experts (MoE) models scale parameters while fixing active computation per token, but the specialization of individual experts remains opaque. In a companion paper we showed that routing topology is quality-neutral: five structurally different configurations converge to statistically equivalent language modeling quality. Here we show that expert identity is nonetheless causally meaningful: individual rank-1 experts are monosemantic by construction, and cosine-similarity routing in a low-dimensional metric space makes their specialization directly inspectable. We present four lines of evidence. First, projecting expert output vectors through the unembedding matrix yields a Semantic Dictionary: 15% of experts are monosemantic specialists spanning 10 categories (temporal, geographic, cardinal, discourse, emotional, financial, military, scientific). Second, routing exhibits a frequency-to-syntax gradient: early layers separate tokens by word frequency, deeper layers by syntactic class (Zipf-confound controls, all $p < 0.001$). Third, causal interventions confirm these labels: steering toward a temporal expert's centroid increases P(temporal) by +321% (median across 44 prompts); suppressing a geographic expert drops P(geographic) by -23%; rewriting an expert's output vector halves target-category probability, and effects compose additively across layers. Fourth, the interventions are not unique to cosine routing: linear routers support comparable steering, but only cosine routing provides geometric transparency -- expert specialization is readable directly from the centroid matrix. MoE expert-level specialization is a first-class interpretability primitive: architecturally monosemantic, causally validated, and controllable at inference with zero overhead.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
