专家混合中的等效性：路由拓扑不决定语言建模质量
Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
AI总结：
本文研究路由拓扑对语言建模质量的影响，通过62次实验发现不同路由方法在渐近困惑度上无显著差异，且通过简化路由机制提升了效率。
摘要：
稀疏混合专家（MoE）架构采用日益复杂的路由机制——学习路由器、多跳轨迹、令牌依赖门控。我们问：路由拓扑实际上决定语言建模质量吗？我们构建了一个几何MoE（ST-MoE），使用余弦相似度路由对抗学习的聚类中心，在低维空间（d_space = 64）中，要求路由参数比标准线性路由器少80%。通过62次在WikiText-103上的受控实验（76-84M参数，训练到收敛（50K步，1.64B tokens）），我们发现路由拓扑不决定渐近困惑度（PPL）：五个余弦路由变体在1-PPL范围内统计上等价（Two One-Sided Tests [TOST]，所有10对比较p < 0.05；15次运行跨3种子，观察范围33.93-34.72）。该发现扩展到哈希、随机固定和top-1路由（单种子；渐进1.1-2.2 PPL退化）并在OpenWebText上重复（0.03 PPL差距，6次运行，每个种子3次）。标准线性路由器有5.3倍更多的路由参数达到PPL 32.76，但等参数余弦路由缩小了67%的差距——真正的机制优势约为1.2%。机理解释是收敛冗余：多跳更新是共线的（cos(Δh_0, Δh_1) = 0.805），实现幅度放大而非组合推理；单个可学习标量复制多跳性能。作为实际收益，零次相对范数停止节省了25%的MoE FLOPS，同时增加0.12%的PPL。专家级专业化和因果可控性——与拓扑级等效性共存——在配套论文中进行了探讨。
英文摘要：
Sparse Mixture-of-Experts (MoE) architectures employ increasingly sophisticated routing mechanisms -- learned routers, multi-hop trajectories, token-dependent gating. We ask: does routing topology actually determine language modeling quality? We build a geometric MoE (ST-MoE) using cosine-similarity routing against learned centroids in a low-dimensional space ($d_{space} = 64$), requiring 80% fewer routing parameters than standard linear routers. Through 62 controlled experiments on WikiText-103 at 76--84M parameters trained to convergence (50K steps, 1.64B tokens), we find that routing topology does not determine asymptotic perplexity (PPL): five cosine-routing variants are statistically equivalent within a 1-PPL margin (Two One-Sided Tests [TOST], $p < 0.05$ for all 10 pairwise comparisons; 15 runs across 3 seeds, observed range 33.93--34.72). The finding extends to hash, random-fixed, and top-1 routing (single-seed; graceful 1.1--2.2 PPL degradation) and replicates on OpenWebText (0.03 PPL gap, 6 runs, 3 seeds each). A standard linear router with 5.3$\times$ more routing parameters reaches PPL 32.76, but iso-parameter cosine routing closes 67% of this gap -- the true mechanism advantage is $\sim$1.2%. The mechanistic explanation is convergent redundancy: multi-hop updates are collinear ($\cos(Δh_0, Δh_1) = 0.805$), implementing magnitude amplification rather than compositional reasoning; a single learnable scalar replicates multi-hop performance. As a practical payoff, zero-shot relative-norm halting saves 25% of MoE FLOPs at +0.12% PPL. Expert-level specialization and causal controllability -- which coexist with topology-level equifinality -- are explored in a companion paper.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
