路由分歧并非等权MoE自蒸馏中行为影响的证据
Routing Divergence Is Not Evidence of Behavioral Influence in Same-Weight MoE Self-Distillation
浏览论文内容
中文总结 AI 辅助
该研究针对等权MoE自蒸馏中路由分歧问题，通过分块分解等方法验证路由项暴露度小，路由移动非行为影响证据，提出需先测暴露度再采取干预。
中文摘要 AI 辅助
两个混合专家（Mixture-of-Experts，MoE）的前向传播可共享所有权重，却将相同token路由至不同专家，这在等权自蒸馏中形成潜在盲区——该场景下由演示条件的教师监督仅含查询的学生。本研究以单步形式探究这种不匹配，采用冻结权重而非完整训练轨迹作为代理。精确的分块分解将路由项（固定内容下改变门控）与类稠密内容项分离。在7个开放权重检查点和2个领域中，路由项仅占分块输出的1.6倍，而其残差流暴露度为3.2倍；暴露度按被路由分块的残差占比排序，对两个验证模型的常开主干进行缩放时，暴露度呈单调变化；共模控制支持质量与一致性机制，而非仅分母稀释。对三个模型的预注册PubMedQA测试显示，完整路由项使输出变化小于自然上下文效应的一半，且在很大程度上可由匹配范数噪声复现，而内容项具有强方向特异性。缩放与合并专家探测表明，分块级窄范围并非普遍现象，不过在测试边界处暴露度仍较小。因此，仅路由移动并非行为影响的证据：需先测量暴露度，在决策重要时采用行为干预。
英文摘要
Two Mixture-of-Experts (MoE) forward passes can share every weight yet route the same token through different experts. This creates a possible blind spot in same-weight self-distillation, where a demonstration-conditioned teacher supervises a query-only student. We study this mismatch in its single-step form, with frozen weights rather than as a proxy for a full training trajectory. An exact blockwise decomposition separates a routing term, which changes gates at fixed content, from a dense-like content term. Across seven open-weight checkpoints and two domains, the routing term spans only $1.6\times$ as a fraction of block output, while its residual-stream exposure spans $3.2\times$. Exposure is ordered by the routed block's share of the residual. Scaling the always-on backbone in two confirmatory models moves exposure monotonically; common-mode controls support a mass-and-coherence mechanism rather than denominator dilution alone. Preregistered PubMedQA patches on three models show that the full routing term moves outputs by less than half the natural context effect and is largely reproduced by matched-norm noise, whereas the content term is strongly direction-specific. Scale and merged-expert probes show that the narrow block-level range is not universal, although exposure remains small at the tested boundaries. Router movement alone is therefore not evidence of behavioral influence: measure exposure first, and use a behavioral intervention when the decision matters.
发表机构
- Lunit（卢尼特公司）
机构由 AI 辅助整理，请以论文原文为准。
