发表机构
Bahçeşehir University（巴切希尔大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对MoE路由缺乏可审计记录的问题，提出SA-MoE，通过低维持久状态形成路由账本，以极低计算开销实现跨层影响的可解释追踪，并显著提升长期路由可达性。
AI 中文摘要
混合专家（Mixture-of-Experts, MoE）模型反复将令牌路由到专家的稀疏子集，但传统路由器不提供关于跨层影响如何累积的路由特定记录。我们引入了 Scratchpad-Augmented Mixture-of-Experts（SA-MoE），它使每个路由器都能访问一个低维持久状态，该状态不提供给专家。学习到的逐层写入更新此状态，其更新后的实际变化精确分解了状态介导的对任何后续路由边际的贡献，形成路由账本。在稀疏升级的基于 SmolLM2 和 Gemma 的模型中，以及每种架构的三个独立训练种子中，该路径增加了不到 1% 的分析前向计算量，并且被训练好的路由器强烈使用：局部移除其路由器贡献分别改变了 87.6% 和 69.9% 的决策中所选的 Top-2 专家集。相对于匹配的仅最新写入的对照组，持久累积将长期未来路由可达性提高了 19.4 和 12.2 个百分点，且在每个种子中都有积极效果。在两个模型家族中，超过 90% 的绝对账本贡献来自非近期写入，并且完全前向抑制账本选择的写入会改变后续路由和输出分布。该账本是持久状态路径的精确来源对象，而非路由的完整因果解释。敏感性感知分数能更好地预测完全前向干预效果，事后方法无需架构修改即可恢复相关的跨层归因。SA-MoE 反而在模型自然前向计算中使一个路由特定的计算历史变得明确且可直接检查。
英文摘要
Mixture-of-Experts (MoE) models repeatedly route tokens to sparse subsets of experts, but conventional routers expose no routing-specific record of how cross-layer influences accumulate. We introduce Scratchpad-Augmented Mixture-of-Experts (SA-MoE), which gives each router access to a low-dimensional persistent state that is not provided to the experts. Learned layerwise writes update this state, and their realized post-update changes exactly decompose the state-mediated contribution to any later routing margin, forming a routing ledger. Across sparsely upcycled SmolLM2- and Gemma-based models and three independent training seeds per architecture, this pathway adds less than 1% analytical forward compute and is strongly used by trained routers: local removal of its router contribution changes the selected Top-2 expert set in 87.6% and 69.9% of decisions, respectively. Relative to a matched latest-write-only control, persistent accumulation increases long-horizon future-routing accessibility by 19.4 and 12.2 percentage points, with positive effects in every seed. More than 90% of absolute ledger contribution comes from non-recent writes in both families, and full-forward suppression of ledger-selected writes changes later routing and output distributions. The ledger is an exact provenance object for the persistent-state pathway, not a complete causal explanation of routing. Sensitivity-aware scores better predict full-forward intervention effects, and post-hoc methods recover related cross-layer attribution without architectural modification. SA-MoE instead makes one routing-specific computational history explicit and directly inspectable within the model's natural forward computation.
