SDG-MoE：带符号辩论图混合专家
SDG-MoE: Signed Debate Graph Mixture-of-Experts
AI总结：
SDG-MoE通过引入轻量级迭代辩论步骤提升混合专家模型性能，通过支持图和批评图捕获专家间相互影响，结合分歧门控机制保持专业性，实验显示在预训练任务中表现优于基线模型。
摘要：
稀疏MoE模型通过将每个token路由到少量专家实现容量与计算的平衡。然而，大多数MoE架构中，选定的专家独立处理token，其输出通过加权和合并。本文提出SDG-MoE，一种新型架构，通过在最终聚合前加入轻量级迭代辩论步骤。SDG-MoE引入三个组件：(i)两个学习的交互矩阵，支持图$A^+$和批评图$A^-$，捕捉强化和纠正影响；(ii)带符号的消息传递步骤，在聚合前更新专家表示；(iii)分歧门控的Friedkin-Johnsen式锚定，控制辩论强度并防止专家漂移。这些组件共同实现结构化辩论过程，其中交互强度随分歧扩大，专业性得以保持。我们还提供了理论分析，建立了专家状态的稳定性条件，并证明辩论仅在活跃集上引入低阶开销。在受控的三种子预训练实验中，SDG-MoE在验证困惑度上优于无符号图通信基线和普通MoE，以19.8%的优势胜过最强基线，并在WikiText-103、C4和Paloma上取得最佳外部困惑度。
英文摘要：
Sparse MoE models achieve a good balance between capacity and compute by routing each token to a small subset of experts. However, in most MoE architectures, once a token is routed, the selected experts process it independently and their outputs are combined via a weighted sum. This leaves open whether enabling communication among them could improve performance. While prior work has raised this question, direct interaction among the active routed experts remains underexplored. In this paper, we propose SDG-MoE (Signed Debate Graph Mixture-of-Experts), a novel architecture that adds a lightweight, iterative deliberation step before final aggregation. SDG-MoE introduces three components: (i) two learned interaction matrices over the active experts, a support graph $A^+$ and a critique graph $A^-$, capturing reinforcing and corrective influences; (ii) a signed message-passing step that updates expert representations before aggregation; and (iii) a disagreement-gated Friedkin-Johnsen-style anchoring that controls deliberation strength while preventing expert drift. Together, these enable a structured deliberation process where interaction strength scales with disagreement and specialization is preserved. We also provide a theoretical analysis establishing stability conditions on expert states and showing that deliberation adds only low-order overhead over the active set. In controlled three-seed pretraining experiments, SDG-MoE improves validation perplexity over both an unsigned graph communication baseline and vanilla MoE, outperforming the strongest baseline by 19.8%, and gives the best external perplexity on WikiText-103, C4, and Paloma among the compared systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
