Grassmannian混合专家：子空间流形上的集中控制路由
Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds
AI总结：
GrMoE通过Grassmannian流形上的浓度控制机制，实现子空间路由的稀疏性优化与负载平衡改进。
摘要：
混合专家模型依赖于学习到的路由器来分配标记给专家，但标准的softmax门控没有提供一种原则性的机制来控制稀疏性和利用率之间的权衡。我们提出了Grassmannian MoE（GrMoE），一种在子空间流形的Grassmannian流形上运行的路由框架，其中门控权重源于矩阵Bingham分布的浓度参数。这种构造产生了一个单一的、可解释的调节器——浓度矩阵Λ——可以连续地控制路由熵，取代离散的top-k选择，用一种平滑的、基于几何的稀疏性机制。我们进一步开发了一种用于后验路由分布的近似变分推断程序，使不确定性的专家分配变得自然，能够抵抗专家崩溃。我们正式证明了Bingham浓度谱与路由熵、预期top-k质量以及专家崩溃的指数界之间的紧密关系，建立了首个关于集中控制稀疏性的正式理论。在合成路由任务中，一个具有8个专家的350M参数MoE语言模型、一个具有16个专家的1.3B参数模型以及一个具有32个专家的2.7B参数模型，GrMoE在所有种子下均实现0%的路由崩溃，与15-30%改进的负载平衡相比， perplexity相当或更优，并且在浓度和有效稀疏性之间存在平滑的单调关系，这使得在不重新训练的情况下能够进行事后稀疏性调整。标记级分析显示，专家学习到异质的浓度值，这些值与语言学专业化相关，提供了可解释的路由行为。
英文摘要：
Mixture-of-Experts models rely on learned routers to assign tokens to experts, yet standard softmax gating provides no principled mechanism to control the tradeoff between sparsity and utilization. We propose Grassmannian MoE (GrMoE), a routing framework that operates on the Grassmannian manifold of subspaces, where gating weights arise from the concentration parameters of Matrix Bingham distributions. This construction yields a single, interpretable knob -- the concentration matrix $Λ$ -- that continuously controls routing entropy, replacing discrete top-$k$ selection with a smooth, geometrically principled sparsity mechanism. We further develop an amortized variational inference procedure for posterior routing distributions, enabling uncertainty-aware expert assignment that naturally resists expert collapse. We formally prove tight bounds relating the Bingham concentration spectrum to routing entropy, expected top-$k$ mass, and an exponential bound on expert collapse, establishing the first formal theory of concentration-controlled sparsity. On synthetic routing tasks, a 350M-parameter MoE language model with 8 experts, a 1.3B-parameter model with 16 experts, and a 2.7B-parameter model with 32 experts, GrMoE achieves 0\% routing collapse across all seeds, comparable or better perplexity with 15--30\% improved load balance, and a smooth monotonic relationship between concentration and effective sparsity that enables post-hoc sparsity tuning without retraining. Token-level analysis reveals that experts learn heterogeneous concentration values that correlate with linguistic specialization, providing interpretable routing behavior.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
