稀疏混合专家模型中的软到硬路由
Soft-to-Hard Routing in Sparse Mixture-of-Experts Models
AI总结：
本文通过边界层微积分方法，研究了稀疏混合专家模型中softmax路由随温度趋于零时趋近于硬top-1路由的极限过程，并给出了基于路由界面邻域概率的定量误差界。
摘要：
随着温度趋于零，softmax路由趋近于硬top-1路由，但极限过程在路由器平局时存在奇异性。本文针对总体平方损失混合专家回归中的软到硬极限，发展了一种边界层微积分方法。对于具有logits $a_k(x;ϕ)$的路由器，相关的局部量是前两名的间隔$Δ(x;ϕ)$，相关的全局量是边界质量$\\mathbb{P}(Δ(X;ϕ)\\\le w)$。在光滑性和横截性假设下，余面积和管状邻域估计展示了该质量如何随板宽缩放；在二元情形中，主导系数是路由界面上的显式曲面积分。这些几何估计给出了软目标$L_τ$和硬目标$L_0$之间的定量界，包括在间隔尾条件下的$O(τ^α)$一致比较，并得到了紧参数空间上软目标的$Γ$-收敛性。主要结论是，零温度近似由路由界面的$O(τ)$邻域所承载的概率控制，而不仅仅由温度本身决定。在分离出问题的这一边界层部分后，我们记录了一个从硬路由到小温度软路由的条件景观传递定理，以及一个简化的双专家高斯计算，展示了局部对称性破缺。仅包含合成诊断作为边界层预测的受控检验。
英文摘要：
Softmax routing approaches hard top-1 routing as the temperature tends to zero, but the limiting passage is singular at router ties. This paper develops a boundary-layer calculus for this soft-to-hard limit in population squared-loss mixture-of-experts regression. For a router with logits $a_k(x;ϕ)$, the relevant local quantity is the top-two margin $Δ(x;ϕ)$, and the relevant global quantity is the boundary mass $\mathbb{P}(Δ(X;ϕ)\le w)$. Under smoothness and transversality assumptions, coarea and tubular-neighborhood estimates show how this mass scales with the slab width; in the binary case the leading coefficient is an explicit surface integral over the routing interface. These geometric estimates give quantitative bounds between the soft objective $L_τ$ and the hard objective $L_0$, including an $O(τ^α)$ uniform comparison under a margin-tail condition, and yield $Γ$-convergence of the soft objectives on compact parameter spaces. The main conclusion is that the zero-temperature approximation is controlled by the probability carried by an $O(τ)$ neighborhood of the routing interfaces, not by temperature alone. After isolating this boundary-layer part of the problem, we record a conditional landscape-transfer theorem from hard to small-temperature soft routing and a reduced two-expert Gaussian calculation illustrating local symmetry breaking. Synthetic diagnostics are included only as controlled checks of the boundary-layer predictions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
