专家混合作为软聚类：双雅可比-PCA谱几何视角
Mixture-of-Experts as Soft Clustering: A Dual Jacobian-PCA Spectral Geometry Perspective
AI总结：
本文从双雅可比-PCA谱几何视角研究MoE，发现其通过软划分函数空间，降低局部曲率并重新分布表示方差，从而提升模型性能。
摘要：
混合专家（MoE）架构广泛用于效率和条件计算，但其对学习函数和表示的几何效应仍不明确。我们通过几何视角研究MoE，将路由解释为软划分成重叠专家局部图表。我们引入双雅可比-PCA谱探针，通过雅可比奇异值谱分析局部函数几何，通过加权PCA分析表示几何。在受控的MLP-MoE设置中，通过精确雅可比计算，我们比较了密集、Top-k和全软路由在匹配容量下的表现。在随机种子下，MoE路由一致减少局部敏感性：专家局部雅可比的主导奇异值更小，谱衰减更快。加权PCA显示，专家局部表示的方差分布在更多主方向上，表明有效秩更高。我们进一步观察到专家雅可比之间低对齐度，表明分解为低重叠专家特定变换。路由锐度调节这些效应：Top-k路由产生更集中、低秩的专家结构，而全软路由产生更宽广、高秩的表示。在3层transformer上进行实验，证实了自然语言的曲率减少，并显示Top-k路由的跨专家对齐度更低。这些发现支持将MoE解释为函数空间的软划分，通过平滑局部曲率并重新分布表示方差，为专家扩展、幻觉减少和集合多样性提供可测试的预测。
英文摘要：
Mixture-of-Experts (MoE) architectures are widely used for efficiency and conditional computation, but their effect on the geometry of learned functions and representations remains poorly understood. We study MoEs through a geometric lens, interpreting routing as soft partitioning into overlapping expert-local charts. We introduce a Dual Jacobian-PCA spectral probe that analyzes local function geometry via Jacobian singular value spectra and representation geometry via weighted PCA of routed hidden states. Using a controlled MLP-MoE setting with exact Jacobian computation, we compare dense, Top-k, and fully soft routing under matched capacity. Across random seeds, MoE routing consistently reduces local sensitivity: expert-local Jacobians show smaller leading singular values and faster spectral decay than dense baselines. Weighted PCA reveals that expert-local representations distribute variance across more principal directions, indicating higher effective rank. We further observe low alignment among expert Jacobians, suggesting decomposition into low-overlap expert-specific transformations. Routing sharpness modulates these effects: Top-k routing yields more concentrated, lower-rank expert structure, while fully soft routing produces broader, higher-rank representations. Experiments on a 3-layer transformer with WikiText confirm curvature reduction on natural language and show lower cross-expert alignment for Top-k routing. These findings support interpreting MoEs as soft partitionings of function space that flatten local curvature while redistributing representation variance, yielding testable predictions for expert scaling, hallucination reduction, and ensemble diversity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
