MoE专业化中的几何不对称性：功能去相关与表征重叠
Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap
AI总结：
研究揭示MoE专业化中功能去相关与表征重叠共存的几何结构，通过实验表明路由稀疏性影响功能分离与子空间分歧。
摘要：
混合专家（MoE）架构通过稀疏路由实现可扩展容量，但专家专业化几何结构仍不明确。本文提出统一的雅可比-PCA-格拉索曼框架，分析MoE层在函数空间和表征空间中的结构。在预训练的MoE Transformer（Mistral、Qwen）中发现一致的结构不对称：专家表现出强功能去相关（跨专家雅可比对齐低，接近零）而路由表征占据 distinct 但部分重叠的子空间。这表明功能去相关与表征重叠共存而非重合。受控路由实验进一步表明，路由稀疏性是塑造此几何结构的关键因素：top-k路由导致更尖锐的功能分离和更大的子空间分歧，而完全软路由产生更纠缠的专家结构。这些结果表明，MoE层可视为在共享表征流形上局部去相关操作符在重叠子流形上的实现，并提供研究现代Transformer架构条件计算的通用诊断框架。
英文摘要：
Mixture-of-Experts (MoE) architectures achieve scalable capacity through sparse routing, yet the geometric structure of expert specialization remains poorly understood. We introduce a unified Jacobian-PCA-Grassmann framework for analyzing MoE layers in both function space and representation space. Across pretrained MoE Transformers (Mistral, Qwen), we find a consistent structural asymmetry: experts exhibit strong functional decorrelation (consistently low, near-zero cross-expert Jacobian alignment) while their routed representations occupy distinct but partially overlapping subspaces. This indicates that functional decorrelation and representation overlap coexist rather than coincide in MoE specialization. Controlled routing experiments further indicate that routing sparsity appears to be a key factor shaping this geometry: top-k routing induces sharper functional separation and larger subspace divergence, whereas fully soft routing yields more entangled expert structure. Together, these results suggest a geometric interpretation in which MoE layers may be viewed as implementing locally decorrelated operators over overlapping submanifolds on a shared representation manifold, and provide a general diagnostic framework for studying conditional computation in modern Transformer architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
