几何度量用于MoE专业化：从Fisher信息到早期故障检测
Geometric Metrics for MoE Specialization: From Fisher Information to Early Failure Detection
AI总结：
本文提出基于信息几何的框架，通过Fisher信息度量分析MoE专业化动态，提出FSI和FHS指标，有效预测训练失败并提升模型性能。
摘要：
专家专业化是混合专家（MoE）模型成功的关键，但现有度量（余弦相似度、路由熵）缺乏理论基础且在重参数化下结论不一致。本文提出一个信息几何框架，首次严格刻画MoE专业化动态。核心发现是专家路由分布在概率单纯形上以Fisher信息度量演化，通过黎曼几何进行形式分析。证明标准启发式度量违反参数化不变性（定理1），确立专业化对应测地线流并有量化近似界（定理2），并推导出具有理论阈值依据的故障预测器（定理3）。框架引入两种原理性度量：Fisher专业化指数（FSI）与Fisher异质性分数（FHS）。FSI与下游性能相关性达r=0.91±0.02，FHS在训练10%时预测失败AUC=0.89±0.03，优于基于验证损失的早停法23%，且计算开销低40倍。验证干预协议在FHS>1时恢复率达87%。在语言建模（WikiText-103，C4）、视觉MoE（ImageNet）及扩展研究（8-64专家，125M-2.7B参数）中验证了理论预测。
英文摘要：
Expert specialization is fundamental to Mixture-of-Experts (MoE) model success, yet existing metrics (cosine similarity, routing entropy) lack theoretical grounding and yield inconsistent conclusions under reparameterization. We present an information-geometric framework providing the first rigorous characterization of MoE specialization dynamics. Our key insight is that expert routing distributions evolve on the probability simplex equipped with the Fisher information metric, enabling formal analysis via Riemannian geometry. We prove that standard heuristic metrics violate parameterization invariance (Theorem 1), establish that specialization corresponds to geodesic flow with quantified approximation bounds (Theorem 2), and derive a failure predictor with theoretical threshold justification (Theorem 3). The framework introduces two principled metrics: Fisher Specialization Index (FSI) achieving r=0.91+/-0.02 correlation with downstream performance, and Fisher Heterogeneity Score (FHS) predicting training failure at 10% completion with AUC=0.89+/-0.03 -- outperforming validation-loss-based early stopping by 23% while requiring 40x fewer compute cycles. We validate intervention protocols achieving 87% recovery rate when FHS>1 is detected. Comprehensive experiments across language modeling (WikiText-103, C4), vision MoE (ImageNet), and scaling studies (8-64 experts, 125M-2.7B parameters) validate our theoretical predictions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
