arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.03577cs.LG
收藏

变分推断、熵与正交性:混合专家模型的统一理论

Variational Inference, Entropy, and Orthogonality: A Unified Theory of Mixture-of-Experts

University of Chinese Academy of Sciences, Beijing, China(中国科学院大学)

Ye Su, Yong Liu

AI总结:

本文从贝叶斯和信息论视角构建混合专家模型的统一理论框架,推导路由机制为最优稀疏后验近似,并证明正交性可缩小全局最优与贪心近似间的差距。

摘要:

混合专家模型使大型语言模型能够高效扩展,因为它们只为每个输入激活专家子集。其核心机制,即Top-k路由和辅助负载平衡,仍然是启发式的,缺乏统一的理论基础。为此,我们构建了首个统一的理论框架,从贝叶斯视角严格推导这些实践作为最优稀疏后验近似和先验正则化的结果,同时将其框架为从信息论视角最小化路由模糊性和最大化通道容量的机制。我们还指出了路由的内在组合复杂性,将其定义为NP难的稀疏子集选择问题。我们严格证明了“一致性障碍”的存在;当专家表示具有高互相关性时,贪心路由策略理论上无法恢复最优专家子集。重要的是,我们正式验证了在专家特征空间中施加几何正交性足以缩小NP难全局最优与多项式时间贪心近似之间的差距。我们的比较分析证实正交性正则化是大规模模型的最佳工程放松。我们的工作为深入理解和新型MoE设计提供了必要的理论支持和技术保证。

英文摘要:

Mixture-of-Experts models enable large language models to scale efficiently, as they only activate a subset of experts for each input. Their core mechanisms, Top-k routing and auxiliary load balancing, remain heuristic, however, lacking a cohesive theoretical underpinning to support them. To this end, we build the first unified theoretical framework that rigorously derives these practices as optimal sparse posterior approximation and prior regularization from a Bayesian perspective, while simultaneously framing them as mechanisms to minimize routing ambiguity and maximize channel capacity from an information-theoretic perspective. We also pinpoint the inherent combinatorial hardness of routing, defining it as the NP-hard sparse subset selection problem. We rigorously prove the existence of a "Coherence Barrier"; when expert representations exhibit high mutual coherence, greedy routing strategies theoretically fail to recover the optimal expert subset. Importantly, we formally verify that imposing geometric orthogonality in the expert feature space is sufficient to narrow the divide between the NP-hard global optimum and polynomial-time greedy approximation. Our comparative analyses confirm orthogonality regularization as the optimal engineering relaxation for large-scale models. Our work offers essential theoretical support and technical assurance for a deeper understanding and novel designs of MoE.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑