arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.03204cs.LG
收藏

稀疏性是组合深度:通过热带几何量化MoE表达性

Sparsity is Combinatorial Depth: Quantifying MoE Expressivity via Tropical Geometry

Ye Su, Huayi Tang, Zixuan Gong, Yong Liu

AI总结:

本文通过热带几何分析MoE架构,揭示稀疏性与组合深度的关系,证明MoE在低维数据中具有组合鲁棒性,提供理论依据支持条件计算的拓扑优势。

摘要:

尽管混合专家(MoE)架构定义了当前最先进的水平,但其理论成功往往归因于启发式的效率,而不是几何表达性。在本工作中,我们首次通过热带几何的视角分析MoE,建立了Top-k路由机制与第k个基本对称热带多项式在代数上的同构性。这种同构性将输入空间划分为超简单形的正则扇形,揭示稀疏性是组合深度,通过二项式系数$\binom{N}{k}$扩展几何容量。超越环境界限,我们引入了在流形假说下的有效容量概念。我们证明,尽管密集网络在低维数据上遭受容量崩溃,MoE架构表现出组合鲁棒性,通过路由锥的横截性维持高表达性。在本研究中,我们的框架将超简单形的离散几何与神经函数的连续几何统一起来,为条件计算的拓扑优势提供了严格的理论依据。

英文摘要:

While Mixture-of-Experts (MoE) architectures define the state-of-the-art, their theoretical success is often attributed to heuristic efficiency rather than geometric expressivity. In this work, we present the first analysis of MoE through the lens of tropical geometry, establishing that the Top-$k$ routing mechanism is algebraically isomorphic to the $k$-th elementary symmetric tropical polynomial. This isomorphism partitions the input space into the Normal Fan of a Hypersimplex, revealing that \textbf{sparsity is combinatorial depth} which scales geometric capacity by the binomial coefficient $\binom{N}{k}$. Moving beyond ambient bounds, we introduce the concept of \textit{Effective Capacity} under the Manifold Hypothesis. We prove that while dense networks suffer from capacity collapse on low-dimensional data, MoE architectures exhibit \textit{Combinatorial Resilience}, maintaining high expressivity via the transversality of routing cones. In this study, our framework unifies the discrete geometry of the Hypersimplex with the continuous geometry of neural functions, offering a rigorous theoretical justification for the topological supremacy of conditional computation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑