arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.14039cs.CLcs.LG
收藏

保持几何结构的专家聚合用于混合专家嵌入模型

Geometry-Preserving Aggregation for Mixture-of-Experts Embedding Models

Department of Electrical Engineering(电气工程系);Sharif University of Technology(谢里夫理工学院)

Sajjad Kachuee, Mohammad Sharifkhani

AI总结:

SBA通过分离径向和角向成分,保持超球面结构,提升混合专家嵌入模型的几何一致性与性能。

摘要:

混合专家(MoE)嵌入模型通过加权线性求和结合专家输出,隐含假设嵌入空间中存在线性子空间结构。研究表明,这一假设与专家表示的几何结构不一致。对现代MoE嵌入模型的几何分析显示,专家输出位于共享的超球面流形上,其特征是紧密集中的范数和显著的角分离。在此几何结构下,线性聚合会向流形内部坍缩,扭曲向量的大小和方向,降低嵌入的可比性。为了解决这一不一致,引入了球面重心聚合(SBA)作为保持几何结构的聚合算子,该算子分离径向和角向成分,以保持超球面结构,同时完全兼容现有路由机制。在大规模文本嵌入基准(MTEB)中选择的任务上,包括语义相似性、聚类和重复问题检测,实验表明在相同的训练成本下,SBA表现出一致的性能提升,并且完全稳定。额外的几何分析证实,SBA防止了聚合引起的坍缩并保持超球面一致性,突显了在MoE嵌入架构中考虑几何的聚合的重要性。

英文摘要:

Mixture-of-Experts (MoE) embedding models combine expert outputs using weighted linear summation, implicitly assuming a linear subspace structure in the embedding space. This assumption is shown to be inconsistent with the geometry of expert representations. Geometric analysis of a modern MoE embedding model reveals that expert outputs lie on a shared hyperspherical manifold characterized by tightly concentrated norms and substantial angular separation. Under this geometry, linear aggregation induces inward collapse toward the manifold interior, distorting vector magnitude and direction and reducing embedding comparability. To address this inconsistency, Spherical Barycentric Aggregation (SBA) is introduced as a geometry-preserving aggregation operator that separates radial and angular components to maintain hyperspherical structure while remaining fully compatible with existing routing mechanisms. Experiments on selected tasks from the Massive Text Embedding Benchmark (MTEB), including semantic similarity, clustering, and duplicate question detection, demonstrate consistent performance improvements with identical training cost and full stability. Additional geometric analyses confirm that SBA prevents aggregation-induced collapse and preserves hyperspherical consistency, highlighting the importance of geometry-aware aggregation in MoE embedding architectures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑