arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.12397cs.LGcs.AIcs.CL
收藏

重新设计混合专家模型的路由器:基于流形幂迭代

Redesign Mixture-of-Experts Routers with Manifold Power Iteration

Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Large Language Model Department, Tencent(腾讯大型语言模型部门)

Songhao Wu, Ang Lv, Ruobing Xie, Yankai Lin

AI总结:

提出将路由器行与专家矩阵主奇异方向对齐,并基于流形幂迭代(MPI)重新设计路由器,通过“幂迭代-收缩”范式实现对齐,理论证明收敛性,实验验证1B至11B参数规模下模型效果提升。

摘要:

路由器是混合专家模型的核心组件。作为专家代理,路由器矩阵的行计算与MoE输入的相似度,以确定激活哪些专家子集。理想情况下,每个路由器行被设计为将专家矩阵编码到该代表性向量中,使得其与token的点积能更好地反映token-专家亲和性。然而,目前没有设计原则来强制这种压缩。在本文中,我们提出将每个路由器行与相关专家的主奇异方向对齐,因为该方向提供了矩阵最具表现力的数学描述。基于这一原则,我们提出了一种基于流形幂迭代(MPI)的路由器重新设计。具体来说,它引入了一种“幂迭代-收缩”范式,其中对路由器权重执行幂迭代步骤,然后进行收缩以施加范数约束,确保效率和稳定性。理论上,我们证明MPI驱动路由器行收敛到相关专家的主奇异方向。实验上,我们在1B到11B参数规模的MoE模型上进行预训练,证实这种对齐有助于更有效的MoE模型。

英文摘要:

Router is the cornerstone component to the Mixture-of-Experts models. Serving as expert proxies, the rows of the router matrix compute their similarity to the MoE inputs to determine which subset of experts is activated. Ideally, each router row is designed to encode the expert matrix into this representative vector, such that its dot-product with token can better reflect token-expert affinity. However, there exists no design principles to enforce this condensation. In this paper, we propose to align each router row with the principal singular direction of the associated expert, as this direction provides the most expressive mathematical description of a matrix. Based on this principle, we propose a router redesign with Manifold Power Iteration (MPI). Specifically, it introduces a "Power-then-Retract" paradigm, where a power iteration step is performed on the router weights, followed by a retraction to impose a norm constraint to ensure both efficiency and stability. Theoretically, we show that MPI drives router rows to converge toward the principal singular directions of associated experts. Empirically, we pretrain MoE model across scales from 1B to 11B parameters to confirm that this alignment facilitates more effective MoE models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑