arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.39442cs.LG
收藏

混合专家模型中的Softmax路由动力学:崩溃与恢复

Softmax Routing Dynamics in Mixture-of-Experts Models: Collapse and Recovery

The Chinese University of Hong Kong(香港中文大学);Beijing Normal-Hong Kong Baptist University(北京师范大学-香港浸会大学联合国际学院);Guangzhou Nanfang College(广州南方学院)

Shengze Xu, Maolin Deng, Tieyong Zeng

AI总结:

该研究分析混合专家模型中Softmax路由的概率集中崩溃问题,通过理论推导揭示恢复机制,并在Transformer-MoE训练中验证类条件路由器梯度对比对恢复的影响。

摘要:

混合专家模型中的Softmax路由会将概率集中在部分专家上,抑制对其他专家的访问。我们研究这种集中的形成方式、持续原因以及被抑制概率的恢复机制。对于固定专家,平稳损失边界可识别支撑最优路由输出的面。在专家输出可自由训练的共同目标平方损失模型中,有限时间范围内慢适应跟踪固定专家动力学,而每个固定正适应率都会产生内部路由极限。精确的法向速度分解将互补专家协方差与相对梯度优势分离。在恢复路径上沿初始质量均匀的正双侧幂界下,所得系数的消失阶给出对数或幂律恢复时间,构造场的积分重现这些阶。精确的离散对数几率恒等式将神经更新与阈值穿越关联。在配对Transformer-MoE训练中,移除类条件路由器梯度对比会延迟确认恢复,并减少固定token上的累积对数几率,无论是否存在容量约束。

英文摘要:

Softmax routing in mixture-of-experts models can concentrate probability on a subset of experts, suppressing access to the others. We study how this concentration forms, why it persists, and how suppressed probabilities recover. For fixed experts, a stationary-loss barrier identifies the face supporting the optimal routed output. In a common-target square-loss model with freely trainable expert outputs, slow adaptation tracks fixed-expert dynamics on finite horizons, while every fixed positive adaptation rate yields an interior routing limit. An exact normal-velocity decomposition separates complementary-expert covariance from relative gradient advantage. Under positive two-sided power bounds along recovery paths, uniform over initial masses, the resulting coefficient's order of vanishing gives logarithmic or power-law recovery times. Integrations of constructed fields reproduce these orders. An exact discrete log-odds identity relates neural updates to threshold crossing. In paired Transformer-MoE training, removing a class-conditional router-gradient contrast delays confirmed recovery and reduces accumulated log-odds on fixed tokens, both with and without capacity constraints.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑