arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.12399cs.LGcs.AIcs.CL
收藏

变废为宝:修正 MoE 的 Top-k 路由

Turn Waste into Worth: Rectifying Top-$k$ Router of MoE

School of Computer Science, Fudan University(复旦大学计算机科学技术学院);Shanghai AI Laboratory(上海人工智能实验室)

Zhiyuan Zeng, Qipeng Guo, Zhaoye Fei, Zhangyue Yin, Yunhua Zhou, Linyang Li, Tianxiang Sun, Hang Yan, Dahua Lin, Xipeng Qiu

AI总结:

本文提出 Rectify-Router,通过 Intra-GPU Rectification 和 Fill-in Rectification 分别解决 MoE 中 token 丢弃和填充问题,最终比 vanilla top-1 路由准确率提升 4.7%。

摘要:

稀疏专家混合(MoE)模型因其计算效率而在训练大语言模型时广受欢迎。然而,常用的 top-k 路由机制由于路由不均衡而存在冗余计算和内存开销问题。一些专家处于溢出状态,超出容量的 token 会被丢弃;而另一些专家则处于空闲状态,被填充零值,从而对模型性能产生负面影响。为了解决 token 丢弃和填充问题,我们提出了 Rectify-Router,包括 Intra-GPU Rectification 和 Fill-in Rectification。Intra-GPU Rectification 处理被丢弃的 token,将其高效地路由到其所在 GPU 内的专家,以避免跨 GPU 通信。Fill-in Rectification 通过用具有高路由分数的 token 替换填充 token 来解决填充问题。我们的实验结果表明,Intra-GPU Rectification 和 Fill-in Rectification 分别有效地处理了 token 丢弃和填充问题。此外,二者的结合实现了更优的性能,比 vanilla top-1 路由器的准确率高出 4.7%。

英文摘要:

Sparse Mixture of Experts (MoE) models are popular for training large language models due to their computational efficiency. However, the commonly used top-$k$ routing mechanism suffers from redundancy computation and memory costs due to the unbalanced routing. Some experts are overflow, where the exceeding tokens are dropped. While some experts are vacant, which are padded with zeros, negatively impacting model performance. To address the dropped tokens and padding, we propose the Rectify-Router, comprising the Intra-GPU Rectification and the Fill-in Rectification. The Intra-GPU Rectification handles dropped tokens, efficiently routing them to experts within the GPU where they are located to avoid inter-GPU communication. The Fill-in Rectification addresses padding by replacing padding tokens with the tokens that have high routing scores. Our experimental results demonstrate that the Intra-GPU Rectification and the Fill-in Rectification effectively handle dropped tokens and padding, respectively. Furthermore, the combination of them achieves superior performance, surpassing the accuracy of the vanilla top-1 router by 4.7%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑