MaskMoE:通过混合专家模型中的路由掩码提升Token级学习
MaskMoE: Boosting Token-Level Learning via Routing Mask in Mixture-of-Experts
AI总结:
本文提出MaskMoE方法,通过在混合专家模型中引入路由掩码技术,解决了动态路由的欠拟合与固定路由的表示多样性受限问题,实现了更全面的Token级训练,并在困惑度和下游任务上超越基线模型。
摘要:
扩大模型规模能增强其能力,但会显著增加计算复杂度。混合专家模型通过允许模型规模扩大而不大幅增加训练或推理成本来解决该问题。在MoE中,有一个名为路由器的重要模块,用于将每个token分发给专家。目前主流的路由方法包括动态路由和固定路由。尽管取得了可喜的结果,MoE模型仍面临一些挑战。首先,对于动态路由方法,训练token分散在多个专家中可能导致欠拟合,特别是对于不频繁的token。此外,尽管固定路由方法可以缓解该问题,但它们在表示多样性上做出了妥协。在本文中,我们提出了MaskMoE,一种旨在通过在混合专家模型中采用路由掩码技术来增强token级学习的方法。MaskMoE能够在实现更全面训练的同时保持表示多样性。实验结果表明,我们的方法在困惑度(PPL)和下游任务性能方面均优于以往占主导地位的混合专家模型。
英文摘要:
Scaling the size of a model enhances its capabilities but significantly increases computation complexity. Mixture-of-Experts models (MoE) address the issue by allowing model size to scale up without substantially increasing training or inference costs. In MoE, there is an important module called the router, which is used to distribute each token to the experts. Currently, the mainstream routing methods include dynamic routing and fixed routing. Despite their promising results, MoE models encounter several challenges. Primarily, for dynamic routing methods, the dispersion of training tokens across multiple experts can lead to underfitting, particularly for infrequent tokens. Additionally, though fixed routing methods can mitigate that issue, they compromise on the diversity of representations. In this paper, we propose \textbf{MaskMoE}, a method designed to enhance token-level learning by employing a routing \textbf{mask}ing technique within the \textbf{M}ixture-\textbf{o}f-\textbf{E}xperts model. MaskMoE is capable of maintaining representation diversity while achieving more comprehensive training. Experimental results demonstrate that our method outperforms previous dominant Mixture-of-Experts models in terms of both perplexity (PPL) and downstream task performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。