SpeechMoE：扩展到具有动态路由的大型声学模型混合专家模型
SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts
摘要：
近年来，基于Transformer的混合专家算法（Mixture of Experts，MoE）在许多领域都取得了可喜的成果。这在很大程度上是由于该架构的以下优点：首先，基于MoE的Transformer可以增加模型容量，而不会增加训练和推理时的计算成本。此外，基于MoE的Transformer是一种动态网络，可以适应实际应用中输入实例复杂性的变化。在这项工作中，我们探讨了基于MoE的语音识别模型，命名为Speech MoE。为了进一步控制路由器激活的稀疏性，提高门值的多样性，我们分别提出了稀疏性L1损失和平均重要性损失。此外，一个新的路由器架构中使用的语音MoE，它可以同时利用信息从一个共享的嵌入式网络和不同的MoE层的分层表示。实验结果表明，与传统静态网络相比，SpeechMoE在计算成本相当的情况下，可以获得更低的字符出错率（CER），在四个评估数据集上的CER相对改善为7.0%-23.0%。
英文摘要：
Recently, Mixture of Experts (MoE) based Transformer has shown promising results in many domains. This is largely due to the following advantages of this architecture: firstly, MoE based Transformer can increase model capacity without computational cost increasing both at training and inference time. Besides, MoE based Transformer is a dynamic network which can adapt to the varying complexity of input instances in realworld applications. In this work, we explore the MoE based model for speech recognition, named SpeechMoE. To further control the sparsity of router activation and improve the diversity of gate values, we propose a sparsity L1 loss and a mean importance loss respectively. In addition, a new router architecture is used in SpeechMoE which can simultaneously utilize the information from a shared embedding network and the hierarchical representation of different MoE layers. Experimental results show that SpeechMoE can achieve lower character error rate (CER) with comparable computation cost than traditional static networks, providing 7.0%-23.0% relative CER improvements on four evaluation datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
