U2++ MoE：在几乎不影响RTF的情况下将参数量扩展4.7倍
U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF
AI总结：
本文提出U2++ MoE，通过将ASR模型中的FFN层简单替换为MoE层，以Dense-225M级RTF实现Dense-1B级WER，并支持流式与非流式统一解码。
摘要：
规模扩展为自然语言处理开辟了新的前沿，但代价高昂。作为应对，通过在训练和推理中仅激活部分参数，专家混合（Mixture-of-Experts, MoE）被提出作为一条通往更大、更强语言模型的节能路径，这种向新一代基础模型转变的趋势正在加速，尤其是在自动语音识别（Automatic Speech Recognition, ASR）领域。近期将MoE引入ASR模型的工作采用了复杂设计，例如通过辅助嵌入网络路由帧、提升专家的多语言能力，以及利用专门的辅助损失进行专家负载均衡或特定语言处理。我们发现，精细的设计并非必要，而将所有前馈网络（Feed-Forward Network, FFN）层直接替换为MoE层这一极其简单的替代方案，足以胜任ASR任务。具体而言，我们在一个大规模内部数据集（160k小时）上对所提模型进行了基准测试，结果表明，我们可以将基线Conformer（Dense-225M）扩展为其MoE对应模型（MoE-1B），在保持Dense-225M级实时率（Real Time Factor, RTF）的同时，达到Dense-1B级的词错误率（Word Error Rate, WER）。此外，通过应用带有双向注意力解码器的统一两遍框架（U2++），我们在单个基于MoE的模型中实现了流式和非流式解码模式，我们称之为U2++ MoE。我们希望本研究能够促进在不牺牲部署效率的前提下扩展语音基础模型的研究。
英文摘要：
Scale has opened new frontiers in natural language processing, but at a high cost. In response, by learning to only activate a subset of parameters in training and inference, Mixture-of-Experts (MoE) have been proposed as an energy efficient path to even larger and more capable language models and this shift towards a new generation of foundation models is gaining momentum, particularly within the field of Automatic Speech Recognition (ASR). Recent works that incorporating MoE into ASR models have complex designs such as routing frames via supplementary embedding network, improving multilingual ability for the experts, and utilizing dedicated auxiliary losses for either expert load balancing or specific language handling. We found that delicate designs are not necessary, while an embarrassingly simple substitution of MoE layers for all Feed-Forward Network (FFN) layers is competent for the ASR task. To be more specific, we benchmark our proposed model on a large scale inner-source dataset (160k hours), the results show that we can scale our baseline Conformer (Dense-225M) to its MoE counterparts (MoE-1B) and achieve Dense-1B level Word Error Rate (WER) while maintaining a Dense-225M level Real Time Factor (RTF). Furthermore, by applying Unified 2-pass framework with bidirectional attention decoders (U2++), we achieve the streaming and non-streaming decoding modes in a single MoE based model, which we call U2++ MoE. We hope that our study can facilitate the research on scaling speech foundation models without sacrificing deployment efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
