MoST：通过模态感知混合专家混合语音和文本
MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts
AI总结：
MoST通过模态感知混合专家架构，实现语音和文本的高效融合，首次公开了基于混合专家的语音-文本大语言模型。
摘要：
我们提出了MoST（混合语音和文本），一种新颖的多模态大语言模型，通过我们提出的模态感知混合专家（MAMoE）架构无缝整合语音和文本处理。尽管当前多模态模型通常使用相同的参数处理多种模态表示，忽视其固有的表示差异，我们引入了专门的路由路径，根据输入类型将标记路由到适合该模态的专家。MAMoE通过两个互补的组件同时增强了模态特定的学习和跨模态理解：捕捉领域特定模式的模态特定专家组和促进模态间信息传输的共享专家。基于此架构，我们开发了一条高效的转换流程，通过在ASR和TTS数据集上进行策略性后训练，然后使用精心编纂的语音-文本指令数据集进行微调，以适应预训练的MoE语言模型。该流程的关键特征是仅依赖于完全可访问的开源数据集，以实现强大的性能和数据效率。在ASR、TTS、音频语言建模和口头问答基准测试中的全面评估显示，MoST在参数量相当的现有模型上表现一致更优。我们的消融研究证实，模态特定的路由机制和共享专家设计在所有测试领域中显著促进了性能提升。据我们所知，MoST是首个基于混合专家架构构建的完全开源的语音-文本LLM。
英文摘要：
We present MoST (Mixture of Speech and Text), a novel multimodal large language model that seamlessly integrates speech and text processing through our proposed Modality-Aware Mixture of Experts (MAMoE) architecture. While current multimodal models typically process diverse modality representations with identical parameters, disregarding their inherent representational differences, we introduce specialized routing pathways that direct tokens to modality-appropriate experts based on input type. MAMoE simultaneously enhances modality-specific learning and cross-modal understanding through two complementary components: modality-specific expert groups that capture domain-specific patterns and shared experts that facilitate information transfer between modalities. Building on this architecture, we develop an efficient transformation pipeline that adapts the pretrained MoE language model through strategic post-training on ASR and TTS datasets, followed by fine-tuning with a carefully curated speech-text instruction dataset. A key feature of this pipeline is that it relies exclusively on fully accessible, open-source datasets to achieve strong performance and data efficiency. Comprehensive evaluations across ASR, TTS, audio language modeling, and spoken question answering benchmarks show that MoST consistently outperforms existing models of comparable parameter counts. Our ablation studies confirm that the modality-specific routing mechanism and shared experts design significantly contribute to performance gains across all tested domains. To our knowledge, MoST represents the first fully open-source speech-text LLM built on a Mixture of Experts architecture. \footnote{We release MoST model, training code, inference code, and training data at https://github.com/NUS-HPC-AI-Lab/MoST
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
