MoE-Mamba：结合专家混合的高效选择性状态空间模型
MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
AI总结：
本文提出 MoE-Mamba，将专家混合与状态空间模型 Mamba 结合，在减少 2.35 倍训练步数的情况下达到与 Mamba 相同性能，并保持推理优势。
摘要：
状态空间模型（SSMs）已成为序列建模领域的有力竞争者，挑战着 Transformer 的主导地位。与此同时，专家混合（Mixture of Experts, MoE）显著提升了基于 Transformer 的大语言模型，包括近期最先进的开放模型。我们提出，要释放 SSMs 的扩展潜力，应将其与 MoE 相结合。我们在 Mamba 上展示了这一思路，Mamba 是近期一种基于 SSM 的模型，取得了卓越的性能。我们的模型 MoE-Mamba 同时优于 Mamba 和基线 Transformer-MoE。特别是，MoE-Mamba 以少 2.35 倍的训练步数达到了与 Mamba 相同的性能，同时保持了 Mamba 相对于 Transformer 的推理性能优势。
英文摘要：
State Space Models (SSMs) have become serious contenders in the field of sequential modeling, challenging the dominance of Transformers. At the same time, Mixture of Experts (MoE) has significantly improved Transformer-based Large Language Models, including recent state-of-the-art open models. We propose that to unlock the potential of SSMs for scaling, they should be combined with MoE. We showcase this on Mamba, a recent SSM-based model that achieves remarkable performance. Our model, MoE-Mamba, outperforms both Mamba and baseline Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba in $2.35\times$ fewer training steps while preserving the inference performance gains of Mamba against Transformer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
