Mixture-of-Mamba：通过模态感知稀疏性增强多模态状态空间模型
Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity
AI总结：
提出 Mixture-of-Mamba 架构，通过对 Mamba 块进行模态特定参数化引入模态感知稀疏性，在保持计算效率的同时显著降低多模态预训练成本并达到等效性能。
摘要：
状态空间模型（SSMs）已成为 Transformers 的高效替代方案，但它们无法利用特定模态特征，限制了其在多模态预训练中的性能。在此，我们提出 Mixture-of-Mamba，一种新颖的 SSM 架构，通过对 Mamba 块进行特定模态参数化来引入模态感知稀疏性。基于 Mixture-of-Transformers，我们将模态感知稀疏性的优势扩展到 SSMs，同时保持其计算效率。我们在三种多模态预训练设置下评估 Mixture-of-Mamba：Transfusion（交错文本与连续图像 token 结合扩散损失）、Chameleon（交错文本与离散图像 token），以及一个结合语音的扩展三模态框架。Mixture-of-Mamba 始终在更早的训练步数达到相同的损失值，且计算成本显著降低。在 Transfusion 设置中，Mixture-of-Mamba 在 1.4B 规模下仅使用 34.76% 的训练 FLOPs 即达到等效的图像损失。在 Chameleon 设置中，Mixture-of-Mamba 在 1.4B 规模下仅使用 42.50% 的 FLOPs 达到相似的图像损失，仅使用 65.40% 的 FLOPs 达到相似的文本损失。在三模态设置中，MoM 在 1.4B 规模下仅用 24.80% 的 FLOPs 匹配了语音损失。我们的消融研究强调了解耦投影组件的协同效应，其中联合解耦比单独修改产生更大的增益。这些结果确立了模态感知稀疏性作为一种通用且有效的设计原则，将其影响从 Transformers 扩展到 SSMs，并在多模态预训练中树立了新基准。
英文摘要：
State Space Models (SSMs) have emerged as efficient alternatives to Transformers for sequential modeling, but their inability to leverage modality-specific features limits their performance in multi-modal pretraining. Here, we propose Mixture-of-Mamba, a novel SSM architecture that introduces modality-aware sparsity through modality-specific parameterization of the Mamba block. Building on Mixture-of-Transformers (W. Liang et al. arXiv:2411.04996; 2024), we extend the benefits of modality-aware sparsity to SSMs while preserving their computational efficiency. We evaluate Mixture-of-Mamba across three multi-modal pretraining settings: Transfusion (interleaved text and continuous image tokens with diffusion loss), Chameleon (interleaved text and discrete image tokens), and an extended three-modality framework incorporating speech. Mixture-of-Mamba consistently reaches the same loss values at earlier training steps with significantly reduced computational costs. In the Transfusion setting, Mixture-of-Mamba achieves equivalent image loss using only 34.76% of the training FLOPs at the 1.4B scale. In the Chameleon setting, Mixture-of-Mamba reaches similar image loss with just 42.50% of the FLOPs at the 1.4B scale, and similar text loss with just 65.40% of the FLOPs. In the three-modality setting, MoM matches speech loss at 24.80% of the FLOPs at the 1.4B scale. Our ablation study highlights the synergistic effects of decoupling projection components, where joint decoupling yields greater gains than individual modifications. These results establish modality-aware sparsity as a versatile and effective design principle, extending its impact from Transformers to SSMs and setting new benchmarks in multi-modal pretraining. Our code can be accessed at https://github.com/Weixin-Liang/Mixture-of-Mamba
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
