SMAR：保持语言能力的基于 MoE 的多模态大语言模型软模态感知路由策略
SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities
AI总结：
针对多模态 MoE 模型适配预训练模型时语言能力退化或训练成本高的问题，提出利用 KL 散度控制跨模态路由分布的正则化技术 SMAR，仅需 2.5% 纯文本即能以 86.6% 保留率维持语言能力并保持多模态性能。
摘要：
专家混合架构已成为扩展大语言模型的关键方法，将其扩展至多模态任务也受到越来越多的关注。构建多模态 MoE 模型的现有方法，要么产生高昂的训练成本，要么在适配预训练模型时导致语言能力退化。为解决此问题，我们提出了软模态感知路由，这是一种新颖的正则化技术，利用 Kullback Leibler 散度控制跨模态的路由概率分布，在不修改模型架构或严重依赖文本数据的前提下促进专家专业化。视觉指令微调实验表明，SMAR 仅使用 2.5% 的纯文本，便以 86.6% 的保留率维持了语言能力，在保持强劲多模态性能的同时超越了基线。我们的方法为平衡多模态 MoE 模型中的模态差异化和语言能力提供了实用且高效的解决方案。
英文摘要：
Mixture of Experts (MoE) architectures have become a key approach for scaling large language models, with growing interest in extending them to multimodal tasks. Existing methods to build multimodal MoE models either incur high training costs or suffer from degraded language capabilities when adapting pretrained models. To address this, we propose Soft ModalityAware Routing (SMAR), a novel regularization technique that uses Kullback Leibler divergence to control routing probability distributions across modalities, encouraging expert specialization without modifying model architecture or heavily relying on textual data. Experiments on visual instruction tuning show that SMAR preserves language ability at 86.6% retention with only 2.5% pure text, outperforming baselines while maintaining strong multimodal performance. Our approach offers a practical and efficient solution to balance modality differentiation and language capabilities in multimodal MoE models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
