Marco-MoE：开放多语言稀疏专家混合语言模型的高效再利用
Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
AI总结：
Marco-MoE通过极稀疏设计和密集模型再利用，在5T tokens上高效预训练，实现多语言和英语基准的领先性能，同时支持可扩展的语言扩展。
摘要：
我们提出了Marco-MoE，一套完全开放的多语言稀疏混合专家（MoE）模型。Marco-MoE具有高度稀疏的设计，每个输入token仅激活约5%的总参数。这种极端稀疏性结合来自密集模型的再利用，使在5T tokens上高效预训练成为可能。我们的模型在英语和多语言基准上超越了同样规模的竞争者，实现了最佳的性能到计算量比。我们进一步对这些模型进行后训练，创建Marco-MoE-Instruct变体，其性能超越了拥有3-14倍更多激活参数的竞争模型。我们的分析表明，Marco-MoE学习了在相关语言中共享的结构化专家激活模式，同时保持对语言上孤立的语言的高度专业化利用。我们进一步展示了Marco-MoE可以在不干扰密集模型典型干扰的情况下实现可扩展的语言扩展。为了支持社区，我们公开了完整的训练数据集、配方和模型权重。
英文摘要：
We present Marco-MoE, a suite of fully open multilingual sparse Mixture-of-Experts (MoE) models. Marco-MoE features a highly sparse design in which only around 5\% of the total parameters are activated per input token. This extreme sparsity, combined with upcycling from dense models, enables efficient pre-training on 5T tokens. Our models surpass similarly-sized competitors on English and multilingual benchmarks, achieving a best-in-class performance-to-compute ratio. We further post-train these models to create Marco-MoE-\textsc{Instruct} variants, which surpass the performance of competing models possessing $3$--$14\times$ more activated parameters. Our analysis reveals that Marco-MoE learns structured expert activation patterns shared across related languages, while maintaining highly specialized utilization for linguistically isolated ones. We further show that Marco-MoE allows for scalable language expansion without the interference typical of dense models. To support the community, we disclose our full training datasets, recipes, and model weights.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
