Linear-MoE：线性序列建模与混合专家模型的结合
Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts
AI总结：
本文提出Linear-MoE生产级系统，整合线性序列建模（LSM）与混合专家模型（MoE），通过统一建模框架和序列并行等训练技术提升效率，在A0.3B-2B及A1B-7B模型系列上验证了其高效性与性能竞争力。
摘要：
线性注意力、状态空间模型和线性循环神经网络等线性序列建模（LSM）方法，以及混合专家模型（MoE）是近年来出现的重要架构改进。本文介绍Linear-MoE，这是一个用于建模和训练集成LSM与MoE的大规模模型的生产级系统。Linear-MoE利用LSM模块的线性复杂度序列建模优势和MoE层的稀疏激活特性，旨在通过高效训练实现高性能。Linear-MoE系统包含：1）建模子系统，提供支持所有LSM实例的统一框架；2）训练子系统，通过整合多种先进并行技术（尤其是为Linear-MoE模型设计的序列并行）促进高效训练。此外，我们探索将Linear-MoE层与标准Transformer-MoE层结合的混合模型，以进一步提升模型灵活性和性能。对A0.3B-2B和A1B-7B两个模型系列的评估表明，Linear-MoE在保持各基准测试竞争力的同时实现了效率提升，展现其作为下一代基础模型架构的潜力。代码：https://github.com/OpenSparseLLMs/Linear-MoE。
英文摘要：
Linear Sequence Modeling (LSM) like linear attention, state space models and linear RNNs, and Mixture-of-Experts (MoE) have recently emerged as significant architectural improvements. In this paper, we introduce Linear-MoE, a production-level system for modeling and training large-scale models that integrate LSM with MoE. Linear-MoE leverages the advantages of both LSM modules for linear-complexity sequence modeling and MoE layers for sparsely activation, aiming to offer high performance with efficient training. The Linear-MoE system comprises: 1) Modeling subsystem, which provides a unified framework supporting all instances of LSM. and 2) Training subsystem, which facilitates efficient training by incorporating various advanced parallelism technologies, particularly Sequence Parallelism designed for Linear-MoE models. Additionally, we explore hybrid models that combine Linear-MoE layers with standard Transformer-MoE layers with its Sequence Parallelism to further enhance model flexibility and performance. Evaluations on two model series, A0.3B-2B and A1B-7B, demonstrate Linear-MoE achieves efficiency gains while maintaining competitive performance on various benchmarks, showcasing its potential as a next-generation foundational model architecture. Code: https://github.com/OpenSparseLLMs/Linear-MoE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
