Lory：用于自回归语言模型预训练的全可微分混合专家模型
Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training
AI总结：
针对MoE模型路由网络不可微的问题，提出全可微分MoE架构Lory，通过因果分段路由和相似度批处理技术实现自回归语言模型预训练，性能优于同参数量稠密模型且具备领域专业化能力。
摘要：
混合专家（Mixture-of-Experts, MoE）模型可实现高效的模型规模扩展；然而，训练路由网络会带来优化不可微的离散目标的挑战。近期，Muqeeth等人（2023）提出了一种全可微分MoE架构SMEAR，它在参数空间中软融合专家，但其有效性仅在分类任务的下游微调中得到验证。本文提出Lory，这是首个将此类架构扩展到自回归语言模型预训练的方法。Lory引入了两项关键技术：（1）因果分段路由策略，在保留语言模型自回归特性的同时，实现高效的专家融合操作；（2）基于相似度的数据批处理方法，通过在训练实例中分组相似文档来促进专家专业化。我们在150B token上从头预训练了一系列Lory模型，最多包含32个专家，总参数量达30B（激活参数量为1.5B）。实验结果表明，在困惑度（提升13.9%）和多种下游任务（提升1.5%-11.1%）上，Lory相比参数量匹配的稠密模型均有显著性能提升。尽管采用分段级路由，Lory模型与采用token级路由的最先进MoE模型相比仍具备竞争力。我们进一步证明，Lory中训练得到的专家无需监督即可捕捉领域层面的专业化特性。本工作凸显了全可微分MoE架构在语言模型预训练中的潜力，并呼吁未来在该领域开展更多研究。
英文摘要：
Mixture-of-experts (MoE) models facilitate efficient scaling; however, training the router network introduces the challenge of optimizing a non-differentiable, discrete objective. Recently, a fully-differentiable MoE architecture, SMEAR, was proposed (Muqeeth et al., 2023), which softly merges experts in the parameter space; nevertheless, its effectiveness was only demonstrated in downstream fine-tuning on classification tasks. In this paper, we present Lory, the first approach that scales such architectures to autoregressive language model pre-training. Lory introduces two key techniques: (1) a causal segment routing strategy that achieves high efficiency for expert merging operations while preserving the autoregressive nature of language models; (2) a similarity-based data batching method that encourages expert specialization by grouping similar documents in training instances. We pre-train a series of Lory models on 150B tokens from scratch, with up to 32 experts and 30B (1.5B active) parameters. Experimental results show significant performance gains over parameter-matched dense models on both perplexity (+13.9%) and a variety of downstream tasks (+1.5%-11.1%). Despite segment-level routing, Lory models achieve competitive performance compared to state-of-the-art MoE models with token-level routing. We further demonstrate that the trained experts in Lory capture domain-level specialization without supervision. Our work highlights the potential of fully-differentiable MoE architectures for language model pre-training and advocates future research in this area.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
