arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.01739cs.CLcs.AIcs.DCcs.LG
收藏

OpenMoE:开放混合专家语言模型的早期探索

OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

National University of Singapore(新加坡国立大学);University of Edinburgh(爱丁堡大学);ETH Zurich(苏黎世联邦理工学院)

Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, Yang You

AI总结:

OpenMoE 开源了一系列 650M 到 34B 参数的混合专家大语言模型,证实其成本效益优于稠密模型,并通过分析路由机制发现上下文无关专业化、早期路由学习和末端丢弃倾向等问题,进而提出改进策略。

摘要:

为了帮助开源社区更好地理解基于混合专家(Mixture-of-Experts,MoE)的大语言模型(LLMs),我们训练并发布了 OpenMoE,这是一系列完全开源且可复现的仅解码器 MoE 大语言模型,参数规模从 650M 到 34B 不等,并在最多超过 1T 的 token 上进行了训练。我们的研究证实,基于 MoE 的大语言模型能够提供比稠密大语言模型更有利的成本效益权衡,凸显了其对未来大语言模型开发的潜在有效性。本研究的另一项重要贡献是对 OpenMoE 模型内部路由机制进行了深入分析,得出了三个重要发现:上下文无关专业化、早期路由学习和末端丢弃倾向。我们发现,MoE 模型中的路由决策主要基于 token ID,与上下文的相关性极小。token 到专家的分配在预训练早期阶段就已确定,并且在之后基本保持不变。这种不完善的路由可能导致性能下降,尤其是在多轮对话等序列任务中,序列中较后出现的 token 更有可能被丢弃。最后,我们基于上述观察和分析重新思考了我们的设计。为了促进未来 MoE 大语言模型的发展,我们提出了缓解所发现问题的潜在策略,并进一步改进现成的 MoE 大语言模型设计。

英文摘要:

To help the open-source community have a better understanding of Mixture-of-Experts (MoE) based large language models (LLMs), we train and release OpenMoE, a series of fully open-sourced and reproducible decoder-only MoE LLMs, ranging from 650M to 34B parameters and trained on up to over 1T tokens. Our investigation confirms that MoE-based LLMs can offer a more favorable cost-effectiveness trade-off than dense LLMs, highlighting the potential effectiveness for future LLM development. One more important contribution of this study is an in-depth analysis of the routing mechanisms within our OpenMoE models, leading to three significant findings: Context-Independent Specialization, Early Routing Learning, and Drop-towards-the-End. We discovered that routing decisions in MoE models are predominantly based on token IDs, with minimal context relevance. The token-to-expert assignments are determined early in the pre-training phase and remain largely unchanged. This imperfect routing can result in performance degradation, particularly in sequential tasks like multi-turn conversations, where tokens appearing later in a sequence are more likely to be dropped. Finally, we rethink our design based on the above-mentioned observations and analysis. To facilitate future MoE LLM development, we propose potential strategies for mitigating the issues we found and further improving off-the-shelf MoE LLM designs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑