JetMoE：以0.1百万美元成本达到Llama2性能
JetMoE: Reaching Llama2 Performance with 0.1M Dollars
AI总结：
本研究提出低成本大语言模型JetMoE-8B，基于稀疏门控混合专家架构，以不足0.1百万美元成本训练，性能超越Llama2-7B，且完全开源以推动领域协作。
摘要：
大语言模型（LLM）已取得显著成果，但其不断增长的资源需求已成为开发强大且可及的超人类智能的主要障碍。本报告介绍了JetMoE-8B，这是一款训练成本低于0.1百万美元的新型大语言模型，使用了经过精心混合的开源语料库中的1.25T tokens，以及30,000个H100 GPU小时。尽管成本低廉，JetMoE-8B仍展现出令人印象深刻的性能：JetMoE-8B的表现优于Llama2-7B模型，JetMoE-8B-Chat则超越了Llama2-13B-Chat模型。这些结果表明，大语言模型训练的成本效益可以比普遍认为的高得多。JetMoE-8B基于高效的稀疏门控混合专家（Sparsely-gated Mixture-of-Experts, SMoE）架构，由注意力专家和前馈专家组成。这两层均采用稀疏激活方式，使得JetMoE-8B拥有8B参数的同时，每个输入token仅激活2B参数，与Llama2-7B相比，推理计算量减少了约70%。此外，JetMoE-8B具有高度开放性且对学术界友好，仅使用公开数据集和训练代码。本报告详细列出了所有训练参数和数据混合方案，以推动未来开放基础模型的开发工作。这种透明度旨在鼓励合作，推动可及、高效的大语言模型领域的进一步发展。模型权重已在https://github.com/myshell-ai/JetMoE 公开提供。
英文摘要：
Large Language Models (LLMs) have achieved remarkable results, but their increasing resource demand has become a major obstacle to the development of powerful and accessible super-human intelligence. This report introduces JetMoE-8B, a new LLM trained with less than $0.1 million, using 1.25T tokens from carefully mixed open-source corpora and 30,000 H100 GPU hours. Despite its low cost, the JetMoE-8B demonstrates impressive performance, with JetMoE-8B outperforming the Llama2-7B model and JetMoE-8B-Chat surpassing the Llama2-13B-Chat model. These results suggest that LLM training can be much more cost-effective than generally thought. JetMoE-8B is based on an efficient Sparsely-gated Mixture-of-Experts (SMoE) architecture, composed of attention and feedforward experts. Both layers are sparsely activated, allowing JetMoE-8B to have 8B parameters while only activating 2B for each input token, reducing inference computation by about 70% compared to Llama2-7B. Moreover, JetMoE-8B is highly open and academia-friendly, using only public datasets and training code. All training parameters and data mixtures have been detailed in this report to facilitate future efforts in the development of open foundation models. This transparency aims to encourage collaboration and further advancements in the field of accessible and efficient LLMs. The model weights are publicly available at https://github.com/myshell-ai/JetMoE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
