arXiv 2312.065502023-12-11cs.CLcs.AIcs.LG

LLM360：迈向完全透明的开源大语言模型

LLM360: Towards Fully Transparent Open-Source LLMs

Petuum（Petuum公司）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；USC（南加州大学）；CMU（卡内基梅隆大学）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；UCSD（加州大学圣地亚哥分校）

Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor … 展开作者

Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Roberto Iriondo, Cun Mu, Zhiting Hu, Mark Schulze, Preslav Nakov, Tim Baldwin, Eric P. Xing

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 LLM360 开源倡议，通过公开训练代码、数据、检查点和中间结果提升 LLM 训练透明度，并首发两个 7B 模型 Amber 与 CrystalCider。

摘要：

近期，LLaMA、Falcon 和 Mistral 等开源大语言模型（LLM）的涌现，为 AI 从业者和研究人员提供了多样化选择。然而，大多数 LLM 仅发布了部分成果，例如最终模型权重或推理代码，技术报告也越来越局限于高层设计选择和表面统计信息。这些做法降低了 LLM 训练过程的透明度，并迫使各团队重新发现训练过程中的许多细节，从而阻碍了该领域的进展。我们提出 LLM360，这是一项完全开源 LLM 的倡议，主张向社区提供所有训练代码和数据、模型检查点以及中间结果。LLM360 的目标是让端到端的 LLM 训练过程对每个人都透明且可复现，从而支持开放、协作的 AI 研究。作为 LLM360 的第一步，我们发布了两个从头开始预训练的 7B 参数 LLM：Amber 和 CrystalCoder，包括其训练代码、数据、中间检查点和分析（见 https://www.llm360.ai）。我们致力于通过这一开源努力持续拓展 LLM 的边界。更多大规模、更强的模型正在开发中，并将在未来发布。

英文摘要：

The recent surge in open-source Large Language Models (LLMs), such as LLaMA, Falcon, and Mistral, provides diverse options for AI practitioners and researchers. However, most LLMs have only released partial artifacts, such as the final model weights or inference code, and technical reports increasingly limit their scope to high-level design choices and surface statistics. These choices hinder progress in the field by degrading transparency into the training of LLMs and forcing teams to rediscover many details in the training process. We present LLM360, an initiative to fully open-source LLMs, which advocates for all training code and data, model checkpoints, and intermediate results to be made available to the community. The goal of LLM360 is to support open and collaborative AI research by making the end-to-end LLM training process transparent and reproducible by everyone. As a first step of LLM360, we release two 7B parameter LLMs pre-trained from scratch, Amber and CrystalCoder, including their training code, data, intermediate checkpoints, and analyses (at https://www.llm360.ai). We are committed to continually pushing the boundaries of LLMs through this open-source effort. More large-scale and stronger models are underway and will be released in the future.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。