arXiv 2311.033012023-11cs.CL

Ziya2：以数据为中心的学习就是所有 LLM 所需的一切

Ziya2: Data-centric Learning is All LLMs Need

International Digital Economy Academy（国际数字经济学院）；University of Science and Technology of China（中国科学技术大学）

Ruyi Gan, Ziwei Wu, Renliang Sun, Junyu Lu, Xiaojun Wu, Dixiang Zhang, Kunhao Pan, Junqing He, Yuanhe Tian, Ping Yang, Qi Yang, Hao Wang, Jiaxing Zhang, Yan Song

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对 LLM 训练成本高、持续预训练易遗忘及数据利用不足的问题，本文提出基于 LLaMA2 的 130 亿参数模型 Ziya2，通过 7000 亿 token 数据中心优化和数据缩放定律提升多基准性能。

摘要：

近年来，人们提出了各种大型语言模型（LLM），包括闭源和开源模型，并不断在多个基准测试上创下新纪录。然而，LLM 的发展仍面临若干问题，例如从头训练模型成本高昂，以及持续预训练导致灾难性遗忘等。尽管沿着 LLM 研究方向已有许多此类问题得到解决，但一个重要且实际的局限是：许多研究过度追求扩大模型规模，而没有全面分析和优化预训练数据在其学习过程中的使用，也没有在具有成本效益的设置下适当组织并利用这些数据来训练 LLM。在本工作中，我们提出 Ziya2，这是一个拥有 130 亿参数的模型，采用 LLaMA2 作为基础模型，并在 7000 亿个 token 上进一步预训练；我们聚焦预训练技术，并使用以数据为中心的优化来增强 Ziya2 在不同阶段的学习过程。我们定义了三个数据属性，并首次建立以数据为中心的缩放定律，以说明不同数据如何影响 LLM。实验表明，Ziya2 在多个基准测试中显著优于其他模型，尤其是与具有代表性的开源模型相比取得了很有前景的结果。Ziya2（Base）已发布于 https://huggingface.co/IDEA-CCNL/Ziya2-13B-Base 和 https://modelscope.cn/models/Fengshenbang/Ziya2-13B-Base/summary。

英文摘要：

Various large language models (LLMs) have been proposed in recent years, including closed- and open-source ones, continually setting new records on multiple benchmarks. However, the development of LLMs still faces several issues, such as high cost of training models from scratch, and continual pre-training leading to catastrophic forgetting, etc. Although many such issues are addressed along the line of research on LLMs, an important yet practical limitation is that many studies overly pursue enlarging model sizes without comprehensively analyzing and optimizing the use of pre-training data in their learning process, as well as appropriate organization and leveraging of such data in training LLMs under cost-effective settings. In this work, we propose Ziya2, a model with 13 billion parameters adopting LLaMA2 as the foundation model, and further pre-trained on 700 billion tokens, where we focus on pre-training techniques and use data-centric optimization to enhance the learning process of Ziya2 on different stages. We define three data attributes and firstly establish data-centric scaling laws to illustrate how different data impacts LLMs. Experiments show that Ziya2 significantly outperforms other models in multiple benchmarks especially with promising results compared to representative open-source ones. Ziya2 (Base) is released at https://huggingface.co/IDEA-CCNL/Ziya2-13B-Base and https://modelscope.cn/models/Fengshenbang/Ziya2-13B-Base/summary.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文