arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.18041cs.CLcs.AI
收藏

面向大语言模型的数据集:综合综述

Datasets for Large Language Models: A Comprehensive Survey

South China University of Technology(华南理工大学);INTSIG Information Co., Ltd(影诺信息有限公司)

Yang Liu, Jiahuan Cao, Chongyu Liu, Kai Ding, Lianwen Jin

AI总结:

该综述从预训练、指令微调、偏好、评测及传统NLP五类数据集出发,系统梳理444个LLM数据集,揭示挑战与未来方向。

摘要:

本文对大语言模型(LLM)数据集展开探索,这些数据集在LLM的显著进展中发挥着关键作用。数据集作为基础性基础设施,类似于维持并滋养LLM发展的根系。因此,考察这些数据集成为研究中的一个关键课题。为解决当前缺乏对LLM数据集的全面概述与深入分析的问题,并洞察其现状与未来趋势,本综述从五个角度整合并分类了LLM数据集的基本方面:(1)预训练语料库;(2)指令微调数据集;(3)偏好数据集;(4)评测数据集;(5)传统自然语言处理(NLP)数据集。本综述阐明了当前普遍存在的挑战,并指出了未来研究的潜在途径。此外,本文还全面回顾了现有可用的数据集资源,包括来自444个数据集的统计信息,覆盖8种语言类别并横跨32个领域。数据集统计纳入了20个维度的信息。所调查的数据总规模超过774.5 TB预训练语料库,以及其他数据集的7亿个实例。我们旨在呈现LLM文本数据集的完整图景,为该领域研究人员提供全面参考,并为未来研究作出贡献。相关资源可在以下地址获取:https://github.com/lmmlzn/Awesome-LLMs-Datasets。

英文摘要:

This paper embarks on an exploration into the Large Language Model (LLM) datasets, which play a crucial role in the remarkable advancements of LLMs. The datasets serve as the foundational infrastructure analogous to a root system that sustains and nurtures the development of LLMs. Consequently, examination of these datasets emerges as a critical topic in research. In order to address the current lack of a comprehensive overview and thorough analysis of LLM datasets, and to gain insights into their current status and future trends, this survey consolidates and categorizes the fundamental aspects of LLM datasets from five perspectives: (1) Pre-training Corpora; (2) Instruction Fine-tuning Datasets; (3) Preference Datasets; (4) Evaluation Datasets; (5) Traditional Natural Language Processing (NLP) Datasets. The survey sheds light on the prevailing challenges and points out potential avenues for future investigation. Additionally, a comprehensive review of the existing available dataset resources is also provided, including statistics from 444 datasets, covering 8 language categories and spanning 32 domains. Information from 20 dimensions is incorporated into the dataset statistics. The total data size surveyed surpasses 774.5 TB for pre-training corpora and 700M instances for other datasets. We aim to present the entire landscape of LLM text datasets, serving as a comprehensive reference for researchers in this field and contributing to future studies. Related resources are available at: https://github.com/lmmlzn/Awesome-LLMs-Datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑