arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.06380cs.CL
收藏

数据,无处不在的数据:预训练数据集构建指南

Data, Data Everywhere: A Guide for Pretraining Dataset Construction

NVIDIA(英伟达)

Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Bo Liu, Aastha Jhunjhunwala, Zhilin Wang, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro

AI总结:

本文首次系统研究预训练数据集构建全流程,通过消融实验识别最有效技术,并对网络爬取数据按毒性、质量等属性分类以优化预训练集质量,为实践者提供可操作的构建指南。

摘要:

近期语言模型令人印象深刻的能力在很大程度上可归因于它们所训练的数万亿 token 规模的预训练数据集。然而,模型开发者并未公开其构建方法,这导致关于如何开发有效预训练集的公开信息十分匮乏。为解决这一问题,我们首次对预训练集构建的整个流程进行了系统性研究。首先,我们对现有的预训练集开发技术进行了消融实验,以确定哪些方法能在下游评估中为模型准确率带来最大提升。然后,我们针对使用最广泛的数据来源——网络爬取快照,按照毒性、质量、语音类型和领域等属性进行了分类。最后,我们展示了如何利用这些属性信息进一步细化和提升预训练集的质量。这些发现构成了一套可操作的步骤,实践者可以据此开发高质量的预训练集。

英文摘要:

The impressive capabilities of recent language models can be largely attributed to the multi-trillion token pretraining datasets that they are trained on. However, model developers fail to disclose their construction methodology which has lead to a lack of open information on how to develop effective pretraining sets. To address this issue, we perform the first systematic study across the entire pipeline of pretraining set construction. First, we run ablations on existing techniques for pretraining set development to identify which methods translate to the largest gains in model accuracy on downstream evaluations. Then, we categorize the most widely used data source, web crawl snapshots, across the attributes of toxicity, quality, type of speech, and domain. Finally, we show how such attribute information can be used to further refine and improve the quality of a pretraining set. These findings constitute an actionable set of steps that practitioners can use to develop high quality pretraining sets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑