arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.01981cs.CLcs.AI
收藏

Zyda:一个用于开放语言建模的1.3T数据集

Zyda: A 1.3T Dataset for Open Language Modeling

Zyphra(齐弗拉(Zyphra))

Yury Tokpanov, Beren Millidge, Paolo Glorioso, Jonathan Pilault, Adam Ibrahim, James Whittington, Quentin Anthony

AI总结:

本文提出Zyda数据集,整合多个开源数据集形成1.3万亿token的高质量语料库,通过严格过滤与去重提升质量,在性能上优于Dolma等开放数据集并显著改善Pythia模型表现。

摘要:

近年来,大型语言模型(LLMs)的规模急剧扩大,其计算和数据需求也随之激增。即使规模相对较小的最先进语言模型,通常也需要在至少一万亿个token上进行训练。这一快速发展已超过了可用于大规模LLM预训练的开源数据集的增长。在本文中,我们介绍了Zyda(Zyphra Dataset),这是一个在宽松许可下包含1.3万亿个token的数据集,通过将多个备受尊重的主要开源数据集整合成一个高质量语料库而构建。我们应用了严格的过滤和去重流程,包括数据集内部和跨数据集的处理,以保持并提升源自原始数据集的质量。我们的评估表明,Zyda不仅在与Dolma、FineWeb和RefinedWeb等其他开放数据集相比时具有竞争力,而且还显著提升了Pythia系列可比模型的性能。我们严格的数据处理方法显著增强了Zyda的有效性,甚至优于其各组成数据集独立使用时的最佳表现。

英文摘要:

The size of large language models (LLMs) has scaled dramatically in recent years and their computational and data requirements have surged correspondingly. State-of-the-art language models, even at relatively smaller sizes, typically require training on at least a trillion tokens. This rapid advancement has eclipsed the growth of open-source datasets available for large-scale LLM pretraining. In this paper, we introduce Zyda (Zyphra Dataset), a dataset under a permissive license comprising 1.3 trillion tokens, assembled by integrating several major respected open-source datasets into a single, high-quality corpus. We apply rigorous filtering and deduplication processes, both within and across datasets, to maintain and enhance the quality derived from the original datasets. Our evaluations show that Zyda not only competes favorably with other open datasets like Dolma, FineWeb, and RefinedWeb, but also substantially improves the performance of comparable models from the Pythia suite. Our rigorous data processing methods significantly enhance Zyda's effectiveness, outperforming even the best of its constituent datasets when used independently.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑