MathPile：一个面向数学的十亿 token 规模预训练语料库
MathPile: A Billion-Token-Scale Pretraining Corpus for Math
AI总结：
本研究推出包含约95亿token的高质量数学核心预训练语料库MathPile，经多流程数据处理与污染检测，通过持续预训练提升了数学推理基准性能，并开源相关资源以推动领域发展。
摘要：
高质量、大规模语料库是构建基础模型的基石。在本研究中，我们推出了MathPile——一个多样化、高质量的以数学为核心的语料库，包含约95亿个token。在其构建全程中，我们遵循“少即是多”的原则，坚信即便在预训练阶段，数据质量也优先于数量。我们精细的数据收集与处理工作包含一整套复杂的预处理、预过滤、语言识别、清洗、过滤和去重流程，保障了语料库的高质量。此外，我们针对下游基准测试集开展了数据污染检测以消除重复项，并进行了持续预训练实验，提升了在通用数学推理基准上的性能。我们期望MathPile能够增强语言模型的数学推理能力，并开源其不同版本及处理脚本以推动该领域发展。
英文摘要：
High-quality, large-scale corpora are the cornerstone of building foundation models. In this work, we introduce MathPile, a diverse and high-quality math-centric corpus comprising about 9.5 billion tokens. Throughout its creation, we adhered to the principle of "less is more", firmly believing in the supremacy of data quality over quantity, even in the pre-training phase. Our meticulous data collection and processing efforts included a complex suite of preprocessing, prefiltering, language identification, cleaning, filtering, and deduplication, ensuring the high quality of our corpus. Furthermore, we performed data contamination detection on downstream benchmark test sets to eliminate duplicates and conducted continual pre-training experiments, booting the performance on common mathematical reasoning benchmarks. We aim for our MathPile to boost language models' mathematical reasoning abilities and open-source its different versions and processing scripts to advance the field.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
