Nemotron-CC：将Common Crawl转化为精炼的长视野预训练数据集
Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset
AI总结：
本文提出Nemotron-CC数据集，通过分类器集成、合成数据重述和减少启发式过滤，在准确性与数据量间取得更优平衡，支持长视野预训练并显著提升模型性能。
摘要：
近期的英语Common Crawl数据集（如FineWeb-Edu和DCLM）通过激进的基于模型的过滤获得了显著的基准测试收益，但代价是移除了90%的数据。这限制了它们对长token视野训练的适用性，例如Llama 3.1的15T tokens训练。在本文中，我们展示了如何通过结合分类器集成、合成数据重述以及减少对启发式过滤器的依赖，在准确性和数据量之间实现更好的权衡。当训练8B参数模型1T tokens时，使用我们数据的高质量子集在MMLU上比DCLM提高了5.6，证明了我们的方法在相对较短的token视野内提升准确性的有效性。此外，我们完整的6.3T token数据集在MMLU上与DCLM相匹配，但包含的真实唯一token数量是DCLM的四倍。这解锁了长token视野下的最先进训练：一个训练了15T tokens的8B参数模型（其中7.2T来自我们的数据集）优于Llama 3.1 8B模型：在MMLU上+5，在ARC-Challenge上+3.1，在十个不同任务上的平均得分+0.5。该数据集可在https://data.commoncrawl.org/contrib/Nemotron/Nemotron-CC/index.html获取。
英文摘要：
Recent English Common Crawl datasets like FineWeb-Edu and DCLM achieved significant benchmark gains via aggressive model-based filtering, but at the cost of removing 90% of data. This limits their suitability for long token horizon training, such as 15T tokens for Llama 3.1. In this paper, we show how to achieve better trade-offs between accuracy and data quantity by a combination of classifier ensembling, synthetic data rephrasing, and reduced reliance on heuristic filters. When training 8B parameter models for 1T tokens, using a high-quality subset of our data improves MMLU by 5.6 over DCLM, demonstrating the efficacy of our methods for boosting accuracies over a relatively short token horizon. Furthermore, our full 6.3T token dataset matches DCLM on MMLU, but contains four times more unique real tokens than DCLM. This unlocks state-of-the-art training over a long token horizon: an 8B parameter model trained for 15T tokens, of which 7.2T came from our dataset, is better than the Llama 3.1 8B model: +5 on MMLU, +3.1 on ARC-Challenge, and +0.5 on average across ten diverse tasks. The dataset is available at https://data.commoncrawl.org/contrib/Nemotron/Nemotron-CC/index.html
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
