BeyondWeb：万亿级预训练中合成数据规模化的经验教训
BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining
AI总结：
针对LLM预训练单纯扩数据量遇数据墙的问题，提出合成数据生成框架BeyondWeb，其生成的合成数据在14项基准评估中优于Cosmopedia等现有数据集，训练速度更快，还得出合成数据质量优化的关键见解。
摘要：
大型语言模型（LLM）预训练的最新进展表明，单纯扩大数据量最终会导致收益递减，遭遇数据墙。对此，将合成数据用于预训练已成为突破性能前沿的有前景范式。尽管如此，影响合成数据质量的因素仍知之甚少。在本工作中，我们引入BeyondWeb，一个为预训练生成高质量合成数据的框架。BeyondWeb显著扩展了传统网络规模数据集的能力，在14项基准评估的平均结果中，分别比最先进的合成预训练数据集（如Cosmopedia和Nemotron-CC的高质量合成子集Nemotron-Synth）高出5.1个百分点（pp）和2.6个百分点。它的训练速度比开放网络数据快7.7倍，比Nemotron-Synth快2.7倍。值得注意的是，在BeyondWeb上用180B tokens训练的3B模型，优于在Cosmopedia上用相同token预算训练的8B模型。我们还从BeyondWeb中得出关于预训练合成数据的几点见解：是什么驱动其收益、哪些数据需要重述及如何重述，以及模型大小和家族对数据质量的影响。总体而言，我们的工作表明，生成高质量合成预训练数据没有万能方案。最佳结果需要联合优化多个因素，这是一项需要严谨科学和实践经验的挑战。朴素方法只能带来有限改进，且可能代价高昂，而执行良好的方法（如BeyondWeb所示）则能带来变革性改进。
英文摘要：
Recent advances in large language model (LLM) pretraining have shown that simply scaling data quantity eventually leads to diminishing returns, hitting a data wall. In response, the use of synthetic data for pretraining has emerged as a promising paradigm for pushing the frontier of performance. Despite this, the factors affecting synthetic data quality remain poorly understood. In this work, we introduce BeyondWeb, a synthetic data generation framework that produces high-quality synthetic data for pretraining. BeyondWeb significantly extends the capabilities of traditional web-scale datasets, outperforming state-of-the-art synthetic pretraining datasets such as Cosmopedia and Nemotron-CC's high-quality synthetic subset (Nemotron-Synth) by up to 5.1 percentage points (pp) and 2.6pp, respectively, when averaged across a suite of 14 benchmark evaluations. It delivers up to 7.7x faster training than open web data and 2.7x faster than Nemotron-Synth. Remarkably, a 3B model trained for 180B tokens on BeyondWeb outperforms an 8B model trained for the same token budget on Cosmopedia. We also present several insights from BeyondWeb on synthetic data for pretraining: what drives its benefits, which data to rephrase and how, and the impact of model size and family on data quality. Overall, our work shows that there's no silver bullet for generating high-quality synthetic pretraining data. The best outcomes require jointly optimizing many factors, a challenging task that requires rigorous science and practical expertise. Naive approaches can yield modest improvements, potentially at great cost, while well-executed methods can yield transformative improvements, as exemplified by BeyondWeb.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
