你的数据能带来愉悦感吗？训练末期领域上采样带来的性能提升
Does your data spark joy? Performance gains from domain upsampling at the end of training
AI总结：
本文提出在LLM训练末期对特定领域数据集进行上采样，以低成本显著提升模型在困难基准上的性能，并验证了10%-20%的上采样比例为最优。
摘要：
大型语言模型（LLMs）的预训练数据集已增长至数万亿个token，其中包含大量的CommonCrawl（CC）网络抓取数据以及较小的特定领域数据集。由于需要在庞大的FLOP规模下进行训练才能揭示困难基准和涌现基准的显著变化，因此了解这些特定领域数据集对模型能力的影响成本高昂。鉴于预训练数据实验成本不断增加，如何确定通用网络抓取数据的多样性与特定领域数据的信息密度之间的最佳平衡？在这项工作中，我们展示了如何通过在训练末期相对于CC对这些较小的特定领域数据集进行上采样，来推动在困难基准上的性能提升。这种简单的技术使我们在MMLU上提升了高达6.90 pp，在GSM8K上提升了8.26 pp，在HumanEval上提升了6.17 pp，这是针对一个训练了1万亿（T）token的7B模型相对于基础数据混合的结果，从而匹敌了Llama-2（7B）——一个训练时间两倍的模型。我们实验了将领域上采样的持续时间从训练的5%消融至30%，发现10%到20%是平衡通用语言建模能力与目标基准之间权衡的最佳选择。我们还使用领域上采样，通过在训练的最后阶段移除各个数据集，来大规模表征单个数据集对改进各种基准的效用。这一工具开启了大规模实验不同预训练数据集影响的能力，但与完整预训练运行相比，成本低了一个数量级。
英文摘要：
Pretraining datasets for large language models (LLMs) have grown to trillions of tokens composed of large amounts of CommonCrawl (CC) web scrape along with smaller, domain-specific datasets. It is expensive to understand the impact of these domain-specific datasets on model capabilities as training at large FLOP scales is required to reveal significant changes to difficult and emergent benchmarks. Given the increasing cost of experimenting with pretraining data, how does one determine the optimal balance between the diversity in general web scrapes and the information density of domain specific data? In this work, we show how to leverage the smaller domain specific datasets by upsampling them relative to CC at the end of training to drive performance improvements on difficult benchmarks. This simple technique allows us to improve up to 6.90 pp on MMLU, 8.26 pp on GSM8K, and 6.17 pp on HumanEval relative to the base data mix for a 7B model trained for 1 trillion (T) tokens, thus rivaling Llama-2 (7B)$\unicode{x2014}$a model trained for twice as long. We experiment with ablating the duration of domain upsampling from 5% to 30% of training and find that 10% to 20% percent is optimal for navigating the tradeoff between general language modeling capabilities and targeted benchmarks. We also use domain upsampling to characterize at scale the utility of individual datasets for improving various benchmarks by removing them during this final phase of training. This tool opens up the ability to experiment with the impact of different pretraining datasets at scale, but at an order of magnitude lower cost compared to full pretraining runs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
