为每个示例编程:像专家一样大规模提升预训练数据质量
Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale
AI总结:
针对传统预训练数据规则缺乏灵活性且人工定制不切实际的问题,提出ProX框架将数据精炼视为编程任务,通过小型模型为每个示例生成细粒度操作实现大规模精炼,提升模型下游表现超2%,特定领域预训练效果显著且节省FLOPs。
摘要:
大型语言模型预训练传统上依赖人类专家设计启发式规则来提升语料库质量,至今已开发出大量规则。然而,这些规则缺乏有效处理单个示例独特特征的灵活性。同时,人类专家为每个示例应用定制规则是不切实际的。在本文中,我们证明即使是参数仅为0.3B的小型语言模型,也能展现出与人类专家相当的显著数据精炼能力。我们引入“为每个示例编程(ProX)”这一新颖框架,将数据精炼视为编程任务,使模型能够通过为每个示例生成并执行细粒度操作(如字符串归一化)来大规模精炼语料库。实验结果表明,在ProX整理的数据上预训练的模型,在各类下游基准测试中比原始数据或其他选择方法过滤的数据表现高出2%以上。其有效性适用于多种模型规模和预训练语料库,包括C4、RedPajama-V2、FineWeb、FineWeb-Edu和DCLM。此外,ProX在特定领域持续预训练中展现出巨大潜力:无需特定领域设计,在ProX精炼的OpenWebMath上训练的模型优于人类设计的基于规则的方法,Mistral-7B平均准确率提升7.6%,Llama-2-7B提升14.6%,CodeLlama-7B提升20.3%,且均在10B tokens内完成,可与使用200B tokens训练的Llemma-7B等模型媲美。进一步分析表明,ProX显著节省训练浮点运算次数(FLOPs),为高效LLM预训练提供了有前景的路径。我们开源了ProX,包含超过500B的语料库、模型,并分享所有训练和实现细节以支持可复现研究和未来创新。代码:https://github.com/GAIR-NLP/ProX
英文摘要:
Large language model pre-training has traditionally relied on human experts to craft heuristics for improving the corpora quality, resulting in numerous rules developed to date. However, these rules lack the flexibility to address the unique characteristics of individual example effectively. Meanwhile, applying tailored rules to every example is impractical for human experts. In this paper, we demonstrate that even small language models, with as few as 0.3B parameters, can exhibit substantial data refining capabilities comparable to those of human experts. We introduce Programming Every Example (ProX), a novel framework that treats data refinement as a programming task, enabling models to refine corpora by generating and executing fine-grained operations, such as string normalization, for each individual example at scale. Experimental results show that models pre-trained on ProX-curated data outperform either original data or data filtered by other selection methods by more than 2% across various downstream benchmarks. Its effectiveness spans various model sizes and pre-training corpora, including C4, RedPajama-V2, FineWeb, FineWeb-Edu, and DCLM. Furthermore, ProX exhibits significant potential in domain-specific continual pre-training: without domain specific design, models trained on OpenWebMath refined by ProX outperform human-crafted rule-based methods, improving average accuracy by 7.6% over Mistral-7B, with 14.6% for Llama-2-7B and 20.3% for CodeLlama-7B, all within 10B tokens to be comparable to models like Llemma-7B trained on 200B tokens. Further analysis highlights that ProX significantly saves training FLOPs, offering a promising path for efficient LLM pre-training. We are open-sourcing ProX with >500B corpus, models, and sharing all training and implementation details for reproducible research and future innovation. Code: https://github.com/GAIR-NLP/ProX
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。