数据达尔文主义 第一部分:解锁科学数据为预训练带来的价值
Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training
AI总结:
数据达尔文主义通过十级分类法提升科学数据预训练效果,利用生成细化和认知完成技术,实现数据与模型的共同进化。
摘要:
数据质量决定了基础模型的性能,但缺乏系统化的处理框架。我们介绍了数据达尔文主义,一个十级分类法(L0-L9),它概念化了数据与模型的共同进化:高级模型产生更优质的数据用于下一代系统。我们通过构建Darwin-Science,一个900B-token语料库(L0-L5)在科学文献中验证了这一点。我们识别出原始科学文本中的可学习差距,并通过L4(生成细化)和L5(认知完成)使用前沿LLM来解释推理和术语。为了确保严谨的归因,我们从头开始预训练daVinci-origin-3B/7B模型,排除科学内容以创建无污染的基准。在600B token的持续预训练后,Darwin-Science在20多个基准测试中比基准高出+2.12(3B)和+2.95(7B)点,达到领域对齐任务的+5.60和+8.40点。系统性地推进到L5带来了+1.36的总收益,证实了更高层次的处理解锁了潜在的数据价值。我们发布了Darwin-Science语料库和daVinci-origin模型以促进原则性的共同进化发展。
英文摘要:
Data quality determines foundation model performance, yet systematic processing frameworks are lacking. We introduce Data Darwinism, a ten-level taxonomy (L0-L9) that conceptualizes data-model co-evolution: advanced models produce superior data for next-generation systems. We validate this on scientific literature by constructing Darwin-Science, a 900B-token corpus (L0-L5). We identify a learnability gap in raw scientific text, which we bridge via L4 (Generative Refinement) and L5 (Cognitive Completion) using frontier LLMs to explicate reasoning and terminology. To ensure rigorous attribution, we pre-trained daVinci-origin-3B/7B models from scratch, excluding scientific content to create contamination-free baselines. After 600B tokens of continued pre-training, Darwin-Science outperforms baselines by +2.12 (3B) and +2.95 (7B) points across 20+ benchmarks, rising to +5.60 and +8.40 points on domain-aligned tasks. Systematic progression to L5 yields a +1.36 total gain, confirming that higher-level processing unlocks latent data value. We release the Darwin-Science corpus and daVinci-origin models to enable principled, co-evolutionary development.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。