发表机构
MIT CSAIL; Microsoft Research; Harvard University（麻省理工学院计算机科学与人工智能实验室; 微软研究院; 哈佛大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究语言模型预训练中的数据协同效应，利用开放权重语言模型的观测变化估计领域间协同效应，其框架提高预测精度，恢复稳定估计，经训练模型验证能正确预测性能排名。
AI 中文摘要
机器学习的进展通常归因于扩大模型规模和数据集大小，但数据的组成同样重要。实证研究反复表明，合并来自不同领域的数据集会产生重要的相互作用。例如，添加代码数据可提升数学推理能力，而某些混合数据会产生干扰，降低模型性能。我们将这些效应统称为数据协同效应，即多个领域的贡献超过或低于其单独贡献之和。在这项工作中，我们对语言模型预训练中的数据协同效应进行了形式化和量化。利用具有不同预训练混合的开放权重语言模型的观测变化，我们估计了直接的领域到基准协同效应（一个领域对另一个领域性能的贡献）和二阶领域间协同效应（需要多个领域共同出现的能力）。我们的框架提高了预测精度，恢复了稳定的协同效应估计。我们通过在预测的最优和预测的反最优混合数据上训练模型来验证这些估计，并确认我们的协同效应估计正确地预测了性能排名。
英文摘要
Machine learning progress is often attributed to scaling model size and dataset volume, yet the composition of data can be just as consequential. Empirical findings repeatedly show that combining datasets from different domains yields nontrivial interactions. For instance, adding code improves mathematical reasoning, while certain mixtures introduce interference that reduces model performance. We refer to these effects collectively as data synergy, where the contribution of multiple domains exceeds or falls short of the sum of their isolated contributions. In this work, we formalize and quantify data synergy in language model pretraining. Leveraging observational variation across open-weight LLMs with diverse pretraining mixtures, we estimate both direct domain-to-benchmark synergy (how one domain contributes to performance on another) and a second-order domain-domain synergy (capabilities that require co-occurrence of multiple domains). Our framework improves predictive accuracy over domain-agnostic scaling laws and recovers stable synergy estimates. We validate these estimates by training models on predicted optimal and predicted anti-optimal mixtures and confirm that our synergy estimates correctly predict performance rankings.
