测量语言模型预训练中与任务无关的训练数据影响

中文摘要 AI 辅助

在语言模型预训练过程中一致地测量训练数据影响颇具挑战，难以选择能代表模型通用能力的下游任务或验证集，且依赖中间检查点的任务表现会使训练间的对比复杂化。本文提出一种无需选择下游任务或验证集作为归因目标的训练数据影响度量方法：具体而言，我们将一个样本的影响定义为其梯度更新减少给定预训练运行最终参数平方距离的程度，并从中间检查点估计该量值，无需重新训练。将该方法应用于Pythia和PolyPythia套件的18种配置后，我们发现有影响力的数据存在系统性时间变化：训练早期，与文献相关的数据与指向最终参数的轨迹更一致，而STEM数据在后期阶段变得更一致；这种定性的交叉在各模型配置中基本一致。我们的结果为预训练过程中有影响力的数据如何变化提供了一种易处理的轨迹级视图，补充了针对特定下游任务或验证集定义的影响分析。

英文摘要

Measuring training data influence consistently across language model pretraining is challenging. It is difficult to select downstream tasks or validation sets representative of a model's general capabilities, and reliance on task performance at intermediate checkpoints complicates comparisons across training. We propose a measure of training data influence that does not require selecting a downstream task or validation set as the attribution target. Specifically, we define an example's influence by how much its gradient update reduces the squared distance to the final parameters of a given pretraining run, and estimate this quantity from intermediate checkpoints without retraining. Applying the method to 18 configurations from the Pythia and PolyPythia suites, we find systematic temporal changes in influential data. Early in training, literature-related data are more strongly aligned with the trajectory toward the final parameters, whereas STEM data become more strongly aligned in later stages. This qualitative crossover is broadly consistent across model configurations. Our results provide a tractable trajectory-level view of how influential data change throughout pretraining, complementing influence analyses defined with respect to specific downstream tasks or validation sets.

发表机构

• Nara Institute of Science and Technology（奈良科学技术研究所）
• Waseda University（早稻田大学）
• The University of Tokyo（东京大学）
• IT University of Copenhagen（哥本哈根信息技术大学）
• Tohoku University（东北大学）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

Accepted to COLM 2026