中期训练连接预训练和后训练分布
Midtraining Bridges Pretraining and Posttraining Distributions
AI总结：
中期训练通过提供更好的初始化提升模型性能，尤其在与通用预训练数据距离较远的领域表现突出，且起始时间和混合权重的相互作用影响其效果。
摘要：
中期训练，即在中间训练阶段将专门数据与更通用的预训练数据混合的做法，在语言模型开发中变得普遍，但对其为何有效的理解仍有限。我们提出中期训练通过为后训练提供更好的初始化来起作用。我们进行了受控的预训练实验，发现中期训练的好处最大体现在与通用预训练数据距离较远的领域，如代码和数学，并且随着中期训练数据接近目标分布的优势而扩大。在这些领域，中期训练在领域内和在缓解遗忘方面均优于仅使用专门数据进行的持续预训练。我们进一步研究了中期训练数据的起始时间和混合权重，以代码为例进行案例研究，发现起始时间和混合权重存在强烈相互作用，即早期引入专门数据可以容忍较高的混合权重，而后期引入则需要较低的权重。这表明，在塑性窗口之外后期引入专门数据无法通过后期增加数据混合来补偿。除了中期训练本身，这还表明任何训练阶段之间的分布转换可能受益于类似的桥接策略。
英文摘要：
Midtraining, the practice of mixing specialized data with more general pretraining data in an intermediate training phase, has become widespread in language model development, yet there is little understanding of what makes it effective. We propose that midtraining functions as distributional bridging by providing better initialization for posttraining. We conduct controlled pretraining experiments, and find that midtraining benefits are largest for domains distant from general pretraining data, such as code and math, and scale with the proximity advantage the midtraining data provides toward the target distribution. In these domains, midtraining consistently outperforms continued pretraining on specialized data alone both in-domain and in terms of mitigating forgetting. We further conduct an investigation on the starting time and mixture weight of midtraining data, using code as a case study, and find that time of introduction and mixture weight interact strongly such that early introduction of specialized data is amenable to high mixture weights, while late introduction requires lower ones. This suggests that late introduction of specialized data outside a plasticity window cannot be compensated for by increasing data mixtures later in training. Beyond midtraining itself, this suggests that distributional transitions between any training phases may benefit from similar bridging strategies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
