复用而非重训练：语言模型持续预训练的一套方案
Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models
AI总结：
针对语言模型预训练成本过高的问题，本研究提出一套涵盖数据分布与学习率调度的持续预训练准则，在15B参数模型上实现了9%的平均准确率提升，为模型复用开发提供了实用方案。
摘要：
随着语言模型的参数量与预训练数据集规模不断扩大，除资源最充足的团队外，预训练的计算成本已变得难以承受。这种不断攀升的成本使得在模型完成预训练后对其进行复用变得愈发重要——无需从头训练就能进一步提升模型能力。本研究详细提出了一套指导准则，涵盖如何为语言模型的持续预训练设计有效的数据分布和学习率调度策略。我们在一个训练完善的15B参数模型上开展持续预训练实验，应用这些研究发现后，模型平均准确率相比在原预训练集上持续训练的基线提升了9%。这套最终方案提供了一个实用起点，可通过复用而非重训练的方式来开发语言模型。
英文摘要：
As language models have scaled both their number of parameters and pretraining dataset sizes, the computational cost for pretraining has become intractable except for the most well-resourced teams. This increasing cost makes it ever more important to be able to reuse a model after it has completed pretraining; allowing for a model's abilities to further improve without needing to train from scratch. In this work, we detail a set of guidelines that cover how to design efficacious data distributions and learning rate schedules for continued pretraining of language models. When applying these findings within a continued pretraining run on top of a well-trained 15B parameter model, we show an improvement of 9\% in average model accuracy compared to the baseline of continued training on the pretraining set. The resulting recipe provides a practical starting point with which to begin developing language models through reuse rather than retraining.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
