持续预训练大语言模型的简单且可扩展策略
Simple and Scalable Strategies to Continually Pre-train Large Language Models
AI总结：
本文提出结合学习率重新预热、重新衰减和旧数据回放的简单可扩展持续预训练策略，在405M和10B参数LLM上以少量计算达到与完全重新训练相当的性能，并进一步提出替代余弦学习率调度的方法以缓解遗忘。
摘要：
大语言模型（LLMs）通常会在数十亿 token 上进行预训练，但一旦有新数据可用，就不得不从头重新开始这一过程。一种更高效的解决方案是对这些模型进行持续预训练，与重新训练相比可节省大量计算。然而，新数据带来的分布偏移通常会导致模型在旧数据上的性能下降，或对新数据的适应能力不佳。在本工作中，我们证明，将学习率（LR）重新预热、学习率重新衰减以及旧数据回放进行简单且可扩展的组合，就足以在所有可用数据上达到与完全从头重新训练相匹配的性能，衡量指标包括最终损失以及在多个语言模型（LM）评估基准上的平均得分。具体而言，我们在 405M 参数规模的模型上，使用大规模数据集（数千亿 token），针对两个常用 LLM 预训练数据集之间较弱但现实的分布偏移（英语→英语）以及更强的分布偏移（英语→德语）验证了这一点。在更大规模实验中选择较弱但现实的偏移后，我们还发现，对于 10B 参数的 LLM，我们的持续学习策略能够达到与重新训练基线相当的性能。我们的结果表明，LLM 可以通过简单且可扩展的持续学习策略成功更新，仅使用重新训练所需计算量的一小部分即可达到与重新训练基线相当的效果。最后，受先前工作的启发，我们提出了余弦学习率调度方案的替代方案，这些替代方案有助于避免由学习率重新预热引发的遗忘，并且不受固定 token 预算的限制。
英文摘要：
Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes available. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (English$\rightarrow$English) and a stronger distribution shift (English$\rightarrow$German) at the $405$M parameter model scale with large dataset sizes (hundreds of billions of tokens). Selecting the weak but realistic shift for larger-scale experiments, we also find that our continual learning strategies match the re-training baseline for a 10B parameter LLM. Our results demonstrate that LLMs can be successfully updated via simple and scalable continual learning strategies, matching the re-training baseline using only a fraction of the compute. Finally, inspired by previous work, we propose alternatives to the cosine learning rate schedule that help circumvent forgetting induced by LR re-warming and that are not bound to a fixed token budget.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
