通过缓解稳定性差距实现高效的持续预训练
Efficient Continual Pre-training by Mitigating the Stability Gap
AI总结：
本文研究持续预训练中的稳定性差距现象，提出子集多轮训练、高质量语料和相似数据混合三种策略，有效提升LLM在新领域的性能并减少遗忘，基于此开发的Llama-3-Physician在医学任务上表现媲美GPT-4。
摘要：
持续预训练已日益成为将大型语言模型（LLMs）适应到新领域的主要方法。这个过程涉及用来自新领域的语料库更新预训练的LLM，导致训练分布发生偏移。为了研究LLMs在这种偏移期间的行为，我们在整个持续预训练过程中测量了模型的性能。我们观察到在开始时出现了暂时的性能下降，随后是恢复阶段，这种现象被称为“稳定性差距”，此前在视觉模型分类新类别时已被注意到。为了解决这个问题并在固定的计算预算内提升LLM性能，我们提出了三种有效的策略：（1）在一个适当大小的子集上对LLM进行多个epoch的持续预训练，这比在大型语料库上单epoch预训练LLM能更快地恢复性能；（2）仅在高质量子语料库上预训练LLM，这能快速提升领域性能；以及（3）使用类似于预训练数据的数据混合来减少分布差距。我们在Llama系列模型上进行了各种实验，以验证我们的策略在医学持续预训练和指令微调中的有效性。例如，我们的策略将OpenLlama-3B模型的平均医学任务性能从36.2%提升到40.7%，仅使用了原始训练预算的40%，并且在不造成遗忘的情况下提升了平均通用任务性能。此外，我们将我们的策略应用于Llama-3-8B模型。生成的模型Llama-3-Physician在当前开源模型中实现了最佳的医学性能，并在几个医学基准上表现与GPT-4相当甚至更好。我们在\url{https://huggingface.co/YiDuo1999/Llama-3-Physician-8B-Instruct}发布了我们的模型。
英文摘要：
Continual pre-training has increasingly become the predominant approach for adapting Large Language Models (LLMs) to new domains. This process involves updating the pre-trained LLM with a corpus from a new domain, resulting in a shift in the training distribution. To study the behavior of LLMs during this shift, we measured the model's performance throughout the continual pre-training process. we observed a temporary performance drop at the beginning, followed by a recovery phase, a phenomenon known as the "stability gap," previously noted in vision models classifying new classes. To address this issue and enhance LLM performance within a fixed compute budget, we propose three effective strategies: (1) Continually pre-training the LLM on a subset with a proper size for multiple epochs, resulting in faster performance recovery than pre-training the LLM on a large corpus in a single epoch; (2) Pre-training the LLM only on high-quality sub-corpus, which rapidly boosts domain performance; and (3) Using a data mixture similar to the pre-training data to reduce distribution gap. We conduct various experiments on Llama-family models to validate the effectiveness of our strategies in both medical continual pre-training and instruction tuning. For example, our strategies improve the average medical task performance of the OpenLlama-3B model from 36.2% to 40.7% with only 40% of the original training budget and enhance the average general task performance without causing forgetting. Furthermore, we apply our strategies to the Llama-3-8B model. The resulting model, Llama-3-Physician, achieves the best medical performance among current open-source models, and performs comparably to or even better than GPT-4 on several medical benchmarks. We release our models at \url{https://huggingface.co/YiDuo1999/Llama-3-Physician-8B-Instruct}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
