指令预训练：语言模型是有监督多任务学习者
Instruction Pre-Training: Language Models are Supervised Multitask Learners
AI总结：
针对有监督多任务学习的潜力，本文提出指令预训练框架，通过开源模型构建的高效合成器生成2亿条指令-响应对增强语料库预训练语言模型，实验表明其能增强基础模型、提升指令调优收益，还能让Llama3-8B媲美甚至超越Llama3-70B。
摘要：
无监督多任务预训练是近期语言模型（LMs）成功的关键方法。然而，有监督多任务学习仍具有重要潜力，因为在训练后阶段对其进行扩展往往能带来更好的泛化性。本文通过提出指令预训练（Instruction Pre-Training）探索有监督多任务预训练，这是一个可扩展地用指令-响应对增强大规模原始语料库以预训练语言模型的框架。指令-响应对由基于开源模型构建的高效指令合成器生成。在实验中，我们合成了覆盖40多个任务类别的2亿条指令-响应对，以验证指令预训练的有效性。在从头预训练中，指令预训练不仅持续增强预训练基础模型，还能从进一步的指令调优中获得更多收益；在持续预训练中，它使Llama3-8B可与Llama3-70B媲美甚至超越。我们的模型、代码和数据可在https://github.com/microsoft/LMOps获取。
英文摘要：
Unsupervised multitask pre-training has been the critical method behind the recent success of language models (LMs). However, supervised multitask learning still holds significant promise, as scaling it in the post-training stage trends towards better generalization. In this paper, we explore supervised multitask pre-training by proposing Instruction Pre-Training, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train LMs. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. In our experiments, we synthesize 200M instruction-response pairs covering 40+ task categories to verify the effectiveness of Instruction Pre-Training. In pre-training from scratch, Instruction Pre-Training not only consistently enhances pre-trained base models but also benefits more from further instruction tuning. In continual pre-training, Instruction Pre-Training enables Llama3-8B to be comparable to or even outperform Llama3-70B. Our model, code, and data are available at https://github.com/microsoft/LMOps.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
