面向大语言模型的分阶段指令微调
Phased Instruction Fine-Tuning for Large Language Models
AI总结：
本文提出分阶段指令微调（Phased IFT），按 GPT-4 评估的难度递增顺序训练模型，实验证明其显著优于一次性指令微调，为高效增强大语言模型指令遵循能力提供了简单方案。
摘要：
指令微调将预训练语言模型从基础的下一个词预测提升为复杂的指令遵循。然而，现有的应用于多样化指令上的一次性指令微调（One-off IFT）方法，由于同时处理不同复杂度的指令，可能无法有效提升模型对指令的遵循能力。为改进这一问题，本文提出分阶段指令微调（Phased IFT），其核心思想是学习遵循指令是一个渐进过程。该方法使用 GPT-4 评估指令难度，将指令数据划分为难度递增的子集，并在这些子集上对模型进行顺序增量训练。使用 Alpaca 数据在 Llama-2 7B/13B/70B、Llama3 8/70B 和 Mistral-7B 模型上的实验表明，Phased IFT 显著优于 One-off IFT，支持渐进对齐假设，并为增强大语言模型提供了一种简单高效的方法。实验代码和数据集可在 https://github.com/xubuvd/PhasedSFT 免费获取。
英文摘要：
Instruction Fine-Tuning enhances pre-trained language models from basic next-word prediction to complex instruction-following. However, existing One-off Instruction Fine-Tuning (One-off IFT) method, applied on a diverse instruction, may not effectively boost models' adherence to instructions due to the simultaneous handling of varying instruction complexities. To improve this, Phased Instruction Fine-Tuning (Phased IFT) is proposed, based on the idea that learning to follow instructions is a gradual process. It assesses instruction difficulty using GPT-4, divides the instruction data into subsets of increasing difficulty, and uptrains the model sequentially on these subsets. Experiments with Llama-2 7B/13B/70B, Llama3 8/70B and Mistral-7B models using Alpaca data show that Phased IFT significantly outperforms One-off IFT, supporting the progressive alignment hypothesis and providing a simple and efficient way to enhance large language models. Codes and datasets from our experiments are freely available at https://github.com/xubuvd/PhasedSFT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
