基于指令部分损失函数的指令微调
Instruction Tuning With Loss Over Instructions
AI总结：
本文提出指令建模（IM）方法，将损失函数应用于指令部分而非仅输出部分，在21个基准上验证其能显著提升语言模型性能，尤其在低资源指令微调场景下效果突出。
摘要：
指令微调在将语言模型（LM）的输出塑造为所需风格方面起着至关重要的作用。在这项工作中，我们提出了一种简单而有效的方法，称为指令建模（Instruction Modelling，IM），该方法通过将损失函数应用于指令和提示部分而非仅应用于输出部分来训练语言模型。通过在21个多样化基准上的实验，我们表明，在许多场景下，IM能够有效提升语言模型在自然语言处理任务（如MMLU、TruthfulQA和HumanEval）以及开放式生成基准（如MT-Bench和AlpacaEval）上的性能。值得注意的是，在最有利的情况下，IM将模型在AlpacaEval 1.0上的性能提升了超过100%。我们确定了影响IM有效性的两个关键因素：（1）训练数据中指令长度与输出长度的比例；（2）训练样本的数量。我们观察到，当使用具有冗长指令搭配简短输出的数据集进行训练时，或者在表层对齐假设（Superficial Alignment Hypothesis，SAH）下使用少量训练样本进行指令微调时，IM尤其有益。进一步的分析证实了我们的假设，即我们的改进可以归因于减少了对指令微调数据集的过拟合。值得指出的是，我们并不是提出将IM作为当前微调流程的替代方案。相反，我们的工作旨在为指令微调语言模型提供实用指导，尤其是在低资源场景下。
英文摘要：
Instruction tuning plays a crucial role in shaping the outputs of language models (LMs) to desired styles. In this work, we propose a simple yet effective method, Instruction Modelling (IM), which trains LMs by applying a loss function to the instruction and prompt part rather than solely to the output part. Through experiments across 21 diverse benchmarks, we show that, in many scenarios, IM can effectively improve the LM performance on both NLP tasks (e.g., MMLU, TruthfulQA, and HumanEval) and open-ended generation benchmarks (e.g., MT-Bench and AlpacaEval). Remarkably, in the most advantageous case, IM boosts model performance on AlpacaEval 1.0 by over 100%. We identify two key factors influencing the effectiveness of IM: (1) The ratio between instruction length and output length in the training data; and (2) The number of training examples. We observe that IM is especially beneficial when trained on datasets with lengthy instructions paired with brief outputs, or under the Superficial Alignment Hypothesis (SAH) where a small amount of training examples are used for instruction tuning. Further analysis substantiates our hypothesis that our improvement can be attributed to reduced overfitting to instruction tuning datasets. It is worth noting that we are not proposing \ours as a replacement for current fine-tuning processes. Instead, our work aims to provide practical guidance for instruction tuning LMs, especially in low-resource scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
