无需指令微调的指令遵循
Instruction Following without Instruction Tuning
AI总结：
本文提出隐式指令微调，发现仅在响应上训练或在窄领域数据上训练均可产生指令遵循，并假设对语言模型分布进行简单改变即可实现该能力。
摘要：
指令微调通常指在指令-响应对上微调语言模型。我们发现了两种与指令微调相比存在不足、但仍能产生指令遵循的适应（微调）形式；我们称之为隐式指令微调。我们首先发现指令-响应对并非必需：仅在响应上进行训练，而没有任何对应的指令，就能产生指令遵循。这表明预训练模型具有指令-响应映射，该映射通过向模型教授期望的响应分布而被揭示。然而，我们随后发现教授期望的响应分布并非必需：在诗歌等窄领域数据上进行指令-响应训练，仍会导致像食谱生成这样广泛的指令遵循行为。特别是，当指令与窄微调领域中的指令截然不同时，模型的响应并不遵循微调领域的风格。为了初步解释隐式指令微调，我们假设对语言模型分布进行非常简单的改变即可产生指令遵循。我们通过手写一个基于规则的语言模型来支持这一点，该模型在与预训练模型的专家乘积中产生了指令遵循。这些规则是缓慢增加结束序列的概率，惩罚重复，并均匀改变15个词的概率。总之，未被设计为产生指令遵循的适应可以隐式地做到这一点。
英文摘要：
Instruction tuning commonly means finetuning a language model on instruction-response pairs. We discover two forms of adaptation (tuning) that are deficient compared to instruction tuning, yet still yield instruction following; we call this implicit instruction tuning. We first find that instruction-response pairs are not necessary: training solely on responses, without any corresponding instructions, yields instruction following. This suggests pretrained models have an instruction-response mapping which is revealed by teaching the model the desired distribution of responses. However, we then find it's not necessary to teach the desired distribution of responses: instruction-response training on narrow-domain data like poetry still leads to broad instruction-following behavior like recipe generation. In particular, when instructions are very different from those in the narrow finetuning domain, models' responses do not adhere to the style of the finetuning domain. To begin to explain implicit instruction tuning, we hypothesize that very simple changes to a language model's distribution yield instruction following. We support this by hand-writing a rule-based language model which yields instruction following in a product-of-experts with a pretrained model. The rules are to slowly increase the probability of ending the sequence, penalize repetition, and uniformly change 15 words' probabilities. In summary, adaptations made without being designed to yield instruction following can do so implicitly.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
