非指令性微调：在无指令遵循数据的情况下赋予预训练语言模型指令遵循能力
Non-instructional Fine-tuning: Enabling Instruction-Following Capabilities in Pre-trained Language Models without Instruction-Following Data
AI总结：
本文提出用OpenWebText随机文本前半段作指令、GPT-3.5/4-turbo补全作响应的非指令性数据微调预训练LLMs，验证其能获得指令遵循能力，且LLaMA-3-70B-Instruct在Arena Hard上可比肩LLaMA-3.1-70B-Instruct。
摘要：
指令微调对于当今的大语言模型（LLMs）学会遵循指令并与人类偏好对齐至关重要。传统上，指令微调需要监督数据，包括指令和正确响应。为了获得此类数据，一些研究者使用GPT-4等训练良好的模型来生成指令和正确响应。在本文中，我们提出了一种新颖的方法，将来自OpenWebText的随机文本的前半部分作为指令，并使用GPT-3.5-turbo或GPT-4-turbo补全文本作为响应。尽管数据是“非指令性”的，我们发现在此数据上微调的预训练LLMs能够获得指令遵循能力。这一观察通过对多个知名预训练LLMs（例如LLaMA-2-7B、LLaMA-3-8B、LLaMA-3-70B、Mistral-7B-v0.1）进行微调得到了验证。“非指令性数据”还改善了一些经过监督微调和人类偏好对齐的模型。我们通过“非指令性数据”微调的LLaMA-3-70B-Instruct在Arena Hard排行榜上与LLaMA-3.1-70B-Instruct相当。我们分析了“非指令性数据”，并确保其不包含与指令微调相关的内容。我们的发现将激发进一步研究如何在无需显式指令相关数据的情况下发展指令遵循能力。
英文摘要：
Instruction fine-tuning is crucial for today's large language models (LLMs) to learn to follow instructions and align with human preferences. Conventionally, supervised data, including the instruction and the correct response, is required for instruction fine-tuning. To obtain such data, some researchers prompted well-trained models like GPT-4 to generate instructions and correct responses. In this paper, we propose a novel approach that uses the first half of a random text from OpenWebText as the instruction and GPT-3.5-turbo or GPT-4-turbo to complete the text as the response. Despite the data being "non-instructional", we found that pre-trained LLMs fine-tuned on this data can gain instruction-following capabilities. This observation is verified by fine-tuning several well-known pre-trained LLMs (e.g., LLaMA-2-7B, LLaMA-3-8B, LLaMA-3-70B, Mistral-7B-v0.1). The "non-instructional data" also improved some models that underwent supervised fine-tuning and human preference alignment. Our LLaMA-3-70B-Instruct fine-tuned through "non-instructional data" is comparable with LLaMA-3.1-70B-Instruct on the Arena Hard leaderboard. We analyzed the "non-instructional data" and ensured it is devoid of content related to instruction fine-tuning. Our findings will inspire further investigation into how to develop instruction-following capabilities without explicit instruction-related data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
