长文本更有利于对齐：一种简单但难以超越的指令微调基线
Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning
AI总结：
本文提出一种极简基线方法，通过从标准数据集中选择响应最长的1,000条指令进行指令微调，在GPT-4和PaLM-2评判下持续优于LIMA和AlpaGasus等复杂方法，并在多个基准上取得竞争力结果。
摘要：
目前存在一个共识，即 LLM 的指令微调需要高质量数据，但高质量数据究竟是什么？LIMA (NeurIPS 2023) 和 AlpaGasus (ICLR 2024) 是用于选择此类高质量样本的最先进方法，它们要么通过人工筛选，要么使用 GPT-3.5-Turbo 作为质量评分器。我们表明，从标准数据集中选择响应最长的 1,000 条指令——直观上包含更多可学习信息且更难过拟合——这一极其简单的基线，根据 GPT-4 和 PaLM-2 作为评判者，能够持续优于这些复杂方法，同时在测试事实知识的 Open LLM 基准上保持竞争力。我们在多个 LLM（Llama-2-7B、Llama-2-13B、Mistral-7B-v0.1）和数据集（Alpaca-52k、Evol-Instruct-70k）上证明了这一点。此外，对这类长指令进行轻量级精炼可以进一步提升微调后 LLM 的能力，使我们能够在 MT-Bench 上获得有竞争力的结果，并在 AlpacaEval 2.0 上获得基于 Llama-2-7B 的排名第二的模型，而训练仅使用 1,000 个样本且无需额外偏好数据。我们还对模型进行了全面分析，以确保其性能提升并非单纯由于 GPT-4 对更长响应的偏好。总体而言，我们的发现表明，对最长响应进行微调应成为任何指令微调工作的默认基线。我们在 https://github.com/tml-epfl/long-is-more-for-alignment 提供代码。
英文摘要：
There is a consensus that instruction fine-tuning of LLMs requires high-quality data, but what are they? LIMA (NeurIPS 2023) and AlpaGasus (ICLR 2024) are state-of-the-art methods for selecting such high-quality examples, either via manual curation or using GPT-3.5-Turbo as a quality scorer. We show that the extremely simple baseline of selecting the 1,000 instructions with longest responses -- that intuitively contain more learnable information and are harder to overfit -- from standard datasets can consistently outperform these sophisticated methods according to GPT-4 and PaLM-2 as judges, while remaining competitive on the Open LLM benchmarks that test factual knowledge. We demonstrate this for several LLMs (Llama-2-7B, Llama-2-13B, Mistral-7B-v0.1) and datasets (Alpaca-52k, Evol-Instruct-70k). In addition, a lightweight refinement of such long instructions can further improve the abilities of the fine-tuned LLMs, and allows us to obtain competitive results on MT-Bench and the 2nd highest-ranked Llama-2-7B-based model on AlpacaEval 2.0, while training on only 1,000 examples and no extra preference data. We also conduct a thorough analysis of our models to ensure that their enhanced performance is not simply due to GPT-4's preference for longer responses. Overall, our findings suggest that fine-tuning on the longest responses should be the default baseline for any work on instruction fine-tuning. We provide our code at https://github.com/tml-epfl/long-is-more-for-alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
