什么造就了优质的对齐数据？指令微调中自动数据选择的综合研究
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning
AI总结：
本文研究指令微调中的自动数据选择，从复杂度、质量和多样性衡量数据，并提出 deita 方法，仅用 6K SFT 样本即达到或超过开源对齐模型的先进水平。
摘要：
指令微调是在初始预训练阶段之后，用于使大型语言模型与终端任务和用户偏好对齐的一项标准技术。近期研究表明，数据工程在指令微调中发挥着关键作用——只要选择得当，仅需有限的数据即可取得卓越性能。然而，对于什么造就了用于对齐的优质指令微调数据，以及我们应如何自动且有效地选择数据，仍缺乏原则性理解。在本工作中，我们深入研究了用于对齐的自动数据选择策略。我们首先开展受控研究，从复杂度、质量和多样性三个维度衡量数据；沿这些维度，我们考察现有方法，并引入新技术以增强数据衡量。随后，我们提出一种基于该衡量来选择数据样本的简单策略。我们展示了 deita（Data-Efficient Instruction Tuning for Alignment 的缩写），这是一系列利用通过我们提出的方法自动选择的数据样本，从 LLaMA 和 Mistral 模型微调得到的模型。实验上，deita 仅使用 6K 个 SFT 训练数据样本，便表现优于或持平于最先进的开源对齐模型——所用数据比基线少 10 倍以上。在进一步使用直接偏好优化（DPO）训练时，使用 6K SFT 和 10K DPO 样本训练的 deita-Mistral-7B + DPO 取得 7.55 的 MT-Bench 分数和 90.06% 的 AlpacaEval 分数。我们预期本工作将提供自动数据选择工具，促进数据高效的对齐。我们发布了模型以及所选数据集，以供未来研究更高效地对齐模型。
英文摘要：
Instruction tuning is a standard technique employed to align large language models to end tasks and user preferences after the initial pretraining phase. Recent research indicates the critical role of data engineering in instruction tuning -- when appropriately selected, only limited data is necessary to achieve superior performance. However, we still lack a principled understanding of what makes good instruction tuning data for alignment, and how we should select data automatically and effectively. In this work, we delve deeply into automatic data selection strategies for alignment. We start with controlled studies to measure data across three dimensions: complexity, quality, and diversity, along which we examine existing methods and introduce novel techniques for enhanced data measurement. Subsequently, we propose a simple strategy to select data samples based on the measurement. We present deita (short for Data-Efficient Instruction Tuning for Alignment), a series of models fine-tuned from LLaMA and Mistral models using data samples automatically selected with our proposed approach. Empirically, deita performs better or on par with the state-of-the-art open-source alignment models with only 6K SFT training data samples -- over 10x less than the data used in the baselines. When further trained with direct preference optimization (DPO), deita-Mistral-7B + DPO trained with 6K SFT and 10K DPO samples achieve 7.55 MT-Bench and 90.06% AlpacaEval scores. We anticipate this work to provide tools on automatic data selection, facilitating data-efficient alignment. We release our models as well as the selected datasets for future researches to effectively align models more efficiently.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
