Magpie：通过向对齐后的大语言模型输入空提示从零合成对齐数据
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
AI总结：
针对对齐数据不公开、现有开源数据构建难扩展的问题，提出Magpie自合成方法，通过空提示从已对齐LLM生成高质量指令数据，实验证明其微调效果优于现有公开数据集，部分任务性能比肩官方对齐模型。
摘要：
高质量指令数据对大语言模型（LLM）的对齐至关重要。尽管Llama-3-Instruct等部分模型已开源权重，但其对齐数据仍未公开，这阻碍了AI的民主化进程。现有开源数据构建方法存在人力成本高、提示词预设范围有限的问题，难以有效扩展，可能会限制公开对齐数据集的多样性与质量。能否直接从已对齐的LLM中提取数据，大规模合成高质量指令数据？我们提出了一种名为Magpie的自合成方法，用于生成大规模对齐数据。我们的核心发现是：由于Llama-3-Instruct等已对齐LLM的自回归特性，仅输入到用户消息预留位置为止的左侧模板时，模型就能生成用户查询。我们使用该方法提示Llama-3-Instruct，生成了400万条指令及其对应回复。我们对提取的数据进行了全面分析，并筛选出30万条高质量样本。为将Magpie数据与其他公开指令数据集进行对比，我们分别用各数据集微调Llama-3-8B-Base，并评估微调后模型的性能。结果表明，在部分任务中，用Magpie微调的模型性能与官方Llama-3-8B-Instruct相当，而后者是通过1000万数据点的监督微调（SFT）及后续反馈学习增强得到的。我们还发现，仅将Magpie用于SFT，性能就能超过此前同时用于SFT和偏好优化（如基于UltraFeedback的直接偏好优化）的公开数据集。这一优势在AlpacaEval、ArenaHard和WildBench等对齐基准上表现明显。
英文摘要：
High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent existing open-source data creation methods from scaling effectively, potentially limiting the diversity and quality of public alignment datasets. Is it possible to synthesize high-quality instruction data at scale by extracting it directly from an aligned LLM? We present a self-synthesis method for generating large-scale alignment data named Magpie. Our key observation is that aligned LLMs like Llama-3-Instruct can generate a user query when we input only the left-side templates up to the position reserved for user messages, thanks to their auto-regressive nature. We use this method to prompt Llama-3-Instruct and generate 4 million instructions along with their corresponding responses. We perform a comprehensive analysis of the extracted data and select 300K high-quality instances. To compare Magpie data with other public instruction datasets, we fine-tune Llama-3-8B-Base with each dataset and evaluate the performance of the fine-tuned models. Our results indicate that in some tasks, models fine-tuned with Magpie perform comparably to the official Llama-3-8B-Instruct, despite the latter being enhanced with 10 million data points through supervised fine-tuning (SFT) and subsequent feedback learning. We also show that using Magpie solely for SFT can surpass the performance of previous public datasets utilized for both SFT and preference optimization, such as direct preference optimization with UltraFeedback. This advantage is evident on alignment benchmarks such as AlpacaEval, ArenaHard, and WildBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
