超越样本级反馈:使用参考级反馈指导数据合成
Beyond Sample-Level Feedback: Using Reference-Level Feedback to Guide Data Synthesis
AI总结:
提出Reference-Level Feedback范式,通过从精选参考样本提取理想特征指导数据合成,构建了REFED数据集,使微调后的模型在同等规模下达到SOTA性能,突破了合成数据的质量上限。
摘要:
高质量的指令微调数据对于开发能够有效处理真实世界任务并遵循人类指令的大型语言模型(LLMs)至关重要。虽然合成数据生成为创建此类数据集提供了可扩展的方法,但它设定了质量上限,即在该数据上训练的模型无法超越生成它的LLM。为了克服这一限制,我们引入了Reference-Level Feedback(参考级反馈),这是一种从精心挑选的参考样本中提取理想特征以指导更高质量指令-响应对合成的范式。使用该方法,我们合成了REFED,一个包含10K指令-响应对的数据集。在REFED上对Llama-3.1-8B-Instruct和Mistral-7B-Instruct进行微调,在同等规模的模型中展现出最先进的性能,特别是在AlpacaEval 2.0上达到了43.96%的长度控制胜率。大量实验表明,Reference-Level Feedback始终优于传统的样本级反馈方法,能跨模型架构泛化,并以低成本生成高质量且多样化的数据。
英文摘要:
High-quality instruction-tuning data is crucial for developing Large Language Models (LLMs) that can effectively navigate real-world tasks and follow human instructions. While synthetic data generation offers a scalable approach for creating such datasets, it imposes a quality ceiling where models trained on the data cannot outperform the LLM generating it. To overcome this limitation, we introduce Reference-Level Feedback, a paradigm that extracts desirable characteristics from carefully curated reference samples to guide the synthesis of higher-quality instruction-response pairs. Using this approach, we synthesize REFED, a dataset of 10K instruction-response pairs. Fine-tuning Llama-3.1-8B-Instruct and Mistral-7B-Instruct on REFED demonstrate state-of-the-art performance among similarly sized models, notably reaching a 43.96\% length-controlled win-rate on AlpacaEval 2.0. Extensive experiments demonstrate that Reference-Level Feedback consistently outperforms traditional sample-level feedback methods, generalizes across model architectures, and produces high-quality and diverse data at low cost.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。