arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.10985cs.CLcs.AI
收藏

当数据成为算法:对偏好优化数据集的系统研究与整理

When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets

Technical University Munich(慕尼黑技术大学);IBM Research(IBM研究院)

Aladin Djuhera, Farhan Ahmed, Swanand Ravindra Kadhe, Syed Zawad, Heiko Ludwig, Holger Boche

AI总结:

本文系统研究并整理了多个开源DPO数据集,提出UltraMix混合数据集,通过Magpie框架进行精细标注,揭示偏好质量差异,并在关键基准上超越最佳单个数据集性能。

摘要:

对大型语言模型(LLMs)进行对齐是训练后阶段的核心目标,通常通过奖励建模和强化学习方法实现。其中,直接偏好优化(DPO)已成为一种广泛采用的技术,通过在更受青睐的完成语和不如意的完成语之间微调LLMs。尽管大多数前沿LLMs不公开其整理的偏好对,但更广泛的LLM社区已发布了几个开源DPO数据集,包括TuluDPO、ORPO、UltraFeedback、HelpSteer和Code-Preference-Pairs。然而,系统性的比较仍然稀缺,这主要是由于计算成本高以及缺乏丰富的质量注释,使得难以理解偏好是如何被选择的,它们覆盖了哪些任务类型,以及它们在单个样本层面如何反映人类判断。在本工作中,我们提出了对流行开源DPO语料库的首次全面、数据导向的分析。我们利用Magpie框架对每个样本进行标注,以任务类别、输入质量和偏好奖励,这是一种基于奖励模型的信号,可以验证偏好顺序,而无需依赖人工注释。这使我们能够对不同数据集的偏好质量进行可扩展的、细致的检查,揭示出奖励边际的结构和质量差异。基于这些见解,我们系统地整理了一个新的DPO混合物,UltraMix,它从所有五个语料库中选择性地提取,同时去除噪声或冗余的样本。UltraMix比表现最好的单个数据集小30%,但在关键基准上超过了其性能。我们公开发布所有注释、元数据和我们整理的混合物,以促进未来数据导向的偏好优化研究。

英文摘要:

Aligning large language models (LLMs) is a central objective of post-training, often achieved through reward modeling and reinforcement learning methods. Among these, direct preference optimization (DPO) has emerged as a widely adopted technique that fine-tunes LLMs on preferred completions over less favorable ones. While most frontier LLMs do not disclose their curated preference pairs, the broader LLM community has released several open-source DPO datasets, including TuluDPO, ORPO, UltraFeedback, HelpSteer, and Code-Preference-Pairs. However, systematic comparisons remain scarce, largely due to the high computational cost and the lack of rich quality annotations, making it difficult to understand how preferences were selected, which task types they span, and how well they reflect human judgment on a per-sample level. In this work, we present the first comprehensive, data-centric analysis of popular open-source DPO corpora. We leverage the Magpie framework to annotate each sample for task category, input quality, and preference reward, a reward-model-based signal that validates the preference order without relying on human annotations. This enables a scalable, fine-grained inspection of preference quality across datasets, revealing structural and qualitative discrepancies in reward margins. Building on these insights, we systematically curate a new DPO mixture, UltraMix, that draws selectively from all five corpora while removing noisy or redundant samples. UltraMix is 30% smaller than the best-performing individual dataset yet exceeds its performance across key benchmarks. We publicly release all annotations, metadata, and our curated mixture to facilitate future research in data-centric preference optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑