arXiv 2508.041492026-05-16cs.CLcs.AIcs.LG

基于难度的偏好数据选择：通过DPO隐式奖励差距

Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap

Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学计算机科学与工程保罗·G·艾伦学校）；Max Planck Institute for Intelligent Systems, Tübingen, Germany（德国图宾根马克斯·普朗克智能系统研究所）；Jinesis Lab, University of Toronto & Vector Institute（多伦多大学Jinesis实验室及向量研究所）

Xuan Qi, Rongwu Xu, Zhijing Jin

备注：: Our code and data are available at https://github.com/Difficulty-Based-Preference-Data-Select/Difficulty-Based-Preference-Data-Select

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出基于难度的偏好数据选择方法，利用DPO隐式奖励机制选择奖励差距小的样本，提升数据效率和模型对齐性能，在多个数据集和对齐任务中优于五个基线方法。

摘要：

对齐大语言模型（LLMs）与人类偏好是AI研究中的关键挑战。尽管强化学习从人类反馈（RLHF）和直接偏好优化（DPO）等方法被广泛使用，但它们通常依赖于大规模、成本高的偏好数据集。本文缺少针对偏好数据的高质量数据选择方法。在本文中，我们引入了一种基于难度的偏好数据选择策略，该策略基于DPO隐式奖励机制。通过选择奖励差距较小的偏好数据示例，这些示例代表更具挑战性的案例，从而提高数据效率和模型对齐。我们的方法在多个数据集和对齐任务中一致优于五个强大的基线方法，仅使用原始数据的10%即可实现优越性能。这种原理上高效的选择方法为在有限资源下扩展LLM对齐提供了有前景的解决方案。

英文摘要：

Aligning large language models (LLMs) with human preferences is a critical challenge in AI research. While methods like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are widely used, they often rely on large, costly preference datasets. The current work lacks methods for high-quality data selection specifically for preference data. In this work, we introduce a novel difficulty-based data selection strategy for preference datasets, grounded in the DPO implicit reward mechanism. By selecting preference data examples with smaller DPO implicit reward gaps, which are indicative of more challenging cases, we improve data efficiency and model alignment. Our approach consistently outperforms five strong baselines across multiple datasets and alignment tasks, achieving superior performance with only 10\% of the original data. This principled, efficient selection method offers a promising solution for scaling LLM alignment with limited resources.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。