arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.09279cs.CL
收藏

拆解DPO与PPO:理清偏好反馈学习的最佳实践

Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback

Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学)

Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, Hannaneh Hajishirzi

AI总结:

本研究拆解偏好学习四大核心要素,系统对比各组件对语言模型性能的影响,明确了不同要素的贡献排序及PPO相对DPO的性能优势,并公开发布相关代码、模型与数据集。

摘要:

从偏好反馈中学习已成为提升现代语言模型(LM)生成质量与性能的关键步骤。尽管该技术应用广泛,但偏好学习的实施方式差异极大,涉及不同的数据、学习算法与评估方式,导致难以厘清各环节的实际影响。本研究明确了偏好学习的四大核心要素:偏好数据、学习算法、奖励模型与策略训练提示词,系统探究了这些组件对下游模型性能的影响,并提出了一套高效的偏好反馈学习方案。研究结果表明,所有要素均对性能至关重要:其中更优质的偏好数据带来的性能提升幅度最大,其次是学习算法的选择、改进型奖励模型的使用,最后是在策略训练中加入额外无标注提示词的做法。值得注意的是,PPO在数学领域的表现优于DPO最高达2.5%,在通用领域的优势最高达1.2%;高质量偏好数据可让指令遵循能力与真实性最高提升8%。尽管扩大奖励模型规模在数学评估中带来了最高达5%的显著提升,但我们意外发现其在其他类别中的提升微乎其微。我们公开发布了模型训练(https://github.com/hamishivi/EasyLM)与评估(https://github.com/allenai/open-instruct)所用的代码,以及模型和数据集本身(https://huggingface.co/collections/allenai/tulu-v25-suite-66676520fd578080e126f618)。

英文摘要:

Learning from preference feedback has emerged as an essential step for improving the generation quality and performance of modern language models (LMs). Despite its widespread use, the way preference-based learning is applied varies wildly, with differing data, learning algorithms, and evaluations used, making disentangling the impact of each aspect difficult. In this work, we identify four core aspects of preference-based learning: preference data, learning algorithm, reward model, and policy training prompts, systematically investigate the impact of these components on downstream model performance, and suggest a recipe for strong learning for preference feedback. Our findings indicate that all aspects are important for performance, with better preference data leading to the largest improvements, followed by the choice of learning algorithm, the use of improved reward models, and finally the use of additional unlabeled prompts for policy training. Notably, PPO outperforms DPO by up to 2.5% in math and 1.2% in general domains. High-quality preference data leads to improvements of up to 8% in instruction following and truthfulness. Despite significant gains of up to 5% in mathematical evaluation when scaling up reward models, we surprisingly observe marginal improvements in other categories. We publicly release the code used for training (https://github.com/hamishivi/EasyLM) and evaluating (https://github.com/allenai/open-instruct) our models, along with the models and datasets themselves (https://huggingface.co/collections/allenai/tulu-v25-suite-66676520fd578080e126f618).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑