arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.19443cs.CL
收藏

混合偏好优化:基于数据选择与更优参考模型的强化学习

Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model

State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室)

Qi Gou, Cam-Tu Nguyen

AI总结:

提出MPO方法,结合DPO和RLHF两阶段训练,利用奖励模型划分难易数据集,以缓解分布偏移,提升LLM对齐效果,并在HH-RLHF和TLDR数据集上验证有效性。

摘要:

大型语言模型(LLMs)因其处理和生成自然语言的能力而日益流行。然而,由于它们在大量文本数据集上进行训练,LLMs可能继承有害的偏见,并产生与人类价值观不一致的输出。本文研究了两种主要的LLM对齐方法:基于人类反馈的强化学习(RLHF)和基于对比学习的方法,如直接偏好优化(DPO)。通过分析RLHF和DPO的稳定性和鲁棒性,我们提出了MPO(混合偏好优化),一种新颖的方法,旨在缓解这两种方法的弱点。具体来说,我们提出了一种两阶段训练流程:首先在简单数据集上训练DPO,然后在困难数据集上以DPO模型作为参考模型执行RLHF。这里,简单和困难数据集由训练良好的奖励模型构建,该模型将响应对分为奖励差距大的(简单)和奖励差距小的(困难)。第一阶段使我们能够快速获得相对最优的策略(LLM)模型,而第二阶段通过在线RLHF细化LLM,从而缓解与DPO相关的分布偏移问题。实验在两个公开的对齐数据集(即HH-RLHF和TLDR)上进行,展示了MPO在GPT4和人工评估方面的有效性。

英文摘要:

Large Language Models (LLMs) have become increasingly popular due to their ability to process and generate natural language. However, as they are trained on massive datasets of text, LLMs can inherit harmful biases and produce outputs that are not aligned with human values. This paper studies two main approaches to LLM alignment: Reinforcement Learning with Human Feedback (RLHF) and contrastive learning-based methods like Direct Preference Optimization (DPO). By analyzing the stability and robustness of RLHF and DPO, we propose MPO (Mixed Preference Optimization), a novel method that mitigates the weaknesses of both approaches. Specifically, we propose a two-stage training procedure: first train DPO on an easy dataset, and then perform RLHF on a difficult set with DPO model being the reference model. Here, the easy and difficult sets are constructed by a well-trained reward model that splits response pairs into those with large gaps of reward (easy), and those with small gaps (difficult). The first stage allows us to obtain a relatively optimal policy (LLM) model quickly, whereas the second stage refines LLM with online RLHF, thus mitigating the distribution shift issue associated with DPO. Experiments are conducted on two public alignment datasets, namely HH-RLHF and TLDR, demonstrating the effectiveness of MPO, both in terms of GPT4 and human evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑