arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.07971cs.CLcs.AIcs.LG
收藏

双管齐下:论 RLHF 中奖励模型与策略模型之间的无缝性

It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF

ByteDance(字节跳动)

Taiming Lu, Lingfeng Shen, Xinyu Yang, Weiting Tan, Beidi Chen, Huaxiu Yao

AI总结:

本文提出“无缝性”概念和自动度量指标 SEAM,用于衡量 RLHF 中奖励模型与策略模型判断的差异,并通过数据筛选和模型增强分别提升 RLHF 性能 4.5% 和 4%。

摘要:

基于人类反馈的强化学习(RLHF)涉及训练策略模型(PMs)和奖励模型(RMs),以使语言模型与人类偏好对齐。我们不再仅独立关注 PMs 和 RMs,而是提出在微调过程中考察它们之间的交互,并引入“无缝性”这一概念。我们的研究从观察饱和现象开始,即 RM 和 PM 的持续改进并未转化为 RLHF 的进展。分析表明,RMs 未能为 PM 的响应分配恰当的分数,导致与人类偏好的不匹配率达到 35%,凸显了 PM 与 RM 之间的显著差异。为了在无需人工努力的情况下衡量 PM 与 RM 之间的无缝性,我们提出了一种自动度量指标 SEAM。SEAM 量化了由数据样本引起的 PM 与 RM 判断之间的差异。我们验证了 SEAM 在数据选择和模型增强方面的有效性。实验结果表明:(1)使用经 SEAM 筛选的数据进行 RL 训练可将 RLHF 性能提升 4.5%;(2)与标准增强方法相比,SEAM 引导的模型增强带来了 4% 的性能提升。

英文摘要:

Reinforcement Learning from Human Feedback (RLHF) involves training policy models (PMs) and reward models (RMs) to align language models with human preferences. Instead of focusing solely on PMs and RMs independently, we propose to examine their interactions during fine-tuning, introducing the concept of seamlessness. Our study starts with observing the saturation phenomenon, where continual improvements in RM and PM do not translate into RLHF progress. Our analysis shows that RMs fail to assign proper scores to PM responses, resulting in a 35% mismatch rate with human preferences, highlighting a significant discrepancy between PM and RM. To measure seamlessness between PM and RM without human effort, we propose an automatic metric, SEAM. SEAM quantifies the discrepancies between PM and RM judgments induced by data samples. We validate the effectiveness of SEAM in data selection and model augmentation. Our experiments demonstrate that (1) using SEAM-filtered data for RL training improves RLHF performance by 4.5%, and (2) SEAM-guided model augmentation results in a 4% performance improvement over standard augmentation methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑