离线RLHF中样本重加权的奖励差优化
Reward Difference Optimization For Sample Reweighting In Offline RLHF
AI总结：
针对现有离线RLHF忽略回复间偏好程度的问题，提出奖励差优化方法，通过奖励差系数重加权样本对并构建差模型预测系数，在7B大模型及相关数据集上验证了其对齐效果。
摘要：
随着大语言模型（LLMs）的快速发展，使LLMs与人类偏好对齐变得愈发重要。尽管基于人类反馈的强化学习（RLHF）已被证明有效，但其流程复杂且资源消耗极高。为此，离线RLHF作为替代方案被提出，它直接在固定偏好数据集上通过排序损失优化LLMs。现有离线RLHF仅捕捉回复间的“顺序关系”，却忽略了一个回复相较于其他回复的偏好程度这一关键维度。为解决该问题，我们提出一种简单有效的方法——奖励差优化（Reward Difference Optimization，简称RDO）。具体而言，我们引入奖励差系数，对离线RLHF中的样本对进行重加权；随后构建一个差模型，捕捉一对回复间的丰富交互以预测这些差系数。在HH和TL;DR数据集上针对7B规模LLMs开展的实验，从自动指标和人工评估两方面均验证了该方法的有效性，凸显了其在使LLMs对齐人类意图与价值观方面的潜力。
英文摘要：
With the rapid advances in Large Language Models (LLMs), aligning LLMs with human preferences become increasingly important. Although Reinforcement Learning with Human Feedback (RLHF) proves effective, it is complicated and highly resource-intensive. As such, offline RLHF has been introduced as an alternative solution, which directly optimizes LLMs with ranking losses on a fixed preference dataset. Current offline RLHF only captures the "ordinal relationship" between responses, overlooking the crucial aspect of how much one is preferred over the others. To address this issue, we propose a simple yet effective solution called Reward Difference Optimization, shorted as RDO. Specifically, we introduce reward difference coefficients to reweigh sample pairs in offline RLHF. We then develop a difference model which captures rich interactions between a pair of responses for predicting these difference coefficients. Experiments with 7B LLMs on the HH and TL;DR datasets substantiate the effectiveness of our method in both automatic metrics and human evaluation, thereby highlighting its potential for aligning LLMs with human intent and values
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
