大语言模型（LLM）中的偏好对齐显著提升了其遵循人类指令与意图的能力。然而，现有的直接对齐算法主要关注相对偏好，并且往往忽视回复的质量层面，尽管在 AI 反馈过程中可以获得包含评判模型所给奖励分数的偏好数据。力求最大化所选回复与略逊一筹的被拒回复之间的隐式奖励差距，可能导致过拟合，以及对高质量被拒回复产生不必要的遗忘。对奖励分数的无意识还会驱使 LLM 不加区分地偏好低质量的所选回复，并且无法泛化到数据中稀疏存在的最优回复。为克服这些不足，本研究引入了奖励条件化的 LLM 策略，用以识别并学习数据集中完整的回复质量谱系，从而帮助外推到更优的区域。我们提出了一种有效而简单的数据重标注方法，该方法依据质量分数对偏好对进行条件化，以构建奖励增强数据集。跨多个基准和不同模型的实验表明，我们的方法持续且以相当大的幅度提升 DPO。通过全面的消融研究，我们证明该方法不仅最大化了偏好数据的效用，还缓解了遗忘问题，显示出其超越单纯数据扩展的广泛有效性。我们的代码可在 https://github.com/shenao-zhang/reward-augmented-preference 获取。
英文摘要：
Preference alignment in Large Language Models (LLMs) has significantly improved their ability to adhere to human instructions and intentions. However, existing direct alignment algorithms primarily focus on relative preferences and often overlook the qualitative aspects of responses, despite having access to preference data that includes reward scores from judge models during AI feedback. Striving to maximize the implicit reward gap between the chosen and the slightly inferior rejected responses can cause overfitting and unnecessary unlearning of the high-quality rejected responses. The unawareness of the reward scores also drives the LLM to indiscriminately favor the low-quality chosen responses and fail to generalize to optimal responses that are sparse in data. To overcome these shortcomings, our study introduces reward-conditioned LLM policies that discern and learn from the entire spectrum of response quality within the dataset, helping extrapolate to more optimal regions. We propose an effective yet simple data relabeling method that conditions the preference pairs on quality scores to construct a reward-augmented dataset. The experiments across various benchmarks and diverse models demonstrate that our approach consistently boosts DPO by a considerable margin. Through comprehensive ablation studies, we demonstrate that our method not only maximizes the utility of preference data but also mitigates the issue of unlearning, demonstrating its broad effectiveness beyond mere data expansion. Our code is available at https://github.com/shenao-zhang/reward-augmented-preference.
Stanford University（斯坦福大学）；University of California, Berkeley（加州大学伯克利分校）；University of Utah（犹他大学）
Learning a reward function from human preferences is challenging as it typically requires having a high-fidelity simulator or using expensive and potentially unsafe actual physical rollouts in the environment. However, in many tasks the agent might have access to offline data from related tasks in the same target environment. While offline data is increasingly being used to aid policy optimization via offline RL, our observation is that it can be a surprisingly rich source of information for preference learning as well. We propose an approach that uses an offline dataset to craft preference qu…
Learning Optimal Advantage from Preferences and Mistaking it for Reward
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Google Research（谷歌研究院）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；UC Berkeley（加州大学伯克利分校）；Sony AI（索尼人工智能）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）
Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learning
Institute of Engineering Research, Korea University（韩国大学工程研究所）；Kim Jaechul Graduate School of AI, KAIST（韩国科学技术院金在哲人工智能研究生院）；Industrial Management Engineering, Korea University（韩国大学产业管理工程学院）
Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学计算机科学与工程保罗·G·艾伦学校）；Max Planck Institute for Intelligent Systems, Tübingen, Germany（德国图宾根马克斯·普朗克智能系统研究所）；Jinesis Lab, University of Toronto & Vector Institute（多伦多大学Jinesis实验室及向量研究所）
Towards Comprehensive Preference Data Collection for Reward Modeling
Renmin University of China（中国人民大学）；Gaoling School of Artificial Intelligence（高瓴人工智能学院）；Kuaishou Technology（快手科技）；University of Chinese Academy of Sciences（中国科学院大学）
Multi-Response Preference Optimization with Augmented Ranking Dataset
INMED DATA；Yonsei University（延世大学）；Asan Medical Center（峨山医院）；University of Ulsan College of Medicine（蔚山大学医学院）；Asan Institute for Life Sciences（峨山生命科学研究所）
Conveying complex objectives to reinforcement learning (RL) agents often requires meticulous reward engineering. Preference-based RL methods are able to learn a more flexible reward model based on human preferences by actively incorporating human feedback, i.e. teacher's preferences between two clips of behaviors. However, poor feedback-efficiency still remains a problem in current preference-based RL algorithms, as tailored human feedback is very expensive. To handle this issue, previous methods have mainly focused on improving query selection and policy initialization. At the same time, rece…
University of Bristol（布里斯托大学）；CortAIx, Thales（泰雷兹公司 CortAIx）
The potential of reinforcement learning (RL) to deliver aligned and performant agents is partially bottlenecked by the reward engineering problem. One alternative to heuristic trial-and-error is preference-based RL (PbRL), where a reward function is inferred from sparse human feedback. However, prior PbRL methods lack interpretability of the learned reward structure, which hampers the ability to assess robustness and alignment. We propose an online, active preference learning algorithm that constructs reward functions with the intrinsically interpretable, compositional structure of a tree. Usi…
Preference-based Reinforcement Learning (PbRL) methods utilize binary feedback from the human in the loop (HiL) over queried trajectory pairs to learn a reward model in an attempt to approximate the human's underlying reward function capturing their preferences. In this work, we investigate the issue of a high degree of variability in the initialized reward models which are sensitive to random seeds of the experiment. This further compounds the issue of degenerate reward functions PbRL methods already suffer from. We propose a data-driven reward initialization method that does not add any addi…
KAIST（韩国科学技术院）；University of Michigan（密歇根大学）；LG AI Research（LG人工智能研究院）；UC Berkeley（加州大学伯克利分校）
Preference-based reinforcement learning (RL) has shown potential for teaching agents to perform the target tasks without a costly, pre-defined reward function by learning the reward with a supervisor's preference between the two agent behaviors. However, preference-based learning often requires a large amount of human feedback, making it difficult to apply this approach to various applications. This data-efficiency problem, on the other hand, has been typically addressed by using unlabeled samples or data augmentation techniques in the context of supervised learning. Motivated by the recent su…
We define a novel neuro-symbolic framework, argumentative reward learning, which combines preference-based argumentation with existing approaches to reinforcement learning from human feedback. Our method improves prior work by generalising human preferences, reducing the burden on the user and increasing the robustness of the reward model. We demonstrate this with a number of experiments.
PREFINE: Preference-Based Implicit Reward and Cost Fine-Tuning for Safety Alignment
TCS Research, \ of CSE, IIT Madras India；Department of Computing Science, \ of Alberta Canada；Qatar Computing Research Institute, \ Bin Khalifa University Qatar；Department of Data Science \& AI, Wadhwani School of Data Science \& AI, IIT Madras India；TCS Research, \ of CSE, IIT Madras；Department of Computing Science, \ of Alberta；Qatar Computing Research Institute, \ Bin Khalifa University；Department of Data Science \& AI, Wadhwani School of Data Science \& AI, IIT Madras
Specifying rewards for reinforcement learned (RL) agents is challenging. Preference-based RL (PbRL) mitigates these challenges by inferring a reward from feedback over sets of trajectories. However, the effectiveness of PbRL is limited by the amount of feedback needed to reliably recover the structure of the target reward. We present the PRIor Over Rewards (PRIOR) framework, which incorporates priors about the structure of the reward function and the preference feedback into the reward learning process. Imposing these priors as soft constraints on the reward learning objective reduces the amou…
Preference-based reinforcement learning (RL) algorithms help avoid the pitfalls of hand-crafted reward functions by distilling them from human preference feedback, but they remain impractical due to the burdensome number of labels required from the human, even for relatively simple tasks. In this work, we demonstrate that encoding environment dynamics in the reward function (REED) dramatically reduces the number of preference labels required in state-of-the-art preference-based RL frameworks. We hypothesize that REED-based methods better partition the state-action space and facilitate generali…
