超越标量奖励模型：从偏好数据中学习生成式评判器
Beyond Scalar Reward Model: Learning Generative Judge from Preference Data
AI总结：
针对传统标量奖励模型可解释性不足、易受数据偏差影响的问题，提出用自生成对比评判（Con-J）结合DPO训练生成式评判器，性能与标量模型相当且更具可解释性和鲁棒性。
摘要：
从偏好反馈中学习是使大语言模型（LLM）与人类价值观对齐的常用做法。传统方法中，偏好数据被学习并编码为标量奖励模型，该模型将价值头与LLM相连，输出标量分数作为偏好或奖励。然而，标量模型缺乏可解释性，且已知易受数据集中偏差的影响。本文研究利用LLM的生成能力一次性解决这两个局限。具体而言，我们提示预训练LLM生成正面和负面评判，二者均配有自然语言形式的理由。自生成的对比评判对被用于通过直接偏好优化（DPO）训练生成式评判器。这种使用自生成对比评判（Con-J）训练生成式评判器的方案，由于生成的理由与评判相伴，保证了自然的可解释性，同时无需额外奖励头就具备对偏差的高鲁棒性。实验结果表明，Con-J的性能与在相同偏好数据集上训练的标量奖励模型相当，并展现出其在编码人类偏好方面更优的可解释性和鲁棒性。
英文摘要：
Learning from preference feedback is a common practice for aligning large language models~(LLMs) with human value. Conventionally, preference data is learned and encoded into a scalar reward model that connects a value head with an LLM to produce a scalar score as preference or reward. However, scalar models lack interpretability and are known to be susceptible to biases in datasets. This paper investigates leveraging the generation capability of LLMs to address both limitations in one shot. Specifically, we prompt the pre-trained LLM to generate positive and negative judgments, both supported with rationales in natural language form. The self-generated contrastive judgment pairs are used to train the generative judge with Direct Preference Optimization (DPO). This proposal of training the generative Judge using self-generated Contrastive judgments (Con-J) ensures natural interpretability due to the generated rationales together with the judgments, as well as high robustness against bias without the need for an additional reward head. Experimental results show that the performance of Con-J is comparable to the scalar reward model trained on the same collection of preference data, and demonstrate its superior interpretability and robustness in encoding human preferences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
