用于大语言模型对齐的基于先验约束的奖励模型训练
Prior Constraints-based Reward Model Training for Aligning Large Language Models
AI总结：
针对大语言模型对齐中奖励模型训练无约束导致强化学习阶段奖励分数失控缩放的问题，提出基于先验约束的PCRM训练方法，引入长度比、余弦相似度约束优化，显著提升对齐效果且易集成到各类排序类对齐方法中。
摘要：
基于人类反馈的强化学习用于对齐大语言模型（LLM）时，通常使用带比较对的排序损失来训练奖励模型。然而，该训练过程存在一个固有问题：由于奖励模型训练时缺乏约束，强化学习过程中奖励分数会出现不受控的缩放问题。本文提出了一种基于先验约束的奖励模型（即PCRM）训练方法来缓解该问题。PCRM在奖励模型训练过程中引入先验约束，具体为每个比较对输出之间的长度比和余弦相似度，以此调节优化幅度并控制分数间隔。我们通过检验PCRM与人类偏好的排序相关性，以及其通过强化学习对齐LLM的有效性，对其进行了全面评估。实验结果表明，PCRM通过有效约束奖励分数缩放，显著提升了对齐性能。另外，该方法可轻松集成到任意基于排序的对齐方法中，例如直接偏好优化，并能带来稳定的性能提升。
英文摘要：
Reinforcement learning with human feedback for aligning large language models (LLMs) trains a reward model typically using ranking loss with comparison pairs.However, the training procedure suffers from an inherent problem: the uncontrolled scaling of reward scores during reinforcement learning due to the lack of constraints while training the reward model.This paper proposes a Prior Constraints-based Reward Model (namely PCRM) training method to mitigate this problem. PCRM incorporates prior constraints, specifically, length ratio and cosine similarity between outputs of each comparison pair, during reward model training to regulate optimization magnitude and control score margins. We comprehensively evaluate PCRM by examining its rank correlation with human preferences and its effectiveness in aligning LLMs via RL. Experimental results demonstrate that PCRM significantly improves alignment performance by effectively constraining reward score scaling. As another bonus, our method is easily integrated into arbitrary rank-based alignment methods, such as direct preference optimization, and can yield consistent improvement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
