利用合成批评改进奖励模型
Improving Reward Models with Synthetic Critiques
AI总结：
本文提出利用大语言模型生成的合成自然语言批评为奖励模型提供额外反馈，以提升其性能、数据效率、可解释性和鲁棒性，并减少对昂贵人工标注的依赖。
摘要：
奖励模型（RMs）在通过人类反馈强化学习对齐语言模型的过程中发挥着关键作用。奖励模型被训练用于预测反映人类偏好的分数，这需要大量的人工标注时间和成本。此外，奖励模型往往会迅速过拟合训练集中的表面特征，从而阻碍其在未见分布上的泛化性能。我们提出了一种新方法，利用大型语言模型生成的合成自然语言批评来提供额外反馈，评估诸如指令遵循、正确性和风格等方面。这为奖励模型进行评估和打分提供了更丰富的信号和更稳健的特征。我们证明，高质量的批评能够改善基于不同预训练模型初始化的奖励模型的性能和数据效率，减少对昂贵人工标注的依赖。此外，引入批评还提高了奖励模型训练的可解释性和鲁棒性。
英文摘要：
Reward models (RMs) play a critical role in aligning language models through the process of reinforcement learning from human feedback. RMs are trained to predict a score reflecting human preference, which requires significant time and cost for human annotation. Additionally, RMs tend to quickly overfit on superficial features in the training set, hindering their generalization performance on unseen distributions. We propose a novel approach using synthetic natural language critiques generated by large language models to provide additional feedback, evaluating aspects such as instruction following, correctness, and style. This offers richer signals and more robust features for RMs to assess and score on. We demonstrate that high-quality critiques improve the performance and data efficiency of RMs initialized from different pretrained models, reducing the reliance on costly human annotations. Furthermore, incorporating critiques improves both the interpretability and robustness of RM training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
