迈向可靠对齐：具备不确定性感知的RLHF
Towards Reliable Alignment: Uncertainty-aware RLHF
AI总结：
针对RLHF中奖励模型波动易导致对齐策略过拟合、风险高的问题，提出具备不确定性感知的保守策略优化算法，通过理论和集成奖励模型实验验证其风险更低、对齐更可靠。
摘要：
大语言模型与人类偏好对齐的最新进展得益于规模更大的奖励模型和质量更高的偏好数据。但这类方法大多依赖奖励模型的准确性。人类反馈强化学习（Reinforcement Learning with Human Feedback, RLHF）中使用的奖励模型通常由小型数据集通过随机优化算法训练得到，因此容易出现高波动性。我们在多个开源数据集上通过实验展示了不同奖励模型之间存在的不一致性。\n我们从理论上证明，奖励模型的波动会对对齐问题产生不利影响，因为由此得到的策略会更严重地过拟合于奖励模型，因此当奖励模型本身存在不确定性时，策略的风险更高。我们利用测度集中原理，提出了一种具备不确定性感知的保守策略优化算法。我们证明这类策略更具风险规避性，即对不确定的奖励更为谨慎。我们从理论上证实，所提方法的风险低于传统方法。\n我们通过设计奖励模型集成开展实验，验证了理论结果。我们使用该奖励模型集成，结合所提方法对语言模型进行对齐，实验结果与理论预测相符。
英文摘要：
Recent advances in aligning Large Language Models with human preferences have benefited from larger reward models and better preference data. However, most of these methodologies rely on the accuracy of the reward model. The reward models used in Reinforcement Learning with Human Feedback (RLHF) are typically learned from small datasets using stochastic optimization algorithms, making them prone to high variability. We illustrate the inconsistencies between reward models empirically on numerous open-source datasets. We theoretically show that the fluctuation of the reward models can be detrimental to the alignment problem because the derived policies are more overfitted to the reward model and, hence, are riskier if the reward model itself is uncertain. We use concentration of measure to motivate an uncertainty-aware, conservative algorithm for policy optimization. We show that such policies are more risk-averse in the sense that they are more cautious of uncertain rewards. We theoretically prove that our proposed methodology has less risk than the vanilla method. We corroborate our theoretical results with experiments based on designing an ensemble of reward models. We use this ensemble of reward models to align a language model using our methodology and observe that our empirical findings match our theoretical predictions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
