arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.18873cs.LG
收藏

最佳策略学习从轨迹偏好反馈

Best Policy Learning from Trajectory Preference Feedback

University of Southern California(南加州大学);Google(谷歌);DeepMind(深度Mind);OpenAI

Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, Zheng Wen

AI总结:

本文研究了基于偏好强化学习的最佳策略识别问题,提出PSPL算法,通过维护奖励模型和动态的后验分布,提供首个贝叶斯简单遗憾保证,并在模拟和图像生成基准中优于现有基线。

摘要:

基于人类反馈的强化学习(RLHF)已发展为对生成模型对齐的强大方法,但其依赖于学习的奖励模型使其容易受到误指定和奖励黑客的攻击。基于偏好的强化学习(PbRL)通过直接利用轨迹上的噪声二元比较提供更稳健的替代方案。我们研究了PbRL中的最佳策略识别问题,受生成模型后训练优化的启发,例如在多轮交互中。在此设置中学习结合了离线偏好数据集——可能有偏见或分布外的数据,并且由能力较差的评估者收集,与在线纯探索相结合,使系统性的在线学习至关重要。为此,我们提出了偏好学习的后验采样(PSPL),一种受Top-Two Thompson Sampling启发的新型算法,维护奖励模型和动态的后验分布。我们为PbRL提供了首个贝叶斯简单遗憾保证,并引入了高效的近似方法,在模拟和图像生成基准中优于现有基线。

英文摘要:

Reinforcement Learning from Human Feedback (RLHF) has emerged as a powerful approach for aligning generative models, but its reliance on learned reward models makes it vulnerable to mis-specification and reward hacking. Preference-based Reinforcement Learning (PbRL) offers a more robust alternative by directly leveraging noisy binary comparisons over trajectories. We study the best policy identification problem in PbRL, motivated by post-training optimization of generative models, for example, during multi-turn interactions. Learning in this setting combines an offline preference dataset - potentially biased or out-of-distribution and collected from a rater of subpar `competence' - with online pure exploration, making systematic online learning essential. To this end, we propose Posterior Sampling for Preference Learning ($\mathsf{PSPL}$), a novel algorithm inspired by Top-Two Thompson Sampling that maintains posteriors over the reward model and dynamics. We provide the first Bayesian simple regret guarantees for PbRL and introduce an efficient approximation that outperforms existing baselines on simulation and image generation benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑