arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.15360cs.LGcs.AIcs.CL
收藏

大型语言模型中的奖励鲁棒RLHF

Reward-Robust RLHF in LLMs

Tsinghua University(清华大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Baichuan AI(百川智能)

Yuzi Yan, Xingzhou Lou, Jialian Li, Yiping Zhang, Jian Xie, Chao Yu, Yu Wang, Dong Yan, Yuan Shen

AI总结:

针对奖励模型不完美导致的奖励黑客和意图不一致问题,提出结合贝叶斯奖励模型集成的奖励鲁棒RLHF框架,通过平衡性能与鲁棒性优化目标,显著提升了LLM对齐的准确性与长期稳定性。

摘要:

随着大型语言模型(LLMs)继续向更高级的智能形式发展,基于人类反馈的强化学习(RLHF)越来越被视为实现通用人工智能(AGI)的关键途径。然而,由于奖励模型(RMs)固有的不稳定性与不完美性,依赖基于奖励模型(RM-based)的对齐方法引入了重大挑战,这可能导致奖励黑客和与人类意图不一致等关键问题。在本文中,我们引入了一个奖励鲁棒的RLHF框架,旨在解决这些根本挑战,为LLMs中更可靠和更具弹性的学习铺平道路。我们的方法引入了一种新颖的优化目标,通过结合贝叶斯奖励模型集成(BRME)来建模奖励函数的不确定性集合,从而仔细平衡性能与鲁棒性。这使得框架能够同时整合标称性能和最小奖励信号,确保即使在不完美的RMs下也能实现更稳定的学习。实证结果表明,我们的框架在多样化的基准测试中始终优于基线,展现出更高的准确性和长期稳定性。我们还提供了理论分析,证明奖励鲁棒的RLHF接近于恒定奖励设置的稳定性,这在随机案例分析中也被证明是可接受的。这些贡献共同突显了该框架在提升LLM对齐的性能和稳定性方面的潜力。

英文摘要:

As Large Language Models (LLMs) continue to progress toward more advanced forms of intelligence, Reinforcement Learning from Human Feedback (RLHF) is increasingly seen as a key pathway toward achieving Artificial General Intelligence (AGI). However, the reliance on reward-model-based (RM-based) alignment methods introduces significant challenges due to the inherent instability and imperfections of Reward Models (RMs), which can lead to critical issues such as reward hacking and misalignment with human intentions. In this paper, we introduce a reward-robust RLHF framework aimed at addressing these fundamental challenges, paving the way for more reliable and resilient learning in LLMs. Our approach introduces a novel optimization objective that carefully balances performance and robustness by incorporating Bayesian Reward Model Ensembles (BRME) to model the uncertainty set of reward functions. This allows the framework to integrate both nominal performance and minimum reward signals, ensuring more stable learning even with imperfect RMs. Empirical results demonstrate that our framework consistently outperforms baselines across diverse benchmarks, showing improved accuracy and long-term stability. We also provide a theoretical analysis, demonstrating that reward-robust RLHF approaches the stability of constant reward settings, which proves to be acceptable even in a stochastic-case analysis. Together, these contributions highlight the framework potential to enhance both the performance and stability of LLM alignment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑