arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.02504stat.MLcs.LG
收藏

基于人类反馈的强化学习的双重主动学习

Dual Active Learning for Reinforcement Learning from Human Feedback

Mitchell E. Daniels, Jr. School of Business, Purdue University(普渡大学米切尔·E·丹尼尔斯 Jr. 商学院);London School of Economics and Political Science(伦敦政治经济学院)

Pangpang Liu, Chengchun Shi, Will Wei Sun

AI总结:

本文提出一种双重主动奖励学习算法,同时选择对话和教师以高效获取人类反馈,并结合悲观离线强化学习实现LLM对齐,理论证明其方差最优性和次优性收敛速度,实验验证其优于现有方法。

摘要:

将大型语言模型(LLMs)与人类偏好对齐是生成式人工智能近期进展中的关键环节。基于人类反馈的强化学习(RLHF)被广泛用于实现这一目标。RLHF中的一个关键步骤是从人类反馈中学习奖励函数。然而,人类反馈成本高昂且耗时,因此必须收集高质量的对话数据供人类教师标注。此外,不同的人类教师具有不同的专业水平。因此,查询最合适的教师以获取其意见至关重要。在本文中,我们使用离线强化学习(RL)来形式化对齐问题。受$D$-最优设计思想的启发,我们首先提出了一种双重主动奖励学习算法,用于同时选择对话和教师。接下来,我们基于学习到的奖励估计器,应用悲观强化学习来解决对齐问题。理论上,我们证明了通过所提出的自适应选择策略获得的奖励估计器在渐近意义下达到最小广义方差,并证明了在给定样本预算$T$的情况下,我们的悲观策略的次优性以$O(1/\sqrt{T})$的规模缩放。通过在LLMs上的模拟和实验,我们展示了所提算法的有效性及其相对于最先进方法的优越性。

英文摘要:

Aligning large language models (LLMs) with human preferences is critical to recent advances in generative artificial intelligence. Reinforcement learning from human feedback (RLHF) is widely applied to achieve this objective. A key step in RLHF is to learn the reward function from human feedback. However, human feedback is costly and time-consuming, making it essential to collect high-quality conversation data for human teachers to label. Additionally, different human teachers have different levels of expertise. It is thus critical to query the most appropriate teacher for their opinions. In this paper, we use reinforcement learning (RL) to formulate the alignment problem. Motivated by the idea of $D$-optimal design, we first propose a dual active reward learning algorithm for the simultaneous selection of conversations and teachers. Next, we apply pessimistic RL to solve the alignment problem, based on the learned reward estimator. Theoretically, we show that the reward estimator obtained through our proposed adaptive selection strategy achieves minimal generalized variance asymptotically, and prove that the sub-optimality of our pessimistic policy scales as $O(1/\sqrt{T})$ with a given sample budget $T$. Through simulations and experiments on LLMs, including annotator-level preference data and downstream Best-of-$N$ response-selection evaluations, we demonstrate the effectiveness of our algorithm and its superiority over existing baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑