arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.09401cs.LGcs.AIcs.CLmath.OCstat.ML
收藏

基于主动查询的人类反馈强化学习

Reinforcement Learning from Human Feedback with Active Queries

University of California, Los Angeles(加利福尼亚大学洛杉矶分校)

Kaixuan Ji, Jiafan He, Quanquan Gu

AI总结:

针对RLHF需大量昂贵人类偏好数据的问题,本文将对齐问题形式化为上下文决斗式多臂老虎机问题,提出ADPO算法,仅用约一半查询量即可匹配最先进DPO方法的性能。

摘要:

大语言模型(LLM)与人类偏好对齐在构建现代生成模型中起着关键作用,可通过人类反馈强化学习(RLHF)实现。尽管当前RLHF方法性能优越,但通常需要大量人类标注的偏好数据,而这类数据的收集成本很高。本文受主动学习成功的启发,提出查询高效的RLHF方法以解决该问题。我们首先将对齐问题形式化为上下文决斗式多臂老虎机问题,设计了一种基于主动查询的近端策略优化(APPO)算法,其依赖实例的遗憾界为$\tilde{O}(d^2/Δ)$,查询复杂度为$\tilde{O}(d^2/Δ^2)$,其中$d$是特征空间维度,$Δ$是所有上下文的次优间隙。随后,我们提出ADPO算法——基于直接偏好优化(DPO)的实用版本,并将其应用于LLM微调。实验表明,ADPO仅需约一半的人类偏好查询量,即可匹配最先进DPO方法的性能。

英文摘要:

Aligning large language models (LLM) with human preference plays a key role in building modern generative models and can be achieved by reinforcement learning from human feedback (RLHF). Despite their superior performance, current RLHF approaches often require a large amount of human-labelled preference data, which is expensive to collect. In this paper, inspired by the success of active learning, we address this problem by proposing query-efficient RLHF methods. We first formalize the alignment problem as a contextual dueling bandit problem and design an active-query-based proximal policy optimization (APPO) algorithm with an $\tilde{O}(d^2/Δ)$ instance-dependent regret bound and an $\tilde{O}(d^2/Δ^2)$ query complexity, where $d$ is the dimension of feature space and $Δ$ is the sub-optimality gap over all the contexts. We then propose ADPO, a practical version of our algorithm based on direct preference optimization (DPO) and apply it to fine-tuning LLMs. Our experiments show that ADPO, while only making about half of queries for human preference, matches the performance of the state-of-the-art DPO method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑