基于人类反馈的强化学习（Reinforcement Learning from Human Feedback, RLHF）在仅依赖少量人类反馈的情况下已取得令人瞩目的实证成功，但目前对这一现象的理论解释仍十分有限。此外，尽管基于策略的算法近期实证表现亮眼，多数最新研究仍聚焦于基于价值的算法。本研究提出一种基于策略优化的RLHF算法（PO-RLHF），该算法以广泛使用的策略覆盖-策略梯度（Policy Cover-Policy Gradient, PC-PG）算法为基础，而PC-PG算法需假设奖励函数已知。PO-RLHF无需假设奖励函数已知，而是通过基于轨迹的比较反馈来推断奖励函数。我们给出了低查询复杂度下PO-RLHF的性能界，为“少量人类反馈即可让RLHF取得良好性能”的现象提供了解释思路。该研究的核心创新点是轨迹级椭圆势分析，该分析可在仅提供比较反馈（而非数值奖励观测）的情况下，对奖励估计误差进行界定。我们针对线性函数逼近和神经函数逼近两种场景，分别提出并分析了PG-RLHF和NN-PG-RLHF算法。
英文摘要：
Reinforcement Learning from Human Feedback (RLHF) has achieved impressive empirical successes while relying on a small amount of human feedback. However, there is limited theoretical justification for this phenomenon. Additionally, most recent studies focus on value-based algorithms despite the recent empirical successes of policy-based algorithms. In this work, we consider an RLHF algorithm based on policy optimization (PO-RLHF). The algorithm is based on the popular Policy Cover-Policy Gradient (PC-PG) algorithm, which assumes knowledge of the reward function. In PO-RLHF, knowledge of the reward function is not assumed, and the algorithm uses trajectory-based comparison feedback to infer the reward function. We provide performance bounds for PO-RLHF with low query complexity, which provides insight into why a small amount of human feedback may be sufficient to achieve good performance with RLHF. A key novelty is a trajectory-level elliptical potential analysis, which bounds the reward estimation error when comparison feedback (rather than numerical reward observation) is given. We provide and analyze algorithms PG-RLHF and NN-PG-RLHF for two settings: linear and neural function approximation, respectively.
Avoiding $\mathbf{exp(R_{max})}$ scaling in RLHF through Preference-based Exploration
Department of Electrical & Computer Engineering（电气与计算机工程系）；Boston University（波士顿大学）；Department of Computer Science（计算机科学系）；Cornell University（康奈尔大学）；Faculty of Computing & Data Sciences（计算与数据科学学院）
Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
Department of Statistics and Data Science, The Chinese University of Hong Kong（统计与数据科学系，香港中文大学）；Department of Statistics, University of Wisconsin-Madison（统计系，威斯康星大学麦迪逊分校）
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Salesforce AI Research（Salesforce AI研究院）；The Hong Kong University of Science and Technology（香港科技大学）
Data-dependent Exploration for Online Reinforcement Learning from Human Feedback
Center for Advanced Intelligence Project, RIKEN（日本理化学研究所先进智能研究中心）；Graduate School of Frontier Sciences, The University of Tokyo（东京大学前沿科学研究生院）；Northeastern University（东北大学）；Zhejiang Gongshang University（浙江工商大学）
$f$-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analyses
Department of Electrical and Computer Engineering, University of Virginia, Virginia, United States（弗吉尼亚大学电气与计算机工程系）；Princeton Language and Intelligence, Princeton University, New Jersey, United States（普林斯顿大学语言与智能）
Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
Chinese University of Hong Kong（香港中文大学）；Microsoft Research Asia（微软亚洲研究院）；Peking University（北京大学）；Hong Kong University of Science and Technology（香港科技大学）
From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Baidu Inc.（百度公司）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
Tsinghua University（清华大学）；Tencent AI Lab（腾讯人工智能实验室）
Model-based deep reinforcement learning has achieved success in various domains that require high sample efficiencies, such as Go and robotics. However, there are some remaining issues, such as planning efficient explorations to learn more accurate dynamic models, evaluating the uncertainty of the learned models, and more rational utilization of models. To mitigate these issues, we present MEEE, a model-ensemble method that consists of optimistic exploration and weighted exploitation. During exploration, unlike prior methods directly selecting the optimal action that maximizes the expected acc…
Facebook AI Research（Facebook人工智能研究院）；ETH Zürich（苏黎世联邦理工学院）
In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. However, prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. In previous work on continuous control, the sensitivity of methods to this trade-off has not been addressed explicitly, as locomotion provides a suitable prior for navigation tasks, which have been of foremost interest. In this work, we analyze this trade-off for low-level policy pre-training with a new benc…
A promising way to improve the sample efficiency of reinforcement learning is model-based methods, in which many explorations and evaluations can happen in the learned models to save real-world samples. However, when the learned model has a non-negligible model error, sequential steps in the model are hard to be accurately evaluated, limiting the model's utilization. This paper proposes to alleviate this issue by introducing multi-step plans to replace multi-step actions for model-based RL. We employ the multi-step plan value estimation, which evaluates the expected discounted return after exe…
Towards Better RL Training Data Utilization via Second-Order Rollout
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机学院，北京大学）；ByteDance BandAI（字节跳动BandAI）
