基于人类反馈的强化学习（Reinforcement Learning with Human Feedback, RLHF）是语言和图像生成式AI模型微调方法的核心。这类反馈通常以人类评分者给出的排序或偏好反馈的形式获取，而非直接采集分数，因为分数往往存在噪声。另一方面，强化学习理论与算法大多假设可获得奖励反馈。尤其是可通过主动学习助力自适应数据收集的在线学习方法，无法整合离线偏好数据。本文采用有限臂线性老虎机模型作为在线学习的原型模型，假设存在由“能力”未知的专家生成的离线偏好数据集。我们提出warmPref-PS，这是一种在线学习后验采样算法，可利用带噪声偏好反馈的离线数据集进行热启动。我们证明，通过对生成该数据集的专家“能力”建模，能够最高效地利用此类数据集。我们通过对其贝叶斯遗憾的全新理论分析，以及针对无限臂优化的近似损失函数的大量实证评估，支撑了我们的结论，该方法的表现显著优于基线方法。
英文摘要：
Reinforcement Learning with Human Feedback (RLHF) is at the core of fine-tuning methods for generative AI models for language and images. Such feedback is often sought as rank or preference feedback from human raters, as opposed to eliciting scores since the latter tends to be noisy. On the other hand, RL theory and algorithms predominantly assume that a reward feedback is available. In particular, approaches for online learning that can be helpful in adaptive data collection via active learning cannot incorporate offline preference data. In this paper, we adopt a finite-armed linear bandit model as a prototypical model of online learning. We consider an offline preference dataset to be available generated by an expert of unknown 'competence'. We propose warmPref-PS, a posterior sampling algorithm for online learning that can be warm-started with an offline dataset with noisy preference feedback. We show that by modeling the 'competence' of the expert that generated it, we are able to use such a dataset most effectively. We support our claims with novel theoretical analysis of its Bayesian regret, as well as, extensive empirical evaluation of an approximate loss function that optimizes for infinitely many arms, and performs substantially better than baselines.
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Salesforce AI Research（Salesforce AI研究院）；The Hong Kong University of Science and Technology（香港科技大学）
Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）；University of Chicago（芝加哥大学）
We present a family $\{\hatπ\}_{p\ge 1}$ of pessimistic learning rules for offline learning of linear contextual bandits, relying on confidence sets with respect to different $\ell_p$ norms, where $\hatπ_2$ corresponds to Bellman-consistent pessimism (BCP), while $\hatπ_\infty$ is a novel generalization of lower confidence bound (LCB) to the linear setting. We show that the novel $\hatπ_\infty$ learning rule is, in a sense, adaptively optimal, as it achieves the minimax performance (up to log factors) against all $\ell_q$-constrained problems, and as such it strictly dominates all other predic…
Efficient Preference Poisoning Attack on Offline RLHF
Department of Electrical and Computer Engineering, University of California, Davis, Davis, CA, USA（加州大学戴维斯分校电气与计算机工程系）；Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA, USA（爱荷华大学电气与计算机工程系）
Computationally efficient contextual bandits are often based on estimating a predictive model of rewards given contexts and arms using past data. However, when the reward model is not well-specified, the bandit algorithm may incur unexpected regret, so recent work has focused on algorithms that are robust to misspecification. We propose a simple family of contextual bandit algorithms that adapt to misspecification error by reverting to a good safe policy when there is evidence that misspecification is causing a regret increase. Our algorithm requires only an offline regression oracle to ensure…
We address policy learning with logged data in contextual bandits. Current offline-policy learning algorithms are mostly based on inverse propensity score (IPS) weighting requiring the logging policy to have \emph{full support} i.e. a non-zero probability for any context/action of the evaluation policy. However, many real-world systems do not guarantee such logging policies, especially when the action space is large and many actions have poor or missing rewards. With such \emph{support deficiency}, the offline learning fails to find optimal policies. We propose a novel approach that uses a hyb…
This paper extends the Distributionally Robust Optimization (DRO) approach for offline contextual bandits. Specifically, we leverage this framework to introduce a convex reformulation of the Counterfactual Risk Minimization principle. Besides relying on convex programs, our approach is compatible with stochastic optimization, and can therefore be readily adapted tothe large data regime. Our approach relies on the construction of asymptotic confidence intervals for offline contextual bandits through the DRO framework. By leveraging known asymptotic results of robust estimators, we also show how…
The Chinese University of Hong Kong（香港中文大学）；Tencent（腾讯）；Chongqing University（重庆大学）
A fundamental question for companies with large amount of logged data is: How to use such logged data together with incoming streaming data to make good decisions? Many companies currently make decisions via online A/B tests, but wrong decisions during testing hurt users' experiences and cause irreversible damage. A typical alternative is offline causal inference, which analyzes logged data alone to make decisions. However, these decisions are not adaptive to the new incoming data, and so a wrong decision will continuously hurt users' experiences. To overcome the aforementioned limitations, we…
Offline Clustering of Linear Bandits: The Power of Clusters under Limited Data
Nanjing University（南京大学）；The Chinese University of Hong Kong（香港中文大学）；University of Massachusetts, Amherst（马萨诸塞大学阿默斯特分校）；Carnegie Mellon University（卡内基梅隆大学）
Applied AI Institute（应用人工智能研究所）；Deakin University（迪肯大学）；University of Oxford（牛津大学）
Offline policy learning (OPL) leverages existing data collected a priori for policy optimization without any active exploration. Despite the prevalence and recent interest in this problem, its theoretical and algorithmic foundations in function approximation settings remain under-developed. In this paper, we consider this problem on the axes of distributional shift, optimization, and generalization in offline contextual bandits with neural networks. In particular, we propose a provably efficient offline contextual bandit with neural network function approximation that does not require any func…
Courant Institute of Mathematical Sciences, New York University（纽约大学库朗数学科学研究所）
Recent results in supervised learning suggest that while overparameterized models have the capacity to overfit, they in fact generalize quite well. We ask whether the same phenomenon occurs for offline contextual bandits. Our results are mixed. Value-based algorithms benefit from the same generalization behavior as overparameterized supervised learning, but policy-based algorithms do not. We show that this discrepancy is due to the \emph{action-stability} of their objectives. An objective is action-stable if there exists a prediction (action-value vector or action distribution) which is optima…
Users of recommender systems often behave in a non-stationary fashion, due to their evolving preferences and tastes over time. In this work, we propose a practical approach for fast personalization to non-stationary users. The key idea is to frame this problem as a latent bandit, where the prototypical models of user behavior are learned offline and the latent state of the user is inferred online from its interactions with the models. We call this problem a non-stationary latent bandit. We propose Thompson sampling algorithms for regret minimization in non-stationary latent bandits, analyze th…
RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Qwen LLM Application Team, Alibaba（阿里巴巴Qwen大模型应用团队）；Stanford University（斯坦福大学）
Technion, Israel Institute of Technology（以色列理工学院）；Nvidia Research（英伟达研究院）
We study linear contextual bandits with access to a large, confounded, offline dataset that was sampled from some fixed policy. We show that this problem is closely related to a variant of the bandit problem with side information. We construct a linear bandit algorithm that takes advantage of the projected information, and prove regret bounds. Our results demonstrate the ability to take advantage of confounded offline data. Particularly, we prove regret bounds that improve current bounds by a factor related to the visible dimensionality of the contexts in the data. Our results indicate that co…
Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles
National University of Defense Technology（国防科技大学）；State Key Laboratory of Complex & Critical Software Environment（复杂与关键软件环境国家重点实验室）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）
Shanghai Jiao Tong University, Shanghai, China（上海交通大学）；University of Washington, Tacoma, United States（华盛顿大学 Tacoma 分校）；University of Massachusetts Amherst, Massachusetts, United States（马萨诸塞大学阿姆赫斯特分校）
