价值激励偏好优化：在线与离线RLHF的统一方法
Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
AI总结：
针对RLHF中偏好数据收集方式不同（在线/离线）的问题，提出价值激励偏好优化（VPO）统一方法，通过值函数正则化奖励函数并调节乐观/悲观策略，兼具理论保证与文本摘要、对话任务的实验有效性。
摘要：
从人类反馈中强化学习（RLHF）在使大型语言模型（LLMs）与人类偏好对齐方面展现出巨大潜力。根据偏好数据的可用性，在线和离线RLHF均为活跃研究领域。无论偏好数据如何收集，关键瓶颈在于理解如何将不确定性估计纳入从RLHF偏好数据中学习到的奖励函数。尽管不确定性下的乐观或悲观原则在标准强化学习（RL）中已确立，但适用于大型语言模型的可实际实现且有理论基础的形式尚未出现，因为构建置信区间的标准技术在任意策略参数化下难以处理。本文提出在线与离线RLHF的统一方法——价值激励偏好优化（VPO），该方法用相应的值函数正则化奖励函数的最大似然估计，通过一个“符号”调节以选择乐观或悲观。VPO还通过隐式奖励建模直接优化策略，因此具有与直接偏好优化相似的更简单RLHF pipeline。本文为在线和离线设置提供了VPO的理论保证，匹配其标准RL对应方法的速率。此外，文本摘要和对话任务的实验验证了VPO的实用性和有效性。
英文摘要：
Reinforcement learning from human feedback (RLHF) has demonstrated great promise in aligning large language models (LLMs) with human preference. Depending on the availability of preference data, both online and offline RLHF are active areas of investigation. A key bottleneck is understanding how to incorporate uncertainty estimation in the reward function learned from the preference data for RLHF, regardless of how the preference data is collected. While the principles of optimism or pessimism under uncertainty are well-established in standard reinforcement learning (RL), a practically-implementable and theoretically-grounded form amenable to large language models is not yet available, as standard techniques for constructing confidence intervals become intractable under arbitrary policy parameterizations. In this paper, we introduce a unified approach to online and offline RLHF -- value-incentivized preference optimization (VPO) -- which regularizes the maximum-likelihood estimate of the reward function with the corresponding value function, modulated by a $\textit{sign}$ to indicate whether the optimism or pessimism is chosen. VPO also directly optimizes the policy with implicit reward modeling, and therefore shares a simpler RLHF pipeline similar to direct preference optimization. Theoretical guarantees of VPO are provided for both online and offline settings, matching the rates of their standard RL counterparts. Moreover, experiments on text summarization and dialog verify the practicality and effectiveness of VPO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
