通过基于偏好的探索避免 RLHF 中的 $\mathbf{exp(R_{max})}$ 缩放
Avoiding $\mathbf{exp(R_{max})}$ scaling in RLHF through Preference-based Exploration
AI总结：
针对在线RLHF中样本复杂度随奖励规模指数增长的问题，提出SE-POPO算法，首次实现多项式复杂度，并在理论和实验上验证其样本效率优于现有基线。
摘要：
基于人类反馈的强化学习（RLHF）已成为大型语言模型（LLM）对齐的关键技术。本文研究在线 RLHF 的设置，并专注于提高样本效率。在线 RLHF 中的所有现有算法，无论是进行被动探索还是主动探索，其样本复杂度都随奖励函数的规模呈指数级增长。这一根本性限制阻碍了它们在偏好严重偏斜的场景（例如具有唯一正确答案的问题）中的有效性。为了解决这个问题，我们引入了自探索偏好激励在线偏好优化（SE-POPO），这是一种在线 RLHF 算法，首次实现了样本复杂度随奖励规模呈多项式增长，回答了 Xie 等人（2024）提出的一个开放问题。理论上，我们证明了 SE-POPO 的样本复杂度优于现有探索算法。实验上，我们的系统评估证实，在 RLHF 的两个主要应用场景以及公共基准测试中，SE-POPO 比探索性和非探索性基线更具样本效率，标志着 RLHF 算法设计的重要进步。代码可在 https://github.com/MYC000801/SE-POPO 获取。
英文摘要：
Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal technique for large language model (LLM) alignment. This paper studies the setting of online RLHF and focus on improving sample efficiency. All existing algorithms in online RLHF, whether doing passive exploration or active exploration, suffer from a sample complexity that scales exponentially with the scale of the reward function. This fundamental limitation hinders their effectiveness in scenarios with heavily skewed preferences, e.g. questions with a unique correct solution. To address this, we introduce Self-Exploring Preference-Incentive Online Preference Optimization (SE-POPO), an online RLHF algorithm that for the first time achieves a sample complexity that scales polynomially with the reward scale, answering an open problem raised by Xie et al. (2024).. Theoretically, we demonstrate that the sample complexity of SE-POPO dominates that of existing exploration algorithms. Empirically, our systematic evaluation confirms that SE-POPO is more sample-efficient than both exploratory and non-exploratory baselines, in two primary application scenarios of RLHF as well as on public benchmarks, marking a significant step forward in RLHF algorithm design. The code is available at https://github.com/MYC000801/SE-POPO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
