离线约束强化学习中的多偏好Oracle
Offline Constrained RLHF with Multiple Preference Oracles
AI总结：
本文基于多偏好Oracle研究离线约束强化学习，通过最大似然估计Oracle奖励并分析统计不确定性传播，将约束目标转化为KL-正则化拉格朗日函数，提出仅需对偶的算法以保证约束满足并提供有限样本性能保证。
摘要：
我们研究了利用多个偏好Oracle进行离线约束强化学习。受在性能与安全或公平之间权衡的应用启发，我们旨在最大化目标群体效用，同时满足受保护群体福利的最小约束。从参考策略下收集的成对比较中，我们通过最大似然估计Oracle特定奖励，并分析统计不确定性如何通过对偶程序传播。我们将约束目标转化为KL-正则化拉格朗日函数，其中对偶优化器是一个吉布斯策略，从而将学习转化为凸对偶问题。我们提出一个仅需对偶的算法，确保高概率的约束满足，并为离线约束偏好学习提供首个有限样本性能保证。最后，我们将理论分析扩展到多约束和一般f-分歧正则化情况。
英文摘要：
We study offline constrained reinforcement learning from human feedback with multiple preference oracles. Motivated by applications that trade off performance with safety or fairness, we aim to maximize target population utility subject to a minimum protected group welfare constraint. From pairwise comparisons collected under a reference policy, we estimate oracle-specific rewards via maximum likelihood and analyze how statistical uncertainty propagates through the dual program. We cast the constrained objective as a KL-regularized Lagrangian whose primal optimizer is a Gibbs policy, reducing learning to a convex dual problem. We propose a dual-only algorithm that ensures high-probability constraint satisfaction and provide the first finite-sample performance guarantees for offline constrained preference learning. Finally, we extend our theoretical analysis to accommodate multiple constraints and general f-divergence regularization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
