离线偏好强化学习的对抗策略优化
Adversarial Policy Optimization for Offline Preference-based Reinforcement Learning
AI总结：
本文提出APPO算法，将离线偏好强化学习建模为策略与模型的双人博弈，在不依赖显式置信集的情况下实现可处理的保守性，并给出样本复杂度界，是首个兼具统计效率与实际适用性的离线PbRL方法。
摘要：
本文研究离线偏好强化学习（PbRL），其学习基于预先收集的轨迹对偏好反馈。尽管离线 PbRL 已在实证上取得显著成功，现有理论方法在不确定性下保证保守性方面面临挑战，需要计算上不可行的置信集构造。我们通过提出对抗偏好策略优化（APPO）来解决这一局限，这是一种计算高效的离线 PbRL 算法，在不依赖显式置信集的情况下保证样本复杂度界。通过将 PbRL 构建为策略与模型之间的双人博弈，我们的方法以可处理的方式强制实现保守性。在函数逼近和有界轨迹集中度的标准假设下，我们推导出样本复杂度界。据我们所知，APPO 是首个同时具备统计效率和实际适用性的离线 PbRL 算法。在连续控制任务上的实验结果表明，APPO 能有效从复杂数据集中学习，展现出与现有最先进方法相当的性能。
英文摘要：
In this paper, we study offline preference-based reinforcement learning (PbRL), where learning is based on pre-collected preference feedback over pairs of trajectories. While offline PbRL has demonstrated remarkable empirical success, existing theoretical approaches face challenges in ensuring conservatism under uncertainty, requiring computationally intractable confidence set constructions. We address this limitation by proposing Adversarial Preference-based Policy Optimization (APPO), a computationally efficient algorithm for offline PbRL that guarantees sample complexity bounds without relying on explicit confidence sets. By framing PbRL as a two-player game between a policy and a model, our approach enforces conservatism in a tractable manner. Using standard assumptions on function approximation and bounded trajectory concentrability, we derive a sample complexity bound. To our knowledge, APPO is the first offline PbRL algorithm to offer both statistical efficiency and practical applicability. Experimental results on continuous control tasks demonstrate that APPO effectively learns from complex datasets, showing comparable performance with existing state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
