面向离线偏好强化学习的后见偏好学习
Hindsight Preference Learning for Offline Preference-based Reinforcement Learning
AI总结：
本文提出后见偏好学习（HPL），通过以轨迹段未来结果为条件的奖励建模人类偏好，并利用变分自编码器近似未来结果分布，从而在离线偏好强化学习中更有效地进行信用分配，实验证明其在多个领域能提供稳健且优越的奖励。
摘要：
离线偏好强化学习（RL）专注于利用人类对从离线数据集中选取的轨迹段对的偏好来优化策略，已成为RL应用的一种实用途径。现有工作依赖于从轨迹级偏好标注中提取逐步奖励信号，假设偏好与累积马尔可夫奖励相关。然而，此类方法未能捕捉数据标注的整体视角：人类在评估动作序列的可取性时，往往考虑整体结果而非即时奖励。为应对这一挑战，我们提出以轨迹段未来结果为条件的奖励来建模人类偏好，即后见信息。在下游RL优化中，每一步的奖励通过边缘化可能的未来结果来计算，其分布由使用离线数据集训练的变分自编码器近似。我们提出的方法——后见偏好学习（HPL）——能够充分利用海量无标注数据集中可用的丰富轨迹数据来促进信用分配。全面的实证研究表明，HPL在多个领域提供了稳健且具有优势的奖励。我们的代码已公开发布于 https://github.com/typoverflow/WiseRL。
英文摘要：
Offline preference-based reinforcement learning (RL), which focuses on optimizing policies using human preferences between pairs of trajectory segments selected from an offline dataset, has emerged as a practical avenue for RL applications. Existing works rely on extracting step-wise reward signals from trajectory-wise preference annotations, assuming that preferences correlate with the cumulative Markovian rewards. However, such methods fail to capture the holistic perspective of data annotation: Humans often assess the desirability of a sequence of actions by considering the overall outcome rather than the immediate rewards. To address this challenge, we propose to model human preferences using rewards conditioned on future outcomes of the trajectory segments, i.e. the hindsight information. For downstream RL optimization, the reward of each step is calculated by marginalizing over possible future outcomes, the distribution of which is approximated by a variational auto-encoder trained using the offline dataset. Our proposed method, Hindsight Preference Learning (HPL), can facilitate credit assignment by taking full advantage of vast trajectory data available in massive unlabeled datasets. Comprehensive empirical studies demonstrate the benefits of HPL in delivering robust and advantageous rewards across various domains. Our code is publicly released at https://github.com/typoverflow/WiseRL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
