基于数据集内轨迹回报正则化的离线偏好强化学习
In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning
AI总结：
提出数据集内轨迹回报正则化（DTR）方法，利用条件序列建模和集成归一化技术缓解离线偏好强化学习中的奖励偏差问题，在多个基准上取得优越性能。
摘要：
离线基于偏好的强化学习（PbRL）通常分为两个阶段：首先，利用人类偏好学习奖励模型，并为无奖励的离线数据集标注奖励；其次，通过离线RL优化学习到的奖励来学习策略。然而，从轨迹级偏好反馈中准确建模逐步奖励存在固有挑战。引入的奖励偏差，特别是预测奖励的高估，会导致乐观的轨迹拼接，从而破坏离线RL阶段至关重要的悲观机制。为应对此挑战，我们提出用于离线PbRL的数据集内轨迹回报正则化（DTR），利用条件序列建模来缓解在奖励偏差下学习不准确轨迹拼接的风险。具体而言，DTR采用Decision Transformer和TD-Learning，在保持对具有高数据集内轨迹回报的行为策略的保真度与基于高奖励标签选择最优动作之间取得平衡。此外，我们引入了一种集成归一化技术，有效整合多个奖励模型，平衡奖励区分度与准确性之间的权衡。在多个基准上的实证评估证明了DTR相对于其他最先进基线的优越性。
英文摘要：
Offline preference-based reinforcement learning (PbRL) typically operates in two phases: first, use human preferences to learn a reward model and annotate rewards for a reward-free offline dataset; second, learn a policy by optimizing the learned reward via offline RL. However, accurately modeling step-wise rewards from trajectory-level preference feedback presents inherent challenges. The reward bias introduced, particularly the overestimation of predicted rewards, leads to optimistic trajectory stitching, which undermines the pessimism mechanism critical to the offline RL phase. To address this challenge, we propose In-Dataset Trajectory Return Regularization (DTR) for offline PbRL, which leverages conditional sequence modeling to mitigate the risk of learning inaccurate trajectory stitching under reward bias. Specifically, DTR employs Decision Transformer and TD-Learning to strike a balance between maintaining fidelity to the behavior policy with high in-dataset trajectory returns and selecting optimal actions based on high reward labels. Additionally, we introduce an ensemble normalization technique that effectively integrates multiple reward models, balancing the tradeoff between reward differentiation and accuracy. Empirical evaluations on various benchmarks demonstrate the superiority of DTR over other state-of-the-art baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
