TROFI：轨迹排序的离线逆向强化学习
TROFI: Trajectory-Ranked Offline Inverse Reinforcement Learning
AI总结：
本文提出TROFI方法，通过从人类偏好中学习奖励函数来标注离线数据集，无需预定义奖励或最优轨迹即可有效训练策略，在D4RL基准和3D游戏环境中表现优于基线且媲美真实奖励。
摘要：
在离线强化学习中，智能体仅使用源自源策略的固定存储转移数据集进行训练。然而，这要求数据集由奖励函数标注。在视频游戏开发等应用场景中，奖励函数的可用性并不总是能得到保证。本文提出了Trajectory-Ranked OFfline Inverse reinforcement learning（TROFI），这是一种无需预定义奖励函数即可有效离线学习策略的新方法。TROFI首先从人类偏好中学习奖励函数，然后使用该函数为原始数据集打标签，使其可用于训练策略。与其他方法不同，我们的方法不需要最优轨迹。通过在D4RL基准上的实验，我们证明TROFI始终优于基线方法，且表现与使用真实奖励学习策略相当。此外，我们在3D游戏环境中验证了该方法的有效性。我们对奖励模型的研究突显了奖励函数在此场景中的重要性：我们表明，为确保价值函数与实际未来折扣奖励的对齐，拥有一个精心设计且易于学习的奖励函数是根本所在。
英文摘要：
In offline reinforcement learning, agents are trained using only a fixed set of stored transitions derived from a source policy. However, this requires that the dataset be labeled by a reward function. In applied settings such as video game development, the availability of the reward function is not always guaranteed. This paper proposes Trajectory-Ranked OFfline Inverse reinforcement learning (TROFI), a novel approach to effectively learn a policy offline without a pre-defined reward function. TROFI first learns a reward function from human preferences, which it then uses to label the original dataset making it usable for training the policy. In contrast to other approaches, our method does not require optimal trajectories. Through experiments on the D4RL benchmark we demonstrate that TROFI consistently outperforms baselines and performs comparably to using the ground truth reward to learn policies. Additionally, we validate the efficacy of our method in a 3D game environment. Our studies of the reward model highlight the importance of the reward function in this setting: we show that to ensure the alignment of a value function to the actual future discounted reward, it is fundamental to have a well-engineered and easy-to-learn reward function.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
