arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.18409cs.LG
收藏

基于状态-动作相似性的离线策略评估表示方法

State-Action Similarity-Based Representations for Off-Policy Evaluation

University of Wisconsin – Madison(威斯康星大学麦迪逊分校)

Brahma S. Pavse, Josiah P. Hanna

AI总结:

本文提出一种面向离线策略评估的状态-动作相似性表示学习方法,通过学习编码器变换固定数据集后再输入FQE,从而提升数据效率并降低OPE误差。

摘要:

在强化学习中,离线策略评估(off-policy evaluation, OPE)是指在给定一个由运行一个或多个不同策略收集的固定数据集的情况下,估计某个评估策略的期望回报的问题。在经验上较为成功的OPE算法之一是拟合Q评估(fitted q-evaluation, FQE)算法,该算法使用时序差分更新来学习动作价值函数,然后用该函数估计评估策略的期望回报。通常,原始固定数据集被直接输入FQE,以学习评估策略的动作价值函数。而在本文中,我们试图通过首先使用学习到的编码器对固定数据集进行变换,然后将变换后的数据集输入FQE,来提高FQE的数据效率。为了学习这样的编码器,我们引入了一种面向OPE定制的状态-动作行为相似性度量,并使用该度量和固定数据集来学习一个能够建模该度量的编码器。理论上,我们证明该度量使我们能够对所得OPE估计中的误差进行上界约束。实验上,我们表明其他状态-动作相似性度量所得到的表示无法表示评估策略的动作价值函数,并且我们的状态-动作表示方法在具有挑战性的OPE任务上相对于其他基于OPE的表示学习方法提高了FQE的数据效率并降低了OPE误差。我们还通过实验表明,学习到的表示在不同分布偏移下显著缓解了FQE的发散问题。我们的代码可在以下地址获取:https://github.com/Badger-RL/ROPE。

英文摘要:

In reinforcement learning, off-policy evaluation (OPE) is the problem of estimating the expected return of an evaluation policy given a fixed dataset that was collected by running one or more different policies. One of the more empirically successful algorithms for OPE has been the fitted q-evaluation (FQE) algorithm that uses temporal difference updates to learn an action-value function, which is then used to estimate the expected return of the evaluation policy. Typically, the original fixed dataset is fed directly into FQE to learn the action-value function of the evaluation policy. Instead, in this paper, we seek to enhance the data-efficiency of FQE by first transforming the fixed dataset using a learned encoder, and then feeding the transformed dataset into FQE. To learn such an encoder, we introduce an OPE-tailored state-action behavioral similarity metric, and use this metric and the fixed dataset to learn an encoder that models this metric. Theoretically, we show that this metric allows us to bound the error in the resulting OPE estimate. Empirically, we show that other state-action similarity metrics lead to representations that cannot represent the action-value function of the evaluation policy, and that our state-action representation method boosts the data-efficiency of FQE and lowers OPE error relative to other OPE-based representation learning methods on challenging OPE tasks. We also empirically show that the learned representations significantly mitigate divergence of FQE under varying distribution shifts. Our code is available here: https://github.com/Badger-RL/ROPE.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑