在什么情况下离线策略选择对强化学习是样本高效的？
When is Offline Policy Selection Sample Efficient for Reinforcement Learning?
AI总结：
本文研究了离线策略选择在强化学习中的样本效率问题，提出了一种基于贝尔曼误差的IBES方法，通过理论分析和实验验证其有效性。
摘要：
离线强化学习算法通常需要仔细调整超参数。在部署之前，我们需要在一组候选策略中进行选择。然而，对于这个离线策略选择（OPS）问题的基本限制仍缺乏理解。在本文中，我们澄清了何时样本高效的OPS是可能的，主要通过将OPS与离线策略评估（OPE）和贝尔曼误差（BE）估计联系起来。我们首先展示了一个难度结果，即在最坏情况下，OPS与OPE一样难，通过证明将OPE还原到OPS。因此，在最坏情况下，没有任何OPS方法可以比OPE更样本高效。然后我们将BE估计与OPS问题联系起来，展示如何将BE用作OPS的工具。虽然基于BE的方法通常比OPE要求更强，但当满足这些条件时，它们可以更样本高效。基于这一见解，我们提出了一种名为可识别BE选择（IBES）的BE方法，其选择自身超参数的方法很简单。最后，我们通过比较OPE和IBES进行实证研究，并通过在离线Atari基准数据集上展示OPS的难度来结束。
英文摘要：
Offline reinforcement learning algorithms often require careful hyperparameter tuning. Before deployment, we need to select amongst a set of candidate policies. However, there is limited understanding about the fundamental limits of this offline policy selection (OPS) problem. In this work we provide clarity on when sample efficient OPS is possible, primarily by connecting OPS to off-policy policy evaluation (OPE) and Bellman error (BE) estimation. We first show a hardness result, that in the worst case, OPS is just as hard as OPE, by proving a reduction of OPE to OPS. As a result, no OPS method can be more sample efficient than OPE in the worst case. We then connect BE estimation to the OPS problem, showing how BE can be used as a tool for OPS. While BE-based methods generally require stronger requirements than OPE, when those conditions are met they can be more sample efficient. Building on this insight, we propose a BE method for OPS, called Identifiable BE Selection (IBES), that has a straightforward method for selecting its own hyperparameters. We conclude with an empirical study comparing OPE and IBES, and by showing the difficulty of OPS on an offline Atari benchmark dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
