自适应收集数据下强化学习的离线策略评估
Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data
AI总结：
本文针对自适应收集数据的表格 MDP 离线策略评估问题，为 TMIS 估计器建立高概率实例相关误差界，并恢复极小极大最优离线学习，通过仿真验证理论结果。
摘要：
为离线强化学习方法的样本复杂度建立理论保证，是使数据饥渴型强化学习算法走向实际可用的重要一步。目前大多数结果依赖于关于数据分布的不切实际假设，即数据由单一日志策略收集的一组独立同分布轨迹组成。我们考虑一个更一般的设定：数据集可能是自适应收集的。我们为表格马尔可夫决策过程（tabular MDPs）中这一广义设定下的 TMIS 离线策略评估（OPE）估计器建立理论，推导其估计误差的高概率、依赖实例的界。我们还在自适应设定下恢复了极小极大最优的离线学习。最后，我们进行仿真实验，实证分析这些估计器在自适应与非自适应机制下的行为。
英文摘要：
Developing theoretical guarantees on the sample complexity of offline RL methods is an important step towards making data-hungry RL algorithms practically viable. Currently, most results hinge on unrealistic assumptions about the data distribution -- namely that it comprises a set of i.i.d. trajectories collected by a single logging policy. We consider a more general setting where the dataset may have been gathered adaptively. We develop theory for the TMIS Offline Policy Evaluation (OPE) estimator in this generalized setting for tabular MDPs, deriving high-probability, instance-dependent bounds on its estimation error. We also recover minimax-optimal offline learning in the adaptive setting. Finally, we conduct simulations to empirically analyze the behavior of these estimators under adaptive and non-adaptive regimes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
