We consider the problem of off-policy evaluation (OPE) in reinforcement learning (RL), where the goal is to estimate the performance of an evaluation policy, $π_e$, using a fixed dataset, $\mathcal{D}$, collected by one or more policies that may be different from $π_e$. Current OPE algorithms may produce poor OPE estimates under policy distribution shift i.e., when the probability of a particular state-action pair occurring under $π_e$ is very different from the probability of that same pair occurring in $\mathcal{D}$ (Voloshin et al. 2021, Fu et al. 2021). In this work, we propose to improve the accuracy of OPE estimators by projecting the high-dimensional state-space into a low-dimensional state-space using concepts from the state abstraction literature. Specifically, we consider marginalized importance sampling (MIS) OPE algorithms which compute state-action distribution correction ratios to produce their OPE estimate. In the original ground state-space, these ratios may have high variance which may lead to high variance OPE. However, we prove that in the lower-dimensional abstract state-space the ratios can have lower variance resulting in lower variance OPE. We then highlight the challenges that arise when estimating the abstract ratios from data, identify sufficient conditions to overcome these issues, and present a minimax optimization problem whose solution yields these abstract ratios. Finally, our empirical evaluation on difficult, high-dimensional state-space OPE tasks shows that the abstract ratios can make MIS OPE estimators achieve lower mean-squared error and more robust to hyperparameter tuning than the ground ratios.
Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learning
UC Santa Barbara（加州大学圣巴巴拉分校）
我们考虑了强化学习的离线策略评估问题，其目标是使用通过运行日志策略$μ$收集的离线数据来估计目标策略$π$的预期奖励。标准的基于重要性抽样的方法在这个问题上受到方差的影响，该方差随时间范围$H$呈指数级增长，这激发了人们对打破“地平线诅咒”的替代方案的兴趣（Liu et al. 2018，Xie et al. 2019）。特别是，它表明，边缘化的重要性采样（MIS）的方法可以用来实现估计误差的顺序$O(H^3/ n)$的均方误差（MSE）下的情节马尔可夫决策过程模型与有限的状态和潜在的无限的行动。然而，MSE界仍然是$H$的因子，远离$Ω(H^2/n)$阶的Cramer-Rao下界。在本文中，我们证明了一个简单的修改MIS估计，我们可以渐近地达到Cramer-Rao下界，只要作用空间是有限的。我们还提供了一个一般的方法来构建MIS估计与高概率误差界。
IntOPE: Off-Policy Evaluation in the Presence of Interference
John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保罗森工程与应用科学学院）；College of Computer Science, Zhejiang University（浙江大学计算机学院）；University of Waterloo（滑铁卢大学）
From Semantic to Token Communication: The Next Paradigm for Large-Model-Driven 6G Intelligent Connectivity
Beijing Institute of Technology（北京理工大学）；Zhongguancun Academy（中关村学院）；The University of Hong Kong（香港大学）；University of Surrey（萨里大学）；Shanghai Jiao Tong University（上海交通大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Southeast University（东南大学）；University of Southampton（南安普顿大学）；Ocean University of China（中国海洋大学）；Singapore University of Technology and Design（新加坡科技设计大学）
