OPERA：基于多估计器重加权聚合的自动离线策略评估
OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators
AI总结：
针对离线策略评估（OPE）算法选型难的问题，本文提出OPERA算法，可自适应融合多类OPE估计器，具备一致性，能在医疗、机器人领域选出更优策略，提升了通用离线RL离策略评估框架的易用性。
摘要：
离线策略评估（Offline Policy Evaluation，简称OPE）可利用从其他策略收集的历史交互数据，评估并估计新型序贯决策策略的性能。在未对新策略性能形成可靠估计的情况下就进行在线评估，可能会产生成本高昂、不安全甚至危险的后果，在教育和医疗领域尤其如此。过去十年间研究者提出了多种OPE估计器，其中很多都带有超参数且需要训练。但遗憾的是，目前仍不清楚如何为不同任务和领域选择最优的OPE算法。本文提出了一种新算法，可在给定数据集的情况下自适应融合一组OPE估计器，无需通过统计流程进行显式选择。我们证明了该估计器具备一致性，且满足策略评估的若干理想性质。此外我们还证明，与其他替代方法相比，该估计器可在医疗和机器人领域筛选出性能更优的策略。本研究为提升离线强化学习通用、与估计器无关的离策略评估框架的易用性做出了贡献。
英文摘要：
Offline policy evaluation (OPE) allows us to evaluate and estimate a new sequential decision-making policy's performance by leveraging historical interaction data collected from other policies. Evaluating a new policy online without a confident estimate of its performance can lead to costly, unsafe, or hazardous outcomes, especially in education and healthcare. Several OPE estimators have been proposed in the last decade, many of which have hyperparameters and require training. Unfortunately, choosing the best OPE algorithm for each task and domain is still unclear. In this paper, we propose a new algorithm that adaptively blends a set of OPE estimators given a dataset without relying on an explicit selection using a statistical procedure. We prove that our estimator is consistent and satisfies several desirable properties for policy evaluation. Additionally, we demonstrate that when compared to alternative approaches, our estimator can be used to select higher-performing policies in healthcare and robotics. Our work contributes to improving ease of use for a general-purpose, estimator-agnostic, off-policy evaluation framework for offline RL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
