arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.05741cs.LGstat.ML
收藏

非政策评估的非渐近置信区间:原始和对偶界

Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Bounds

University of Texas at Austin(德克萨斯大学奥斯汀分校);Tsinghua University(清华大学)

Yihao Feng, Ziyang Tang, Na Zhang, Qiang Liu

摘要:

离线策略评估(OPE)是基于先前在不同策略下收集的离线数据来估计给定策略的预期回报的任务。因此,OPE是将强化学习应用于医疗等现实世界领域的关键一步,在这些领域,交互式数据汇集是昂贵的,甚至是不安全的。由于观察到的数据往往是嘈杂和有限的,它是必不可少的,以提供严格的不确定性量化,而不仅仅是一个点估计,当应用OPE作出高风险的决策。本文研究了在无限时域外策略评估中构造非渐近置信区间的问题,这仍然是一个具有挑战性的公开问题。我们开发了一个实用的算法,通过原始对偶优化为基础的方法,利用内核贝尔曼损失(KBL)冯等。(2019)和一个新的鞅浓度不等式的KBL适用于时间依赖的数据与未知的混合条件。我们的算法对数据和Q函数的函数类做了最小的假设,并适用于行为不可知的设置,其中数据是在任意未知行为策略的混合下收集的。我们提出的实证结果清楚地表明,我们的方法比现有的方法的优势。

英文摘要:

Off-policy evaluation (OPE) is the task of estimating the expected reward of a given policy based on offline data previously collected under different policies. Therefore, OPE is a key step in applying reinforcement learning to real-world domains such as medical treatment, where interactive data collection is expensive or even unsafe. As the observed data tends to be noisy and limited, it is essential to provide rigorous uncertainty quantification, not just a point estimation, when applying OPE to make high stakes decisions. This work considers the problem of constructing non-asymptotic confidence intervals in infinite-horizon off-policy evaluation, which remains a challenging open question. We develop a practical algorithm through a primal-dual optimization-based approach, which leverages the kernel Bellman loss (KBL) of Feng et al.(2019) and a new martingale concentration inequality of KBL applicable to time-dependent data with unknown mixing conditions. Our algorithm makes minimum assumptions on the data and the function class of the Q-function, and works for the behavior-agnostic settings where the data is collected under a mix of arbitrary unknown behavior policies. We present empirical results that clearly demonstrate the advantages of our approach over existing methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑