arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.13554cs.LGecon.EMstat.ML
收藏

批更新策略下相依样本Bandit算法的离策略评价

Off-Policy Evaluation of Bandit Algorithm from Dependent Samples under Batch Update Policy

CyberAgent Inc.(赛博特工公司)

Masahiro Kato, Yusuke Kaneko

摘要:

政策外评估(OPE)的目标是使用通过行为策略获得的历史数据来评估新策略。然而,由于上下文强盗算法基于过去的观察来更新策略,因此样本不是独立且同分布的(i.i.d.)。本文通过构造相依样本的鞅差序列(MDS)的估计量来解决这个问题。在数据生成过程中,我们不假设策略的收敛性,但策略在一定时期内使用相同的条件概率选择行动。然后,我们得到了一个渐近正常的估计值的评价政策。作为我们的方法的另一个优点,基于批处理的方法同时解决了支持不足的问题。使用基准和真实世界的数据集,我们实验证实了所提出的方法的有效性。

英文摘要:

The goal of off-policy evaluation (OPE) is to evaluate a new policy using historical data obtained via a behavior policy. However, because the contextual bandit algorithm updates the policy based on past observations, the samples are not independent and identically distributed (i.i.d.). This paper tackles this problem by constructing an estimator from a martingale difference sequence (MDS) for the dependent samples. In the data-generating process, we do not assume the convergence of the policy, but the policy uses the same conditional probability of choosing an action during a certain period. Then, we derive an asymptotically normal estimator of the value of an evaluation policy. As another advantage of our method, the batch-based approach simultaneously solves the deficient support problem. Using benchmark and real-world datasets, we experimentally confirm the effectiveness of the proposed method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑