arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2202.13337cs.LGcs.IR
收藏

面向不确定性的鲁棒离策略学习

Towards Robust Off-policy Learning for Runtime Uncertainty

Walmart Labs(沃尔玛实验室);University of California, Berkeley(加州大学伯克利分校);Instacart;LinkedIn(领英)

Da Xu, Yuting Ye, Chuanwei Ruan, Bo Yang

摘要:

离策略学习在在线部署之前优化和评估策略方面起着关键作用。然而,在实时服务过程中,我们观察到各种各样的干预和约束,导致在线和离线设置之间的不一致,我们总结和术语为运行时的不确定性。由于其异常性和稀有性,这种不确定性不能从记录的数据中学习。为了保证一定程度的鲁棒性,考虑到运行时的不确定性,我们沿着对抗方向扰动了非策略估计器。它允许所得到的估计是强大的,不仅观察到的,但也意想不到的运行时的不确定性。利用这一思想,我们带来了运行时的不确定性鲁棒性的三个主要的离策略学习方法:逆倾向得分方法,奖励模型方法,和双鲁棒方法。我们从理论上证明我们的方法的鲁棒性运行时的不确定性,并证明其有效性,使用模拟和现实世界的在线实验。

英文摘要:

Off-policy learning plays a pivotal role in optimizing and evaluating policies prior to the online deployment. However, during the real-time serving, we observe varieties of interventions and constraints that cause inconsistency between the online and offline settings, which we summarize and term as runtime uncertainty. Such uncertainty cannot be learned from the logged data due to its abnormality and rareness nature. To assert a certain level of robustness, we perturb the off-policy estimators along an adversarial direction in view of the runtime uncertainty. It allows the resulting estimators to be robust not only to observed but also unexpected runtime uncertainties. Leveraging this idea, we bring runtime-uncertainty robustness to three major off-policy learning methods: the inverse propensity score method, reward-model method, and doubly robust method. We theoretically justify the robustness of our methods to runtime uncertainty, and demonstrate their effectiveness using both the simulation and the real-world online experiments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑