arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2111.04835cs.LG
收藏

为离线和在线策略学习提供安全的数据汇集

Safe Data Collection for Offline and Online Policy Learning

Cornell University(康奈尔大学);SC Johnson College of Business(SC约翰逊商学院);Amazon(亚马逊)

Ruihao Zhu, Branislav Kveton

摘要:

基于在线平台实验和策略学习的实际需求,我们研究了安全的数据汇集问题。具体来说,我们的目标是开发一个日志记录政策,有效地探索不同的行动,以获取信息,同时实现有竞争力的奖励与基线生产政策。我们首先表明,一个常见的做法,混合的生产政策与随机探索,尽管是安全的,是次优的信息增益最大化。然后,我们提出了一个安全的最优日志策略,通过一种新的注水技术的情况下,没有侧信息的行动的预期回报。我们通过考虑边信息来改进这种设计,并将我们的方法扩展到线性上下文模型,以考虑大量的动作。在此过程中,我们分析了我们的数据记录策略如何影响离线(线)-策略学习中的错误,并通过使用合成和MNIST数据集进行广泛的数值实验来实证验证我们的设计的好处。为了进一步证明我们的方法的通用性,我们还考虑了安全的在线学习设置。通过自适应地应用我们的技术,我们开发的安全阶段消除(SafePE)算法,可以实现最佳的遗憾界限,只有对数数量的政策更新。

英文摘要:

Motivated by practical needs of experimentation and policy learning in online platforms, we study the problem of safe data collection. Specifically, our goal is to develop a logging policy that efficiently explores different actions to elicit information while achieving competitive reward with a baseline production policy. We first show that a common practice of mixing the production policy with randomized exploration, despite being safe, is sub-optimal in maximizing information gain. Then, we propose a safe optimal logging policy via a novel water-filling technique for the case when no side information about the actions' expected reward is available. We improve upon this design by considering side information and also extend our approaches to the linear contextual model to account for a large number of actions. Along the way, we analyze how our data logging policies impact errors in off(line)-policy learning and empirically validate the benefit of our design by conducting extensive numerical experiments with synthetic and MNIST datasets. To further demonstrate the generality of our approach, we also consider the safe online learning setting. By adaptively applying our techniques, we develop the Safe Phased-Elimination (SafePE) algorithm that can achieve optimal regret bound with only logarithmic number of policy updates.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑