arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2211.07614cs.LG
收藏

面向数据驱动离线模拟的在线强化学习

Towards Data-Driven Offline Simulations for Online Reinforcement Learning

Microsoft Azure AI(微软Azure人工智能部门);Microsoft Research NYC(微软研究院纽约分院);University of Michigan, Ann Arbor(密歇根大学安娜堡分校)

Shengpu Tang, Felipe Vieira Frujeri, Dipendra Misra, Alex Lamb, John Langford, Paul Mineiro, Sebastian Kochman

摘要:

现代决策系统,从机器人到网络推荐引擎,都需要适应用户偏好、不断变化的环境甚至新的任务。然而,将动态学习代理(而不是固定策略)部署到生产系统中仍然很少见,因为它被认为是不安全的。使用历史数据来推理学习算法,类似于应用于固定策略的离线策略评估(OPE),可以帮助从业者评估并最终将此类自适应代理部署到生产环境中。在这项工作中,我们正式离线学习者模拟(OLS)的强化学习(RL),并提出了一种新的评估协议,衡量保真度和效率的模拟。对于具有复杂高维观测的环境,我们提出了一种半参数方法,该方法利用了潜在状态发现的最新进展,以实现准确有效的离线模拟。在初步的实验中,我们展示了我们的方法相比完全非参数基线的优势。重现这些实验的代码将在https://github.com/microsoft/rl-offline-simulation上提供。

英文摘要:

Modern decision-making systems, from robots to web recommendation engines, are expected to adapt: to user preferences, changing circumstances or even new tasks. Yet, it is still uncommon to deploy a dynamically learning agent (rather than a fixed policy) to a production system, as it's perceived as unsafe. Using historical data to reason about learning algorithms, similar to offline policy evaluation (OPE) applied to fixed policies, could help practitioners evaluate and ultimately deploy such adaptive agents to production. In this work, we formalize offline learner simulation (OLS) for reinforcement learning (RL) and propose a novel evaluation protocol that measures both fidelity and efficiency of the simulation. For environments with complex high-dimensional observations, we propose a semi-parametric approach that leverages recent advances in latent state discovery in order to achieve accurate and efficient offline simulations. In preliminary experiments, we show the advantage of our approach compared to fully non-parametric baselines. The code to reproduce these experiments will be made available at https://github.com/microsoft/rl-offline-simulation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑