arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1811.02597cs.LGcs.AIstat.ML
收藏

在线离线预测

Online Off-policy Prediction

University of Alberta(阿尔伯塔大学)

Sina Ghiassian, Andrew Patterson, Martha White, Richard S. Sutton, Adam White

摘要:

本文研究了在线预测学习的问题,学习过程中不断与环境的代理交互。智能体的预测取决于特定的行为方式,表现为价值函数。但是,用于选择操作和生成行为数据的行为可能与用于定义预测的行为不同,因此样本是在策略外生成的。长期以来,在线和离线学习行为相关预测的能力一直被认为是预测知识学习系统的关键能力,但几十年来仍然是一个开放的算法挑战。问题在于大多数预测算法的核心是时间差(TD)学习更新:结合自举、偏离策略采样和函数近似可能会导致值估计发散。一个突破来了一个新的目标函数,承认随机梯度下降变种TD的发展。从那时起,许多健全的在线离线预测算法已经开发出来,但一直有有限的实证工作调查的相对优点的所有变量。本文旨在填补这些经验空白,并澄清每种方法背后的关键思想。我们总结了大量的文献离策略学习,重点是1-方法,使用计算线性的特征数量和下的离策采样收敛,和2-其他方法,已被证明是有用的非固定的,非线性函数逼近。我们在两个具有挑战性的微观世界中提供了一个非政策预测方法的实证研究。我们报告了每种方法的参数敏感性,经验收敛速度和最终性能,提供了新的见解,应该使从业者能够成功地将这些新方法扩展到大规模应用。[摘要摘要]

英文摘要:

This paper investigates the problem of online prediction learning, where learning proceeds continuously as the agent interacts with an environment. The predictions made by the agent are contingent on a particular way of behaving, represented as a value function. However, the behavior used to select actions and generate the behavior data might be different from the one used to define the predictions, and thus the samples are generated off-policy. The ability to learn behavior-contingent predictions online and off-policy has long been advocated as a key capability of predictive-knowledge learning systems but remained an open algorithmic challenge for decades. The issue lies with the temporal difference (TD) learning update at the heart of most prediction algorithms: combining bootstrapping, off-policy sampling and function approximation may cause the value estimate to diverge. A breakthrough came with the development of a new objective function that admitted stochastic gradient descent variants of TD. Since then, many sound online off-policy prediction algorithms have been developed, but there has been limited empirical work investigating the relative merits of all the variants. This paper aims to fill these empirical gaps and provide clarity on the key ideas behind each method. We summarize the large body of literature on off-policy learning, focusing on 1- methods that use computation linear in the number of features and are convergent under off-policy sampling, and 2- other methods which have proven useful with non-fixed, nonlinear function approximation. We provide an empirical study of off-policy prediction methods in two challenging microworlds. We report each method's parameter sensitivity, empirical convergence rate, and final performance, providing new insights that should enable practitioners to successfully extend these new methods to large-scale applications.[Abridged abstract]

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑