arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1906.04328cs.LGcs.AIstat.ML
收藏

非政策预测的重要性恢复

Importance Resampling for Off-policy Prediction

University of Alberta(阿尔伯塔大学);McGill University(麦吉尔大学);Huawei(华为)

Matthew Schlegel, Wesley Chung, Daniel Graves, Jian Qian, Martha White

摘要:

重要性采样(IS)是强化学习中用于偏离策略预测的常见重加权策略。虽然它是一致和无偏的,但它可能导致对价值函数的权重的高方差更新。在这项工作中,我们探讨了一个rescue战略作为一种替代重新加权。我们提出了重要性恢复(IR)的政策预测,从重放缓冲区重新采样的经验,并适用于标准的政策更新。该方法避免了在更新中使用重要性采样比率,而是在更新之前校正分布。我们的特点的偏见和一致性的IR,特别是相比加权IS(WIS)。我们证明了在几个微观世界,IR提高了样本效率和较低的方差更新,相比,IS和几个方差减少的IS策略,包括变体的WIS和V-跟踪剪辑IS比率。我们还提供了一个演示,显示IR提高了IS学习的价值函数从图像中的赛车模拟器。

英文摘要:

Importance sampling (IS) is a common reweighting strategy for off-policy prediction in reinforcement learning. While it is consistent and unbiased, it can result in high variance updates to the weights for the value function. In this work, we explore a resampling strategy as an alternative to reweighting. We propose Importance Resampling (IR) for off-policy prediction, which resamples experience from a replay buffer and applies standard on-policy updates. The approach avoids using importance sampling ratios in the update, instead correcting the distribution before the update. We characterize the bias and consistency of IR, particularly compared to Weighted IS (WIS). We demonstrate in several microworlds that IR has improved sample efficiency and lower variance updates, as compared to IS and several variance-reduced IS strategies, including variants of WIS and V-trace which clips IS ratios. We also provide a demonstration showing IR improves over IS for learning a value function from images in a racing car simulator.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑