arXiv 2109.051102021-09-10cs.LGcs.AI

四个房间环境下的离线预测学习算法的实证比较

An Empirical Comparison of Off-policy Prediction Learning Algorithms in the Four Rooms Environment

University of Alberta（阿尔伯塔大学）；DeepMind

Sina Ghiassian, Richard S. Sutton

备注：: 13 pages

摘要：

在过去的十年中，已经提出了许多离线预测学习算法，但仍然不清楚哪些算法比其他算法学习得更快。我们在两个小任务上实证比较了11个离线预测学习算法与线性函数逼近：房间任务和高方差房间任务。这些任务的设计使得快速学习具有挑战性。在“房间”任务中，重要性采样比率的乘积可以与$2^{14}$一样大，有时可能为2。为了控制由重要性采样比率的乘积引起的高方差，步长应该设置得小，这反过来减慢了学习。“高方差房间”任务更为极端，因为比率的乘积可能与$2^{14}\times 25$一样大。本文基于Ghiassian和Sutton（2021）对政策外预测学习算法的实证研究。我们考虑与他们相同的一组算法，并采用相同的实验方法。所考虑的算法是：非策略TD（$λ$）、五种强制TD算法、两种强调TD算法、树备份（$λ$）、Vtrace（$λ$）和ABTD（$ζ$）。我们发现，算法的性能受到由重要性采样比率引起的方差的高度影响。数据显示，树备份（$λ$）、Vtrace（$λ$）和ABTD（$ζ$）不像其他算法那样受高方差的影响，但是它们以对于不存在高方差的任务而言过于限制的方式限制有效引导参数。我们观察到强调TD（$λ$）往往比其他算法具有更低的渐进误差，但在某些情况下可能学习得更慢。我们建议从业者的算法，根据他们感兴趣的问题，并建议的方法，可以应用于特定的算法，可能会导致大幅改善算法。

英文摘要：

Many off-policy prediction learning algorithms have been proposed in the past decade, but it remains unclear which algorithms learn faster than others. We empirically compare 11 off-policy prediction learning algorithms with linear function approximation on two small tasks: the Rooms task, and the High Variance Rooms task. The tasks are designed such that learning fast in them is challenging. In the Rooms task, the product of importance sampling ratios can be as large as $2^{14}$ and can sometimes be two. To control the high variance caused by the product of the importance sampling ratios, step size should be set small, which in turn slows down learning. The High Variance Rooms task is more extreme in that the product of the ratios can become as large as $2^{14}\times 25$. This paper builds upon the empirical study of off-policy prediction learning algorithms by Ghiassian and Sutton (2021). We consider the same set of algorithms as theirs and employ the same experimental methodology. The algorithms considered are: Off-policy TD($λ$), five Gradient-TD algorithms, two Emphatic-TD algorithms, Tree Backup($λ$), Vtrace($λ$), and ABTD($ζ$). We found that the algorithms' performance is highly affected by the variance induced by the importance sampling ratios. The data shows that Tree Backup($λ$), Vtrace($λ$), and ABTD($ζ$) are not affected by the high variance as much as other algorithms but they restrict the effective bootstrapping parameter in a way that is too limiting for tasks where high variance is not present. We observed that Emphatic TD($λ$) tends to have lower asymptotic error than other algorithms, but might learn more slowly in some cases. We suggest algorithms for practitioners based on their problem of interest, and suggest approaches that can be applied to specific algorithms that might result in substantially improved algorithms.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

摘要：

英文摘要：

相关论文