arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1507.00353cs.AIcs.LGstat.ML
收藏

真实在线TD(λ)的实证评价

An Empirical Evaluation of True Online TD(λ)

University of Alberta(阿尔伯塔大学)

Harm van Seijen, A. Rupam Mahmood, Patrick M. Pilarski, Richard S. Sutton

摘要:

最近提出了真正的在线TD(λ)算法(van Seijen和Sutton,2014),作为时间差学习和强化学习中流行的TD(λ)算法的通用替代。真正的在线TD(λ)比传统的TD(λ)具有更好的理论特性,并且期望它也会导致更快的学习。在本文中,我们对这一假设进行了检验。具体来说,我们比较了真正的在线TD(λ)与TD(λ)在挑战性示例、随机马尔可夫奖励过程和真实世界的肌电假肢上的性能。我们使用线性函数近似,并使用表格、二进制和非二进制特征。我们沿着三个维度评估算法:计算成本,学习速度和易用性。我们的结果证实了真正的在线TD(λ)的强度:1)对于稀疏特征向量,相对于TD(λ)的计算开销是最小的;对于非稀疏特征,计算时间至多是TD(λ)的两倍,2)在所有域/表示上,真正在线TD(λ)的学习速度通常更好,但绝不会比TD(λ)的学习速度更差,3)真正的在线TD(λ)更容易使用,因为它不需要在迹线类型之间进行选择,并且它通常在步长方面更稳定。总的来说,我们的研究结果表明,真正的在线TD(λ)应该是第一选择时,寻找一个有效的,通用的TD方法。

英文摘要:

The true online TD(λ) algorithm has recently been proposed (van Seijen and Sutton, 2014) as a universal replacement for the popular TD(λ) algorithm, in temporal-difference learning and reinforcement learning. True online TD(λ) has better theoretical properties than conventional TD(λ), and the expectation is that it also results in faster learning. In this paper, we put this hypothesis to the test. Specifically, we compare the performance of true online TD(λ) with that of TD(λ) on challenging examples, random Markov reward processes, and a real-world myoelectric prosthetic arm. We use linear function approximation with tabular, binary, and non-binary features. We assess the algorithms along three dimensions: computational cost, learning speed, and ease of use. Our results confirm the strength of true online TD(λ): 1) for sparse feature vectors, the computational overhead with respect to TD(λ) is minimal; for non-sparse features the computation time is at most twice that of TD(λ), 2) across all domains/representations the learning speed of true online TD(λ) is often better, but never worse than that of TD(λ), and 3) true online TD(λ) is easier to use, because it does not require choosing between trace types, and it is generally more stable with respect to the step-size. Overall, our results suggest that true online TD(λ) should be the first choice when looking for an efficient, general-purpose TD method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑