arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 1809.07435cs.LGcs.AIeess.SP
收藏

基于时间差分学习的周期性预测

Predicting Periodicity with Temporal Difference Learning

University of Alberta(阿尔伯塔大学)

Kristopher De Asis, Brendan Bennett, Richard S. Sutton

摘要:

时间差分(TD)学习是强化学习中的一种重要方法,因为它结合了动态规划和蒙特卡罗方法的思想,允许在线和增量无模型学习。TD学习的一个关键思想是,它以价值函数的形式学习有关环境的预测知识,从中可以导出其行为以解决长期顺序决策问题。代理的兴趣范围,即TD学习代理预测未来的即时或长期程度,通过贴现率参数进行调整。在本文中,我们介绍了另一种观点的贴现率,从数字信号处理的洞察力,包括复值贴现。我们的研究结果表明,将贴现率设置为适当选择的复数,可以通过TD学习在线和增量估计感兴趣信号的离散傅里叶变换(DFT)。因此,我们扩展了知识的类型表示的价值函数,我们发现是特别有用的识别周期性的奖励序列的影响。

英文摘要:

Temporal difference (TD) learning is an important approach in reinforcement learning, as it combines ideas from dynamic programming and Monte Carlo methods in a way that allows for online and incremental model-free learning. A key idea of TD learning is that it is learning predictive knowledge about the environment in the form of value functions, from which it can derive its behavior to address long-term sequential decision making problems. The agent's horizon of interest, that is, how immediate or long-term a TD learning agent predicts into the future, is adjusted through a discount rate parameter. In this paper, we introduce an alternative view on the discount rate, with insight from digital signal processing, to include complex-valued discounting. Our results show that setting the discount rate to appropriately chosen complex numbers allows for online and incremental estimation of the Discrete Fourier Transform (DFT) of a signal of interest with TD learning. We thereby extend the types of knowledge representable by value functions, which we show are particularly useful for identifying periodic effects in the reward sequence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑