跨时间尺度分离价值函数
Separating value functions across time-scales
摘要：
在许多有限时域情景强化学习（RL）设置中，期望针对未贴现回报设置进行优化，例如Atari，目标是在长期生存的同时收集最多的点。然而，用数学方法学习这个目标可能很困难（甚至是棘手的）。因此，时间折扣通常用于在较短的有效规划范围内进行优化。这有可能使优化目标偏离未贴现目标的风险。在这种偏差不可接受的设置中-系统必须以更高的折扣优化更长的时间范围-价值函数近似器的目标可能会增加方差，导致学习困难。我们提出了一种扩展的时间差（TD）学习，我们称之为TD（$Δ$），它将一个值函数分解为一系列组件的基础上的差异价值函数具有较小的折扣因子。将较长范围的价值函数分离为这些组件在可伸缩性和性能方面具有有用的特性。我们讨论了这些属性，并在某些设置中显示了理论和经验上的改进，超过标准TD学习。
英文摘要：
In many finite horizon episodic reinforcement learning (RL) settings, it is desirable to optimize for the undiscounted return - in settings like Atari, for instance, the goal is to collect the most points while staying alive in the long run. Yet, it may be difficult (or even intractable) mathematically to learn with this target. As such, temporal discounting is often applied to optimize over a shorter effective planning horizon. This comes at the risk of potentially biasing the optimization target away from the undiscounted goal. In settings where this bias is unacceptable - where the system must optimize for longer horizons at higher discounts - the target of the value function approximator may increase in variance leading to difficulties in learning. We present an extension of temporal difference (TD) learning, which we call TD($Δ$), that breaks down a value function into a series of components based on the differences between value functions with smaller discount factors. The separation of a longer horizon value function into these components has useful properties in scalability and performance. We discuss these properties and show theoretic and empirical improvements over standard TD learning in certain settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
