带经验回放的时序差分学习的有限时间分析
Finite-Time Analysis of Temporal Difference Learning with Experience Replay
AI总结：
针对时序差分（TD）学习的有限时间行为研究缺口及经验回放理论影响不明问题，本文通过马尔可夫噪声项分解，给出带经验回放TD学习的有限时间误差界，证明回放缓冲区和小批量大小可有效控制恒定步长误差。
摘要：
时序差分（TD）学习被广泛认为是强化学习（RL）中最流行的算法之一。尽管其应用广泛，但研究人员直到最近才开始积极研究其有限时间行为，包括均方误差的有限时间界和样本复杂度。在实证方面，经验回放是深度强化学习算法成功的关键因素，但其对强化学习的理论影响尚未被完全理解。本文提出了一种马尔可夫噪声项的简单分解方法，并为带经验回放的TD学习提供了有限时间误差界。具体而言，在马尔可夫观测模型下，我们证明对于平均迭代和最终迭代两种情况，由恒定步长引起的误差项可通过回放缓冲区的大小和从经验回放缓冲区中采样的小批量大小得到有效控制。
英文摘要：
Temporal-difference (TD) learning is widely regarded as one of the most popular algorithms in reinforcement learning (RL). Despite its widespread use, it has only been recently that researchers have begun to actively study its finite time behavior, including the finite time bound on mean squared error and sample complexity. On the empirical side, experience replay has been a key ingredient in the success of deep RL algorithms, but its theoretical effects on RL have yet to be fully understood. In this paper, we present a simple decomposition of the Markovian noise terms and provide finite-time error bounds for TD-learning with experience replay. Specifically, under the Markovian observation model, we demonstrate that for both the averaged iterate and final iterate cases, the error term induced by a constant step-size can be effectively controlled by the size of the replay buffer and the mini-batch sampled from the experience replay buffer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
