反向经验回放的一个更紧收敛性证明
A Tighter Convergence Proof of Reverse Experience Replay
AI总结：
本文针对反向经验回放（RER）现有理论仅适用于小学习率和短序列的局限，提出更紧的收敛性分析，证明其在更大学习率和更长序列下仍能收敛。
摘要：
在强化学习中，反向经验回放（Reverse Experience Replay, RER）是最近提出的一种算法，其样本复杂度优于经典经验回放方法。RER 要求学习算法按逆序通过连续的状态-动作-奖励元组更新参数。然而，最新的理论分析仅适用于极小学习率和较短连续步数，其收敛速度慢于那些不使用 RER 的大学习率算法。鉴于这一理论与实证之间的差距，我们提供了一个更紧的分析，缓解了对学习率和连续步数长度的限制。此外，我们从理论上证明 RER 在更大的学习率和更长的序列下能够收敛。
英文摘要：
In reinforcement learning, Reverse Experience Replay (RER) is a recently proposed algorithm that attains better sample complexity than the classic experience replay method. RER requires the learning algorithm to update the parameters through consecutive state-action-reward tuples in reverse order. However, the most recent theoretical analysis only holds for a minimal learning rate and short consecutive steps, which converge slower than those large learning rate algorithms without RER. In view of this theoretical and empirical gap, we provide a tighter analysis that mitigates the limitation on the learning rate and the length of consecutive steps. Furthermore, we show theoretically that RER converges with a larger learning rate and a longer sequence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
