基于随机重洗的体验回放
Experience Replay with Random Reshuffling
AI总结：
本文提出将随机重洗方法扩展到强化学习的经验回放，通过理论分析和实验验证其在深度强化学习中的有效性。
摘要：
经验回放是强化学习中用于稳定学习和提高样本效率的关键组件。其典型实现是从回放缓冲区中以有放回的方式采样转移。相比之下，在固定数据集的监督学习中，常见的做法是每个epoch都对数据集进行洗牌并按顺序消耗数据，这被称为随机重洗（RR）。RR在理论上具有更好的收敛性质，并在实验证明中优于有放回采样。为了利用RR在强化学习中的优势，我们提出了扩展RR到经验回放的采样方法，包括均匀和优先级设置，并通过理论分析和模拟分析其性质。我们在Atari基准上评估了我们的采样方法，展示了其在深度强化学习中的有效性。代码可在https://github.com/pfnet-research/errr获得。
英文摘要：
Experience replay is a key component in reinforcement learning for stabilizing learning and improving sample efficiency. Its typical implementation samples transitions with replacement from a replay buffer. In contrast, in supervised learning with a fixed dataset, it is a common practice to shuffle the dataset every epoch and consume data sequentially, which is called random reshuffling (RR). RR enjoys theoretically better convergence properties and has been shown to outperform with-replacement sampling empirically. To leverage the benefits of RR in reinforcement learning, we propose sampling methods that extend RR to experience replay, both in uniform and prioritized settings, and analyze their properties via theoretical analysis and simulations. We evaluate our sampling methods on Atari benchmarks, demonstrating their effectiveness in deep reinforcement learning. Code is available at https://github.com/pfnet-research/errr.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
