优先经验回放与泛化之间相互作用的研究
Investigating the Interplay of Prioritized Replay and Generalization
AI总结：
本文系统研究优先经验回放与神经网络泛化的相互作用，发现PER在表格设置中有价值传播优势，但与神经网络结合时通常不优于均匀回放，并提出若干改进方案。
摘要：
经验回放，即重用过去数据以提高样本效率，在强化学习中无处不在。尽管已经引入了多种智能采样方案来提升性能，但均匀采样至今仍是最常见的方法。一个例外是优先经验回放（PER），其采样与TD误差成比例，这一思想源于动态规划中优先扫描的成功。关于PER的原始工作展示了在Atari上的改进，但后续结果好坏参半。在本文中，我们研究了PER的若干变体，试图理解PER在何处以及何时可能有用。我们在预测任务中的发现表明，尽管PER可以在表格设置中改善价值传播，但当与神经网络结合时，行为显著不同。某些缓解措施——例如延迟目标网络更新以控制泛化，以及在PER中使用期望TD误差的估计以避免追逐随机性——可以避免PER与神经网络结合时出现大的误差峰值，但通常并不优于均匀回放。在控制任务中，没有任何优先变体能够一致地优于均匀回放。我们对优先化、自举和神经网络之间的相互作用提出了新的见解，并为表格设置和噪声领域中的PER提出了若干改进。
英文摘要：
Experience replay, the reuse of past data to improve sample efficiency, is ubiquitous in reinforcement learning. Though a variety of smart sampling schemes have been introduced to improve performance, uniform sampling by far remains the most common approach. One exception is Prioritized Experience Replay (PER), where sampling is done proportionally to TD errors, inspired by the success of prioritized sweeping in dynamic programming. The original work on PER showed improvements in Atari, but follow-up results were mixed. In this paper, we investigate several variations on PER, to attempt to understand where and when PER may be useful. Our findings in prediction tasks reveal that while PER can improve value propagation in tabular settings, behavior is significantly different when combined with neural networks. Certain mitigations $-$ like delaying target network updates to control generalization and using estimates of expected TD errors in PER to avoid chasing stochasticity $-$ can avoid large spikes in error with PER and neural networks but generally do not outperform uniform replay. In control tasks, none of the prioritized variants consistently outperform uniform replay. We present new insight into the interaction between prioritization, bootstrapping, and neural networks and propose several improvements for PER in tabular settings and noisy domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
