CUER：面向离策略连续深度强化学习算法的校正均匀经验回放
CUER: Corrected Uniform Experience Replay for Off-Policy Continuous Deep Reinforcement Learning Algorithms
AI总结：
针对经验回放中重要性动态变化及旧策略导致离策略更新偏差的问题，提出CUER算法，通过公平随机采样并使状态分布更在策略，提升连续控制算法的样本效率、性能与稳定性。
摘要：
经验回放机制的运用使智能体能够在多个场合有效利用其经验。在以往研究中，转移样本的采样概率会根据其相对重要性进行修改。每次迭代后为回放缓冲区中的每个转移样本重新分配采样概率被认为效率极低。因此，为提高计算效率，经验回放优先级算法会在某个转移样本被采样时重新评估其重要性。然而，当智能体的策略和价值函数被迭代更新时，转移样本的相对重要性会经历动态调整。此外，经验回放是一种保留由智能体过去策略生成的转移样本的机制，这些过去策略可能与智能体最新策略存在显著差异。与智能体最新策略的偏差越大，离策略更新的频率就越高，从而对智能体性能产生负面影响。本文提出了一种名为校正均匀经验回放（Corrected Uniform Experience Replay，CUER）的新颖算法，该算法随机采样已存储的经验，同时考虑所有其他经验之间的公平性，并且通过使被采样的状态分布更接近在策略状态分布，不忽视转移重要性的动态特性。CUER在样本效率、最终性能以及训练期间策略稳定性方面，为离策略连续控制算法带来了有前景的改进。
英文摘要：
The utilization of the experience replay mechanism enables agents to effectively leverage their experiences on several occasions. In previous studies, the sampling probability of the transitions was modified based on their relative significance. The process of reassigning sample probabilities for every transition in the replay buffer after each iteration is considered extremely inefficient. Hence, in order to enhance computing efficiency, experience replay prioritization algorithms reassess the importance of a transition as it is sampled. However, the relative importance of the transitions undergoes dynamic adjustments when the agent's policy and value function are iteratively updated. Furthermore, experience replay is a mechanism that retains the transitions generated by the agent's past policies, which could potentially diverge significantly from the agent's most recent policy. An increased deviation from the agent's most recent policy results in a greater frequency of off-policy updates, which has a negative impact on the agent's performance. In this paper, we develop a novel algorithm, Corrected Uniform Experience Replay (CUER), which stochastically samples the stored experience while considering the fairness among all other experiences without ignoring the dynamic nature of the transition importance by making sampled state distribution more on-policy. CUER provides promising improvements for off-policy continuous control algorithms in terms of sample efficiency, final performance, and stability of the policy during the training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
