通过解耦优先经验回放增强连续控制任务的深度确定性策略梯度
Enhancing Deep Deterministic Policy Gradients on Continuous Control Tasks with Decoupled Prioritized Experience Replay
AI总结：
通过解耦优先经验回放提升连续控制任务中深度确定性策略梯度算法的性能
摘要：
背景：基于深度确定性策略梯度的强化学习算法使用Actor-Critic架构，其中两个网络通常使用相同的回放过渡批次进行训练。然而，Actor和Critic的学习目标和更新动态不同，引发了对统一过渡使用是否最优的疑问。\n目标：我们旨在通过解耦用于训练Actor和Critic的过渡批次来提高深度确定性策略梯度算法的性能。我们的目标是设计一种经验回放机制，通过使用分别定制的批次为每个组件提供适当的学习信号。\n方法：我们引入了解耦优先经验回放（DPER），一种新的方法，允许Actor和Critic独立地从过渡批次中进行采样。DPER可以整合到任何在连续控制领域运行的off-policy深度强化学习算法中。我们结合DPER与最先进的双延迟DDPG算法，并在标准连续控制基准上评估其性能。\n结果：DPER在多个OpenAI Gym套件中的MuJoCo任务中优于传统的经验回放策略，如 vanilla经验回放和优先经验回放。\n结论：我们的发现表明，为Actor和Critic网络解耦经验回放可以增强训练动态和最终策略质量。DPER提供了一种可推广的机制，能够提高广泛Actor-Critic off-policy强化学习算法的性能。
英文摘要：
Background: Deep Deterministic Policy Gradient-based reinforcement learning algorithms utilize Actor-Critic architectures, where both networks are typically trained using identical batches of replayed transitions. However, the learning objectives and update dynamics of the Actor and Critic differ, raising concerns about whether uniform transition usage is optimal. Objectives: We aim to improve the performance of deep deterministic policy gradient algorithms by decoupling the transition batches used to train the Actor and the Critic. Our goal is to design an experience replay mechanism that provides appropriate learning signals to each component by using separate, tailored batches. Methods: We introduce Decoupled Prioritized Experience Replay (DPER), a novel approach that allows independent sampling of transition batches for the Actor and the Critic. DPER can be integrated into any off-policy deep reinforcement learning algorithm that operates in continuous control domains. We combine DPER with the state-of-the-art Twin Delayed DDPG algorithm and evaluate its performance across standard continuous control benchmarks. Results: DPER outperforms conventional experience replay strategies such as vanilla experience replay and prioritized experience replay in multiple MuJoCo tasks from the OpenAI Gym suite. Conclusions: Our findings show that decoupling experience replay for Actor and Critic networks can enhance training dynamics and final policy quality. DPER offers a generalizable mechanism that enhances performance for a wide class of actor-critic off-policy reinforcement learning algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
