MEPG：一种用于深度强化学习的最小包围策略梯度框架
MEPG: A Minimalist Ensemble Policy Gradient Framework for Deep Reinforcement Learning
摘要：
在强化学习（RL）智能体的训练过程中，由于智能体的行为随时间变化，训练数据的分布是非平稳的。因此，存在代理过度专业化到特定分布的风险，并且其性能在更大范围内受到影响。EnergyRL可以通过学习一个强大的策略来缓解这个问题。然而，它遭受沉重的计算资源消耗，由于新引入的价值和政策的功能。在本文中，为了避免臭名昭著的资源消耗问题，我们设计了一个新颖而简单的集成深度RL框架，将多个模型集成到一个模型中。具体来说，我们提出了\underline{M}最小\underline{E} numerical \underline{P} policy \underline{G} radient framework（MEPG），它通过利用修改的dropout算子引入最小集合一致的Bellman更新。MEPG通过保持Bellman方程两边的丢弃一致性来保持系综性质。此外，dropout算子还提高了MEPG的泛化能力。此外，我们从理论上表明，在MEPG的政策评估阶段保持两个同步的深高斯过程。为了验证MEPG框架的泛化能力，我们在健身房模拟器上进行了实验，这表明MEPG框架优于或达到了与当前最先进的集成方法和无模型方法相似的性能水平，而不会增加额外的计算资源成本。
英文摘要：
During the training of a reinforcement learning (RL) agent, the distribution of training data is non-stationary as the agent's behavior changes over time. Therefore, there is a risk that the agent is overspecialized to a particular distribution and its performance suffers in the larger picture. Ensemble RL can mitigate this issue by learning a robust policy. However, it suffers from heavy computational resource consumption due to the newly introduced value and policy functions. In this paper, to avoid the notorious resources consumption issue, we design a novel and simple ensemble deep RL framework that integrates multiple models into a single model. Specifically, we propose the \underline{M}inimalist \underline{E}nsemble \underline{P}olicy \underline{G}radient framework (MEPG), which introduces minimalist ensemble consistent Bellman update by utilizing a modified dropout operator. MEPG holds ensemble property by keeping the dropout consistency of both sides of the Bellman equation. Additionally, the dropout operator also increases MEPG's generalization capability. Moreover, we theoretically show that the policy evaluation phase in the MEPG maintains two synchronized deep Gaussian Processes. To verify the MEPG framework's ability to generalize, we perform experiments on the gym simulator, which presents that the MEPG framework outperforms or achieves a similar level of performance as the current state-of-the-art ensemble methods and model-free methods without increasing additional computational resource costs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
