驯服多智能体强化学习中的估计器方差缩减
Taming Multi-Agent Reinforcement Learning with Estimator Variance Reduction
AI总结:
针对CT-DE多智能体强化学习因单次联合动作采样导致高方差估计的问题,提出PERLA框架,在评论家中引入联合策略采样,降低方差,提升可扩展性,并在多个基准上验证了其优越性能。
摘要:
集中训练与分散执行(CT-DE)是许多领先的多智能体强化学习(MARL)算法的基础。尽管其广受欢迎,但由于它依赖于在给定状态下对联合动作的单一采样进行学习,因此存在一个关键缺陷。随着智能体在训练过程中探索并更新其策略,这些单一采样可能无法很好地代表智能体系统的实际联合策略,从而导致高方差的梯度估计,阻碍学习。为解决这一问题,我们提出了一种增强工具,可适用于任何actor-critic MARL方法。我们的框架,即性能增强强化学习装置(PERLA),在智能体训练时将智能体联合策略的采样技术引入评论家。这使得TD更新能够紧密逼近当前联合策略下的真实期望值,而不是基于给定状态下联合动作的单一采样的估计。这产生了低方差且精确的期望回报估计,最小化了通常阻碍学习的评论家估计器中的方差。此外,正如我们所展示的,通过消除联合策略单一采样带来的大部分评论家方差,PERLA使CT-DE方法能够更高效地随智能体数量扩展。理论上,我们证明PERLA在保持集中训练优势的同时,将价值估计中的方差降低到类似于分散训练的水平。实证上,我们在包括Multi-agent Mujoco和StarCraft II Multi-agent Challenge在内的一系列基准测试中展示了PERLA的优越性能及其降低估计器方差的能力。
英文摘要:
Centralised training with decentralised execution (CT-DE) serves as the foundation of many leading multi-agent reinforcement learning (MARL) algorithms. Despite its popularity, it suffers from a critical drawback due to its reliance on learning from a single sample of the joint-action at a given state. As agents explore and update their policies during training, these single samples may poorly represent the actual joint-policy of the system of agents leading to high variance gradient estimates that hinder learning. To address this problem, we propose an enhancement tool that accommodates any actor-critic MARL method. Our framework, Performance Enhancing Reinforcement Learning Apparatus (PERLA), introduces a sampling technique of the agents' joint-policy into the critics while the agents train. This leads to TD updates that closely approximate the true expected value under the current joint-policy rather than estimates from a single sample of the joint-action at a given state. This produces low variance and precise estimates of expected returns, minimising the variance in the critic estimators which typically hinders learning. Moreover, as we demonstrate, by eliminating much of the critic variance from the single sampling of the joint policy, PERLA enables CT-DE methods to scale more efficiently with the number of agents. Theoretically, we prove that PERLA reduces variance in value estimates similar to that of decentralised training while maintaining the benefits of centralised training. Empirically, we demonstrate PERLA's superior performance and ability to reduce estimator variance in a range of benchmarks including Multi-agent Mujoco, and StarCraft II Multi-agent Challenge.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。