重新思考大规模强化学习中集成策略梯度中的策略多样性
Rethinking Policy Diversity in Ensemble Policy Gradient in Large-Scale Reinforcement Learning
AI总结：
本文提出耦合策略优化方法，通过KL约束调节策略多样性，提升大规模强化学习中的探索效率和学习性能。
摘要：
将强化学习扩展到数万并行环境需要克服单一策略有限的探索能力。基于集成的策略梯度方法，通过多个策略收集多样化样本，最近被提出以促进探索。然而，仅仅扩大探索空间并不总是增强学习能力，因为过度探索可能会降低探索质量或破坏训练稳定性。在本工作中，我们从理论上分析了策略间多样性对策略集成学习效率的影响，并提出耦合策略优化方法，通过策略间的KL约束来调节多样性。所提出的方法实现了有效的探索，并在多个任务中优于SAPG、PBT和PPO等强大基线方法，在样本效率和最终性能方面均表现优异。此外，对训练过程中策略多样性和有效样本大小的分析表明，跟随策略自然分布在领导者周围，显示出结构化和高效的探索行为。我们的结果表明，在适当调节下，多样化的探索是实现集成策略梯度方法稳定且高效学习的关键。项目页面：https://naoki04.github.io/paper-cpo/
英文摘要：
Scaling reinforcement learning to tens of thousands of parallel environments requires overcoming the limited exploration capacity of a single policy. Ensemble-based policy gradient methods, which employ multiple policies to collect diverse samples, have recently been proposed to promote exploration. However, merely broadening the exploration space does not always enhance learning capability, since excessive exploration can reduce exploration quality or compromise training stability. In this work, we theoretically analyze the impact of inter-policy diversity on learning efficiency in policy ensembles, and propose Coupled Policy Optimization which regulates diversity through KL constraints between policies. The proposed method enables effective exploration and outperforms strong baselines such as SAPG, PBT, and PPO across multiple tasks, including challenging dexterous manipulation, in terms of both sample efficiency and final performance. Furthermore, analysis of policy diversity and effective sample size during training reveals that follower policies naturally distribute around the leader, demonstrating the emergence of structured and efficient exploratory behavior. Our results indicate that diverse exploration under appropriate regulation is key to achieving stable and sample-efficient learning in ensemble policy gradient methods. Project page at https://naoki04.github.io/paper-cpo/ .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
