arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.00309cs.LGcs.AI
收藏

用于同策略强化学习的基于单峰概率分布的连续动作空间离散化

Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learning

Nanjing University(南京大学);Research Center for Novel Technology of Intelligent Equipment, Nanjing University(南京大学智能装备新技术研究中心);Chinese Academy of Sciences(中国科学院);University of Chinese Academy of Sciences(中国科学院大学)

Yuanyang Zhu, Zhi Wang, Yuanheng Zhu, Chunlin Chen, Dongbin Zhao

AI总结:

本文提出利用 Poisson 单峰概率分布约束离散策略,以保留连续动作空间的顺序与连续性,从而降低策略梯度方差并提升同策略强化学习在复杂控制任务中的收敛速度与性能。

摘要:

对于同策略强化学习而言,将连续控制的动作空间离散化可以轻松表达多种模态,并且易于优化。然而,如果不考虑离散原子动作之间固有的顺序关系,离散动作数量的激增可能会带来不良特性,并导致策略梯度估计器具有更高的方差。本文提出一种简洁的架构,通过利用 Poisson 概率分布将离散策略约束为单峰来解决这一问题。该单峰架构能够借助显式的单峰概率分布,更好地利用底层连续动作空间中的连续性。我们开展了大量实验,结果表明,在具有挑战性的控制任务中,尤其是在 Humanoid 等高复杂度任务中,采用单峰概率分布的离散策略能够为同策略强化学习算法带来显著更快的收敛速度和更高的性能。我们还对策略梯度估计器的方差进行了理论分析,结果表明,我们精心设计的单峰离散策略能够保持较低的方差,并产生稳定的学习过程。

英文摘要:

For on-policy reinforcement learning, discretizing action space for continuous control can easily express multiple modes and is straightforward to optimize. However, without considering the inherent ordering between the discrete atomic actions, the explosion in the number of discrete actions can possess undesired properties and induce a higher variance for the policy gradient estimator. In this paper, we introduce a straightforward architecture that addresses this issue by constraining the discrete policy to be unimodal using Poisson probability distributions. This unimodal architecture can better leverage the continuity in the underlying continuous action space using explicit unimodal probability distributions. We conduct extensive experiments to show that the discrete policy with the unimodal probability distribution provides significantly faster convergence and higher performance for on-policy reinforcement learning algorithms in challenging control tasks, especially in highly complex tasks such as Humanoid. We provide theoretical analysis on the variance of the policy gradient estimator, which suggests that our attentively designed unimodal discrete policy can retain a lower variance and yield a stable learning process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑