arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2110.10522cs.LGcs.AI
收藏

CIM-PPO:基于刘- correntropy诱导度量的近端策略优化

CIM-PPO:Proximal Policy Optimization with Liu-Correntropy Induced Metric

College of Science, National University of Defense Technology(国防科技大学理学院);School of System Science and Engineering, Sun-Yat-Sen University(中山大学系统科学与工程学院);College of System Engineering, National University of Defense Technology(国防科技大学系统工程学院)

Yunxiao Guo, Han Long, Xiaojun Duan, Kaiyuan Feng, Maochu Li, Xiaying Ma

AI总结:

针对KL散度不对称性误导PPO-KL策略优化的问题,提出基于刘-correntropy诱导度量的PPO-CIM算法,在RKHS中重定义内积,兼具低计算成本与信任域保障,在多数Mujoco任务上表现优于两类经典PPO。

摘要:

作为一种流行的深度强化学习(DRL)算法,近端策略优化(PPO)已在众多复杂任务中展现出卓越的效能。根据替代目标中的惩罚机制,PPO可分为带KL散度的PPO(PPO-KL)和带裁剪的PPO(PPO-Clip)。本文分析了KL散度的不对称性对PPO-KL的影响,并指出当这种不对称性显著时,会误导替代目标的优化方向。为解决该问题,我们将PPO-KL表示为内积形式,并证明KL散度是欧氏空间中的一种Correntropy诱导度量(CIM,即刘-correntropy诱导度量)。随后,我们将PPO-KL扩展到再生核希尔伯特空间(RKHS),利用RKHS重新定义内积,提出了PPO-CIM算法。此外,本文指出PPO-CIM算法在策略梯度计算上成本更低,并证明当核函数满足一定条件时,PPO-CIM可保证新策略处于信任域内。最后,我们基于6个Mujoco连续动作任务设计实验以验证所提算法。实验结果证实了KL散度的不对称性会影响PPO-KL的策略提升,且PPO-CIM在大多数任务中的表现优于PPO-KL和PPO-Clip。

英文摘要:

As a popular Deep Reinforcement Learning (DRL) algorithm, Proximal Policy Optimization (PPO) has demonstrated remarkable efficacy in numerous complex tasks. According to the penalty mechanism in a surrogate, PPO can be classified into PPO with KL divergence (PPO-KL) and PPO with Clip (PPO-Clip). In this paper, we analyze the impact of asymmetry in KL divergence on PPO-KL and highlight that when this asymmetry is pronounced, it will misguide the improvement of the surrogate. To address this issue, we represent the PPO-KL in inner product form and demonstrate that the KL divergence is a Correntropy Induced Metric (CIM) in Euclidean space. Subsequently, we extend the PPO-KL to the Reproducing Kernel Hilbert Space (RKHS), redefine the inner products with RKHS, and propose the PPO-CIM algorithm. Moreover, this paper states that the PPO-CIM algorithm has a lower computation cost in policy gradient and proves that PPO-CIM can guarantee the new policy is within the trust region while the kernel satisfies some conditions. Finally, we design experiments based on six Mujoco continuous-action tasks to validate the proposed algorithm. The experimental results validate that the asymmetry of KL divergence can affect the policy improvement of PPO-KL and show that the PPO-CIM can perform better than both PPO-KL and PPO-Clip in most tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑