arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.06945cs.LGcs.AI
收藏

带函数逼近的重球动量加速演员-评论家算法

Heavy-Ball Momentum Accelerated Actor-Critic With Function Approximation

Guangdong-Hong Kong-Macao Joint Laboratory for Emotional Intelligence and Pervasive Computing, Shenzhen MSU-BIT University(深圳北理莫斯科大学粤港澳情感智能与普适计算联合实验室);University of Science and Technology Beijing(北京科技大学);Beijing Institute of Technology(北京理工大学)

Yanjie Dong, Haijun Zhang, Gang Wang, Shisheng Cui, Xiping Hu

AI总结:

针对现有演员-评论家算法动量影响研究不足的问题,提出整合重球动量的线性参数化HB-A2C算法,验证其在马尔可夫噪声下的O(ε⁻²)迭代加速能力,揭示学习率与轨迹长度的依赖关系,可平衡初始化与随机近似误差。

摘要:

通过使用参数化价值函数替代蒙特卡洛滚动来进行价值估计,演员-评论家(actor-critic, AC)算法能够降低随机策略梯度的方差,从而提升收敛速度。现有研究主要聚焦于分析马尔可夫噪声下AC算法的收敛速度,而动量对AC算法的影响仍在很大程度上未被探索。在本研究中,我们首先提出了一种基于重球动量的优势演员-评论家(HB-A2C)算法,该算法将重球动量整合到由线性函数参数化的评论家递归过程中。当采样轨迹服从马尔可夫决策过程时,我们定量验证了所提出的HB-A2C算法的加速能力。我们的理论结果表明,对于存在马尔可夫噪声的强化学习任务,所提出的HB-A2C算法以O(ε⁻²)次迭代找到ε近似平稳点。此外,我们还揭示了学习率与采样轨迹长度之间的依赖关系。通过精心选择评论家递归的动量因子,所提出的HB-A2C算法能够平衡初始化和随机近似带来的误差。

英文摘要:

By using an parametric value function to replace the Monte-Carlo rollouts for value estimation, the actor-critic (AC) algorithms can reduce the variance of stochastic policy gradient so that to improve the convergence rate. While existing works mainly focus on analyzing convergence rate of AC algorithms under Markovian noise, the impacts of momentum on AC algorithms remain largely unexplored. In this work, we first propose a heavy-ball momentum based advantage actor-critic (\mbox{HB-A2C}) algorithm by integrating the heavy-ball momentum into the critic recursion that is parameterized by a linear function. When the sample trajectory follows a Markov decision process, we quantitatively certify the acceleration capability of the proposed HB-A2C algorithm. Our theoretical results demonstrate that the proposed HB-A2C finds an $ε$-approximate stationary point with $\oo{ε^{-2}}$ iterations for reinforcement learning tasks with Markovian noise. Moreover, we also reveal the dependence of learning rates on the length of the sample trajectory. By carefully selecting the momentum factor of the critic recursion, the proposed HB-A2C can balance the errors introduced by the initialization and the stoschastic approximation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑