arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2007.06680cs.LGcs.ROcs.SYeess.SYmath.OCstat.ML
收藏

基于矩的策略梯度方法

Momentum-Based Policy Gradient Methods

University of Pittsburgh(匹兹堡大学);Simon Fraser University(西蒙弗雷泽大学);JD Finance America Corporation(京东金融美国公司)

Feihu Huang, Shangqian Gao, Jian Pei, Heng Huang

摘要:

针对无模型强化学习,提出了一类基于动量的策略梯度方法,该方法采用自适应学习率,不需要大批量学习.具体来说,我们提出了一种快速的重要性抽样动量为基础的策略梯度(IS-MBPG)方法的基础上,一个新的动量为基础的方差减少技术和重要性采样技术。同时,基于动量方差缩减技术和Hessian辅助技术,提出了一种快速的Hessian辅助动量策略梯度(HA-MBPG)方法。此外,我们证明了IS-MBPG和HA-MBPG方法达到了$O(ε^{-3})$的最佳已知的样本复杂度,用于找到非凹性能函数的$ε$ -稳定点,这只需要一个轨迹在每次迭代。特别地,我们提出了IS-MBPG方法的非自适应版本,即,IS-MBPG*,它也达到了$O(ε^{-3})$的最佳已知样品复杂性,没有任何大批次。在实验中,我们应用四个基准任务来证明我们的算法的有效性。

英文摘要:

In the paper, we propose a class of efficient momentum-based policy gradient methods for the model-free reinforcement learning, which use adaptive learning rates and do not require any large batches. Specifically, we propose a fast important-sampling momentum-based policy gradient (IS-MBPG) method based on a new momentum-based variance reduced technique and the importance sampling technique. We also propose a fast Hessian-aided momentum-based policy gradient (HA-MBPG) method based on the momentum-based variance reduced technique and the Hessian-aided technique. Moreover, we prove that both the IS-MBPG and HA-MBPG methods reach the best known sample complexity of $O(ε^{-3})$ for finding an $ε$-stationary point of the non-concave performance function, which only require one trajectory at each iteration. In particular, we present a non-adaptive version of IS-MBPG method, i.e., IS-MBPG*, which also reaches the best known sample complexity of $O(ε^{-3})$ without any large batches. In the experiments, we apply four benchmark tasks to demonstrate the effectiveness of our algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑