arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.10145cs.LGcs.AI
收藏

利用价值函数搜索提高深度策略梯度

Improving Deep Policy Gradients with Value Function Search

Khoury College of Computer Sciences(库里计算机学院);Northeastern University(东北大学)

Enrico Marchesini, Christopher Amato

摘要:

深度策略梯度(PG)算法采用价值网络来驱动参数化策略的学习并减少梯度估计的方差。然而,价值函数近似会陷入局部最优,难以拟合实际收益,从而限制了方差降低的功效,并导致策略的次优性能。本文重点介绍了改进值近似和分析对Deep PG基元的影响,例如值预测,方差减少以及梯度估计与真实梯度的相关性。为此,我们引入了一个价值函数搜索,采用人口的扰动值网络搜索一个更好的近似。我们的框架不需要额外的环境交互,梯度计算,或合奏,提供了一个计算成本低廉的方法来增强监督学习任务上的价值网络训练。至关重要的是,我们表明,改善深度PG原语的结果提高了样本效率和政策,使用共同的连续控制基准域的回报更高。

英文摘要:

Deep Policy Gradient (PG) algorithms employ value networks to drive the learning of parameterized policies and reduce the variance of the gradient estimates. However, value function approximation gets stuck in local optima and struggles to fit the actual return, limiting the variance reduction efficacy and leading policies to sub-optimal performance. This paper focuses on improving value approximation and analyzing the effects on Deep PG primitives such as value prediction, variance reduction, and correlation of gradient estimates with the true gradient. To this end, we introduce a Value Function Search that employs a population of perturbed value networks to search for a better approximation. Our framework does not require additional environment interactions, gradient computations, or ensembles, providing a computationally inexpensive approach to enhance the supervised learning task on which value networks train. Crucially, we show that improving Deep PG primitives results in improved sample efficiency and policies with higher returns using common continuous control benchmark domains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑