arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.04440cs.LGcs.AImath.OCstat.ML
收藏

使用残差方差学习深度策略梯度中的价值函数

Learning Value Functions in Deep Policy Gradients using Residual Variance

Inria(法国国家信息与自动化研究所);Univ. Lille(里尔大学);CNRS(法国国家科学研究中心)

Yannis Flet-Berliac, Reda Ouhamma, Odalric-Ambrym Maillard, Philippe Preux

摘要:

策略梯度算法已被证明是成功的,在不同的决策和控制任务。然而,这些方法遭受高样本复杂性和不稳定性问题。在本文中,我们通过提供一种不同的方法来解决这些挑战,在演员-评论家框架中训练评论家。我们的工作建立在最近的研究表明,传统的演员-评论家算法不成功拟合真正的价值函数,需要确定一个更好的目标的批评。在我们的方法中,评论家使用一个新的状态值(分别为)。状态-动作-值)函数近似,其学习状态(分别为状态-动作对)相对于它们的平均值,而不是如传统的行动者-批评者中的绝对值。我们证明了新的梯度估计的理论一致性,并观察到戏剧性的经验改进,在各种连续的控制任务和算法。此外,我们在稀疏奖励任务中验证了我们的方法,在那里我们提供了实验证据和理论见解。

英文摘要:

Policy gradient algorithms have proven to be successful in diverse decision making and control tasks. However, these methods suffer from high sample complexity and instability issues. In this paper, we address these challenges by providing a different approach for training the critic in the actor-critic framework. Our work builds on recent studies indicating that traditional actor-critic algorithms do not succeed in fitting the true value function, calling for the need to identify a better objective for the critic. In our method, the critic uses a new state-value (resp. state-action-value) function approximation that learns the value of the states (resp. state-action pairs) relative to their mean value rather than the absolute value as in conventional actor-critic. We prove the theoretical consistency of the new gradient estimator and observe dramatic empirical improvement across a variety of continuous control tasks and algorithms. Furthermore, we validate our method in tasks with sparse rewards, where we provide experimental evidence and theoretical insights.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑