策略梯度衍生工具的实证分析
An Empirical Analysis of Measure-Valued Derivatives for Policy Gradients
摘要：
由于更好的策略梯度技术的不断发展，机器人的强化学习方法越来越成功。精确（低方差）和准确（低偏差）的梯度估计器对于面对日益复杂的任务至关重要。传统的策略梯度算法使用似然比技巧，这是众所周知的无偏，但高方差估计。更现代的方法利用重新参数化技巧，它提供了较低的方差梯度估计，但需要可微的价值函数逼近。在这项工作中，我们研究了一种不同类型的随机梯度估计：测度值导数。该估计是无偏的，具有低方差，并可用于可微和不可微函数逼近。我们经验评估此估计在演员-评论家策略梯度设置，并表明它可以达到可比的性能与方法的基础上的似然比或重新参数化技巧，无论是在低维和高维的动作空间。
英文摘要：
Reinforcement learning methods for robotics are increasingly successful due to the constant development of better policy gradient techniques. A precise (low variance) and accurate (low bias) gradient estimator is crucial to face increasingly complex tasks. Traditional policy gradient algorithms use the likelihood-ratio trick, which is known to produce unbiased but high variance estimates. More modern approaches exploit the reparametrization trick, which gives lower variance gradient estimates but requires differentiable value function approximators. In this work, we study a different type of stochastic gradient estimator: the Measure-Valued Derivative. This estimator is unbiased, has low variance, and can be used with differentiable and non-differentiable function approximators. We empirically evaluate this estimator in the actor-critic policy gradient setting and show that it can reach comparable performance with methods based on the likelihood-ratio or reparametrization tricks, both in low and high-dimensional action spaces.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
