arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2211.10550cs.LGcs.AI
收藏

通过学习外价值函数的去偏亚梯度强化学习

Debiasing Meta-Gradient Reinforcement Learning by Learning the Outer Value Function

InstaDeep

Clément Bonnet, Laurence Midgley, Alexandre Laterre

摘要:

元梯度强化学习(RL)允许代理在训练期间以在线方式自我调整其超参数。在本文中,我们确定当前元梯度RL方法的元梯度中的偏差。这种偏差来自于使用批评者,该批评者是使用元学习的折扣因子训练的,用于外部目标中的优势估计,这需要不同的折扣因子。由于元学习折扣因子通常低于外部目标中使用的折扣因子,因此产生的偏差可能导致元梯度偏向短视策略。我们提出了一个简单的解决方案,这个问题:我们消除这种偏见,使用替代,\emph{outer}价值函数的估计外损失。为了获得这个外部价值函数我们在评论家网络中添加第二个头,并使用外部损失折扣因子与经典评论家一起对其进行训练。在一个说明性的玩具问题上,我们证明了偏置会导致当前元梯度RL方法的灾难性失败,并表明我们提出的解决方案修复了它。然后,我们将我们的方法应用于更复杂的环境,并证明修复元梯度偏置可以显着提高性能。

英文摘要:

Meta-gradient Reinforcement Learning (RL) allows agents to self-tune their hyper-parameters in an online fashion during training. In this paper, we identify a bias in the meta-gradient of current meta-gradient RL approaches. This bias comes from using the critic that is trained using the meta-learned discount factor for the advantage estimation in the outer objective which requires a different discount factor. Because the meta-learned discount factor is typically lower than the one used in the outer objective, the resulting bias can cause the meta-gradient to favor myopic policies. We propose a simple solution to this issue: we eliminate this bias by using an alternative, \emph{outer} value function in the estimation of the outer loss. To obtain this outer value function we add a second head to the critic network and train it alongside the classic critic, using the outer loss discount factor. On an illustrative toy problem, we show that the bias can cause catastrophic failure of current meta-gradient RL approaches, and show that our proposed solution fixes it. We then apply our method to a more complex environment and demonstrate that fixing the meta-gradient bias can significantly improve performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑