arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节（Veterans Day），12 日无更新
arXiv 2004.05002cs.AIcs.LG
收藏

用于深度Q学习的自我惩罚和奖励回填

Self Punishment and Reward Backfill for Deep Q-Learning

Microsoft Development Center Norway（微软挪威开发中心）；Turnto Health；Kavli Institute for Systems Neuroscience, Norwegian University of Science and Technology（挪威科技大学卡夫利系统神经科学研究所）；Queensland Brain Institute, University of Queensland（昆士兰大学昆士兰脑研究所）

Mohammad Reza Bonyadi, Rui Wang, Maryam Ziaei

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

摘要：

强化学习代理通过鼓励最大化其总奖励的行为来学习，通常由环境提供。然而，在许多环境中，奖励是在一系列行动之后提供的，而不是在每个单一行动之后提供的，这导致代理在这些行动是否有效方面存在不确定性，这个问题被称为信用分配问题。在本文中，我们提出了两个策略的启发行为心理学，使代理本质上估计更多的信息奖励值的行动没有奖励。第一种策略，称为自我惩罚（SP），阻止代理犯错误，导致不受欢迎的终端状态。第二种策略称为奖励回填（RB），在两个奖励动作之间反向传播奖励。我们证明，在一定的假设下，无论使用的强化学习算法，这两个策略保持在空间中的所有可能的政策在其总回报的政策顺序，并通过扩展，保持最优的政策。因此，我们提出的策略与任何通过经验学习值或动作-价值函数的强化学习算法相结合。我们将这两种策略整合到三种流行的深度强化学习方法中，并在30款Atari游戏上评估了结果。参数调整后，我们的研究结果表明，所提出的策略提高了测试方法在超过65%的测试游戏的性能提高了25倍以上。

英文摘要：

Reinforcement learning agents learn by encouraging behaviours which maximize their total reward, usually provided by the environment. In many environments, however, the reward is provided after a series of actions rather than each single action, leading the agent to experience ambiguity in terms of whether those actions are effective, an issue known as the credit assignment problem. In this paper, we propose two strategies inspired by behavioural psychology to enable the agent to intrinsically estimate more informative reward values for actions with no reward. The first strategy, called self-punishment (SP), discourages the agent from making mistakes that lead to undesirable terminal states. The second strategy, called the rewards backfill (RB), backpropagates the rewards between two rewarded actions. We prove that, under certain assumptions and regardless of the reinforcement learning algorithm used, these two strategies maintain the order of policies in the space of all possible policies in terms of their total reward, and, by extension, maintain the optimal policy. Hence, our proposed strategies integrate with any reinforcement learning algorithm that learns a value or action-value function through experience. We incorporated these two strategies into three popular deep reinforcement learning approaches and evaluated the results on thirty Atari games. After parameter tuning, our results indicate that the proposed strategies improve the tested methods in over 65 percent of tested games by up to over 25 times performance improvement.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑