从灾难性的动作效果中学习盾牌：永远不要重复同样的错误
Learning a Shield from Catastrophic Action Effects: Never Repeat the Same Mistake
摘要：
在未知环境中运行的智能体在学习过程中必然会犯错误，包括，至少偶尔，一些导致灾难性后果的错误。当人类犯下灾难性的错误时，人们希望他们学会永远不要重复这些错误，比如一个蹒跚学步的孩子触摸了一个热炉子，立即学会永远不要再这样做。在这项工作中，我们考虑了一类新的POMDP，称为POMDP与灾难性的行动（POMDP-CA），其中对状态和行动被标记为灾难性的。在POMDP-CA中行动的Agent没有关于哪些（状态，动作）对是灾难性的先验知识，因此他们在尝试学习任何有意义的策略时肯定会犯错误。相反，他们的目标是最大化回报，同时从不重复错误。作为避免错误重复的第一步，我们利用了一个盾牌的概念，防止代理从特定的状态执行特定的操作。特别是，我们存储灾难性的错误（不安全的状态和动作对），代理在数据库中。然后禁止代理拾取数据库中出现的操作。这种方法在持续学习环境中特别有用，在这种环境中，代理组在同一底层环境中随着时间的推移执行各种任务。在这种情况下，任务不可知的盾牌可以以一种存储任何代理所犯错误的方式构建，这样一旦一个组中的一个代理犯了错误，整个组就学会永远不会重复这个错误。本文介绍了一种变体的PPO算法，利用这个盾牌，称为ShieldPPO，并在受控环境中进行经验评估。结果表明，在一系列设置中，ShieldPPO优于PPO以及安全强化学习文献中的基线方法。
英文摘要：
Agents that operate in an unknown environment are bound to make mistakes while learning, including, at least occasionally, some that lead to catastrophic consequences. When humans make catastrophic mistakes, they are expected to learn never to repeat them, such as a toddler who touches a hot stove and immediately learns never to do so again. In this work we consider a novel class of POMDPs, called POMDP with Catastrophic Actions (POMDP-CA) in which pairs of states and actions are labeled as catastrophic. Agents that act in a POMDP-CA do not have a priori knowledge about which (state, action) pairs are catastrophic, thus they are sure to make mistakes when trying to learn any meaningful policy. Rather, their aim is to maximize reward while never repeating mistakes. As a first step of avoiding mistake repetition, we leverage the concept of a shield which prevents agents from executing specific actions from specific states. In particular, we store catastrophic mistakes (unsafe pairs of states and actions) that agents make in a database. Agents are then forbidden to pick actions that appear in the database. This approach is especially useful in a continual learning setting, where groups of agents perform a variety of tasks over time in the same underlying environment. In this setting, a task-agnostic shield can be constructed in a way that stores mistakes made by any agent, such that once one agent in a group makes a mistake the entire group learns to never repeat that mistake. This paper introduces a variant of the PPO algorithm that utilizes this shield, called ShieldPPO, and empirically evaluates it in a controlled environment. Results indicate that ShieldPPO outperforms PPO, as well as baseline methods from the safe reinforcement learning literature, in a range of settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
