arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.15650cs.LGcs.AI
收藏

面向多约束安全关键应用的具有目标抑制的均匀安全强化学习

Uniformly Safe RL with Objective Suppression for Multi-Constraint Safety-Critical Applications

University of Toronto(多伦多大学);Vector Institute(矢量研究所);Nuro Inc.(Nuro公司);Georgia Tech(佐治亚理工学院)

Zihan Zhou, Jonathan Booher, Khashayar Rohanimanesh, Wei Liu, Aleksandr Petiushko, Animesh Garg

AI总结:

针对CMDP期望约束在长尾状态留下危险隐患的问题,提出UCMDP模型与目标抑制方法,在多约束安全关键领域显著减少约束违规并保持任务奖励。

摘要:

安全强化学习任务尽管在现实世界中非常普遍,却是一个极具挑战性的领域。广泛采用的 CMDP 模型在期望层面约束风险,这为长尾状态中的危险行为留下了空间。在安全关键领域,此类行为可能导致灾难性后果。为了解决这个问题,我们首先使用一个更强的均匀约束马尔可夫决策过程(UCMDP)模型来描述该问题,在该模型中我们对所有可达状态施加约束;然后我们提出了目标抑制(Objective Suppression),这是一种根据安全评论家自适应地抑制任务奖励最大化目标的新颖方法,作为 UCMDP 拉格朗日对偶的解。我们在两个多约束安全领域中对目标抑制进行了基准测试,包括一个任何错误行为都可能导致灾难性后果的自动驾驶领域。在驾驶领域,我们在开源和专有数据上进行评估,并评估向真实自动驾驶车队的迁移。实验表明,我们提出的方法与现有的安全强化学习算法相结合时,能够匹配基线所达到的任务奖励,同时显著减少约束违规。

英文摘要:

Safe reinforcement learning tasks are a challenging domain despite being very common in the real world. The widely adopted CMDP model constrains the risks in expectation, which makes room for dangerous behaviors in long-tail states. In safety-critical domains, such behaviors could lead to disastrous outcomes. To address this issue, we first describe the problem with a stronger Uniformly Constrained MDP (UCMDP) model where we impose constraints on all reachable states; we then propose Objective Suppression, a novel method that adaptively suppresses the task reward maximizing objectives according to a safety critic, as a solution to the Lagrangian dual of a UCMDP. We benchmark Objective Suppression in two multi-constraint safety domains, including an autonomous driving domain where any incorrect behavior can lead to disastrous consequences. On the driving domain, we evaluate on open source and proprietary data and evaluate transfer to a real autonomous fleet. Empirically, we demonstrate that our proposed method, when combined with existing safe RL algorithms, can match the task reward achieved by baselines with significantly fewer constraint violations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑