arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2207.13446cs.LGcs.FL
收藏

黑箱环境下强化学习的动态屏蔽

Dynamic Shielding for Reinforcement Learning in Black-Box Environments

Graduate School of Informatics, Kyoto University(京都大学信息学研究科);National Institute of Informatics(国立信息学研究所);National Institute of Advanced Industrial Science and Technology(国立先进工业科学技术研究所);University of Electronic Science and Technology of China(电子科技大学);The Graduate University for Advanced Studies(综合研究大学院大学)

Masaki Waga, Ezequiel Castellano, Sasinee Pruekprasert, Stefan Klikovits, Toru Takisaka, Ichiro Hasuo

摘要:

由于在学习过程中缺乏安全保证,在信息物理系统中使用强化学习(RL)具有挑战性。虽然已经有各种各样的建议,以减少在学习过程中不希望的行为,这些技术大多需要事先的系统知识,其适用性是有限的。本文旨在减少学习过程中的不良行为,而不需要任何先验的系统知识。我们提出了动态屏蔽:基于模型的安全RL技术的扩展,称为使用自动机学习的屏蔽。动态屏蔽技术使用RPNI算法的变体与RL并行构建近似系统模型,并抑制由于从学习模型构建的屏蔽而引起的不期望的探索。通过这种组合,潜在的不安全行为可以在代理经历它们之前预见到。实验表明,我们的动态屏蔽显着减少了训练过程中的不良事件的数量。

英文摘要:

It is challenging to use reinforcement learning (RL) in cyber-physical systems due to the lack of safety guarantees during learning. Although there have been various proposals to reduce undesired behaviors during learning, most of these techniques require prior system knowledge, and their applicability is limited. This paper aims to reduce undesired behaviors during learning without requiring any prior system knowledge. We propose dynamic shielding: an extension of a model-based safe RL technique called shielding using automata learning. The dynamic shielding technique constructs an approximate system model in parallel with RL using a variant of the RPNI algorithm and suppresses undesired explorations due to the shield constructed from the learned model. Through this combination, potentially unsafe actions can be foreseen before the agent experiences them. Experiments show that our dynamic shield significantly decreases the number of undesired events during training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑