arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2201.08355cs.ROcs.LG
收藏

Sim-to-Lab-to-Real:具有屏蔽机制和泛化保证的安全强化学习

Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guarantees

Princeton University(普林斯顿大学)

Kai-Chieh Hsu, Allen Z. Ren, Duy Phuong Nguyen, Anirudha Majumdar, Jaime F. Fisac

AI总结:

本文提出Sim-to-Lab-to-Real安全强化学习框架,通过双策略设置、监督控制屏蔽不安全动作及PAC-Bayes框架提供安全性与性能下界,实现策略在未见环境中的安全泛化。

摘要:

安全性是自主系统的关键组成部分,对于基于学习的策略在现实世界中的应用而言仍是一项挑战。具体而言,使用强化学习训练的策略由于不安全行为,往往无法泛化到新环境中。在本文中,我们提出Sim-to-Lab-to-Real框架,利用具有概率保证的安全感知策略分布来弥合现实差距。为提高安全性,我们采用双策略设置:使用累积任务奖励训练性能策略,并通过基于Hamilton-Jacobi(HJ)可达性分析求解Safety Bellman Equation来训练备份(安全)策略。在Sim-to-Lab迁移中,我们应用监督控制方案在探索期间屏蔽不安全动作;在Lab-to-Real迁移中,我们利用Probably Approximately Correct (PAC)-Bayes框架为未见环境中的策略提供预期性能和安全性的下界。此外,继承自HJ可达性分析,该界限考虑了每个环境中最坏情况安全性的期望。我们在两种具有不同照片级真实度的室内环境中,实证研究了该框架在自我视觉导航中的应用。我们还通过在真实室内空间中使用四足机器人的硬件实验,展示了强大的泛化性能。补充材料请参见 https://sites.google.com/princeton.edu/sim-to-lab-to-real。

英文摘要:

Safety is a critical component of autonomous systems and remains a challenge for learning-based policies to be utilized in the real world. In particular, policies learned using reinforcement learning often fail to generalize to novel environments due to unsafe behavior. In this paper, we propose Sim-to-Lab-to-Real to bridge the reality gap with a probabilistically guaranteed safety-aware policy distribution. To improve safety, we apply a dual policy setup where a performance policy is trained using the cumulative task reward and a backup (safety) policy is trained by solving the Safety Bellman Equation based on Hamilton-Jacobi (HJ) reachability analysis. In Sim-to-Lab transfer, we apply a supervisory control scheme to shield unsafe actions during exploration; in Lab-to-Real transfer, we leverage the Probably Approximately Correct (PAC)-Bayes framework to provide lower bounds on the expected performance and safety of policies in unseen environments. Additionally, inheriting from the HJ reachability analysis, the bound accounts for the expectation over the worst-case safety in each environment. We empirically study the proposed framework for ego-vision navigation in two types of indoor environments with varying degrees of photorealism. We also demonstrate strong generalization performance through hardware experiments in real indoor spaces with a quadrupedal robot. See https://sites.google.com/princeton.edu/sim-to-lab-to-real for supplementary material.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑