安全性是自主系统的关键组成部分,对于基于学习的策略在现实世界中的应用而言仍是一项挑战。具体而言,使用强化学习训练的策略由于不安全行为,往往无法泛化到新环境中。在本文中,我们提出Sim-to-Lab-to-Real框架,利用具有概率保证的安全感知策略分布来弥合现实差距。为提高安全性,我们采用双策略设置:使用累积任务奖励训练性能策略,并通过基于Hamilton-Jacobi(HJ)可达性分析求解Safety Bellman Equation来训练备份(安全)策略。在Sim-to-Lab迁移中,我们应用监督控制方案在探索期间屏蔽不安全动作;在Lab-to-Real迁移中,我们利用Probably Approximately Correct (PAC)-Bayes框架为未见环境中的策略提供预期性能和安全性的下界。此外,继承自HJ可达性分析,该界限考虑了每个环境中最坏情况安全性的期望。我们在两种具有不同照片级真实度的室内环境中,实证研究了该框架在自我视觉导航中的应用。我们还通过在真实室内空间中使用四足机器人的硬件实验,展示了强大的泛化性能。补充材料请参见 https://sites.google.com/princeton.edu/sim-to-lab-to-real。
英文摘要:
Safety is a critical component of autonomous systems and remains a challenge for learning-based policies to be utilized in the real world. In particular, policies learned using reinforcement learning often fail to generalize to novel environments due to unsafe behavior. In this paper, we propose Sim-to-Lab-to-Real to bridge the reality gap with a probabilistically guaranteed safety-aware policy distribution. To improve safety, we apply a dual policy setup where a performance policy is trained using the cumulative task reward and a backup (safety) policy is trained by solving the Safety Bellman Equation based on Hamilton-Jacobi (HJ) reachability analysis. In Sim-to-Lab transfer, we apply a supervisory control scheme to shield unsafe actions during exploration; in Lab-to-Real transfer, we leverage the Probably Approximately Correct (PAC)-Bayes framework to provide lower bounds on the expected performance and safety of policies in unseen environments. Additionally, inheriting from the HJ reachability analysis, the bound accounts for the expectation over the worst-case safety in each environment. We empirically study the proposed framework for ego-vision navigation in two types of indoor environments with varying degrees of photorealism. We also demonstrate strong generalization performance through hardware experiments in real indoor spaces with a quadrupedal robot. See https://sites.google.com/princeton.edu/sim-to-lab-to-real for supplementary material.
Robust Model Predictive Shielding for Safe Reinforcement Learning with Stochastic Dynamics
GRASP Lab, University of Pennsylvania, USA(宾夕法尼亚大学GRASP实验室);Department of Computer and Information Science, University of Pennsylvania, USA(宾夕法尼亚大学计算机与信息科学系(注:根据机构粒度要求,这里提取到大学级别,即“宾夕法尼亚大学”,但原文明确写了具体部门,所以保留原文部门名+大学的组合))
Department of Computer Science, University of Manchester(曼彻斯特大学计算机科学系);Faculty of Computer Science & DSME, RWTH Aachen University(亚琛工业大学计算机科学与DSME学院);Department of Computer Science, University of Oxford(牛津大学计算机科学系);Department of Computing, Imperial College London(伦敦帝国理工学院计算系)
Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
Radboud University Nijmegen The Netherlands;Ruhr University\,\&\,Radboud University Bochum Germany;Radboud University;Ruhr University\,\&\,Radboud University
Model-based Dynamic Shielding for Safe and Efficient Multi-Agent Reinforcement Learning
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳市人工智能与机器人研究院);Carnegie Mellon University(卡内基梅隆大学)
Verification-Guided Shielding for Deep Reinforcement Learning
University of California, Irvine(加利福尼亚大学欧文分校);The Hebrew University of Jerusalem(耶路撒冷希伯来大学);IMDEA Software Institute(IMDEA软件研究所);Universidad Politécnica de Madrid(马德里理工大学)
Dynamic Shielding for Reinforcement Learning in Black-Box Environments
Graduate School of Informatics, Kyoto University(京都大学信息学研究科);National Institute of Informatics(国立信息学研究所);National Institute of Advanced Industrial Science and Technology(国立先进工业科学技术研究所);University of Electronic Science and Technology of China(电子科技大学);The Graduate University for Advanced Studies(综合研究大学院大学)
SHARP: Shielding-Aware Robust Planning for Safe and Efficient Human-Robot Interaction
Princeton University(普林斯顿大学)
在人机交互(HRI)设置中共同实现安全性和效率是一个具有挑战性的问题,因为机器人的规划目标可能与人类自己的意图和期望不一致。最近的方法,确保安全的机器人操作在不确定的环境中,通过监督控制方案,有时被称为“屏蔽”,它覆盖机器人的标称计划与安全回退策略时,一个安全的关键事件是迫在眉睫的。这些反应性的“最后手段”策略(通常以积极的紧急机动的形式)专注于保护安全而不考虑效率;当名义计划者不知道可能的安全覆盖时,屏蔽可以比必要的更频繁地激活,导致性能下降。在这项工作中,我们提出了一个新的屏蔽为基础的规划方法,允许机器人规划效率明确占未来可能的屏蔽事件。利用贝叶斯人体运动预测的最新工作,由此产生的机器人政策积极平衡标称性能与低概率人类行为引发的高成本紧急演习的风险。我们形式化屏蔽感知鲁棒规划(SHARP)作为一个随机最优控制问题,并提出了一个计算效率高的框架,在运行时找到易于处理的近似解。我们的方法在模拟驾驶示例中的性能优于屏蔽不可知的运动规划基线(配备相同的人类意图推理方案),这些示例中的人类轨迹来自最近发布的Waymo Open Motion Dataset。
Predictive Safety Shield for Dyna-Q Reinforcement Learning
Departement U2IS, ENSTA Paris, Institut Polytechnique de Paris(ENSTA巴黎大学U2IS部门,巴黎理工学院);Department of Electrical Engineering and Computer Science, University of California, Berkeley, USA(加州大学伯克利分校电气工程与计算机科学系)
Realizable Continuous-Space Shields for Safe Reinforcement Learning
University of California, Irvine(加利福尼亚大学欧文分校);IMDEA Software Institute(IMDEA软件研究所);Universidad Politécnica de Madrid(马德里理工大学);Universitat Politècnica de Catalunya(加泰罗尼亚理工大学)
ShieldNN: A Provably Safe NN Filter for Unsafe NN Controllers
Electrical Engineering and Computer Science, University of California, Irvine(加州大学欧文分校电气工程与计算机科学系);Electrical and Computer Engineering, University of Virginia(弗吉尼亚大学电气与计算机工程系);Mechanical Engineering, Iowa State University(爱荷华州立大学机械工程系)
Shield-Loco: Shielding Locomotion Policies with Predictive Safety Filtering
Robert Bosch Center for Cyber Physical Systems(罗伯特·博世网络物理系统中心);Indian Institute of Science(印度科学研究院);Munich Institute of Robotics and Machine Intelligence (MIRMI)(慕尼黑机器人与机器智能研究所(MIRMI));Technical University of Munich(慕尼黑技术大学);Department of Computer Science & Automation(计算机科学与自动化部门);Department of Mechanical Engineering(机械工程系);Carnegie Mellon University(卡内基梅隆大学);Institute for Advanced Study(高级研究 institute)
IMDEA Software Institute(IMDEA软件研究所);Cornell University(康奈尔大学);University of California, Irvine(加州大学欧文分校);The Hebrew University of Jerusalem(耶路撒冷希伯来大学)
Safe Multi-Agent Reinforcement Learning through Decentralized Multiple Control Barrier Functions
School of Mechanical Engineering and Automation, Harbin Institute of Technology Shenzhen(哈尔滨工业大学(深圳)机械工程与自动化学院);Department of Aerospace Engineering and Engineering Mechanics, University of Cincinnati(辛辛那提大学航空航天工程与工程力学系)
REASAN: Learning Reactive Safe Navigation for Legged Robots
Bernoulli Institute for Mathematics, Computer Science and Artificial Intelligence, University of Groningen, The Netherlands(格罗宁根大学伯努利数学、计算机科学与人工智能研究所);Department of Electrical Engineering, Linköping University, Sweden(利厄普大学电气工程系);Engineering and Technology Institute Groningen, University of Groningen, The Netherlands(格罗宁根大学格罗宁根工程与技术研究所)
LabEvolver: Training-Free Experience Evolution for Safe and Grounded Wet-Lab Agents
School of AI for Science, Peking University(北京大学AI科学学院);School of Electronic and Computer Engineering, Peking University(北京大学电子与计算机工程学院);School of Computer Science, Peking University(北京大学计算机学院)