In this paper, we present a methodology to deploy the deterministic policy gradient method, using actor-critic techniques, when the optimal policy is approximated using a parametric optimization problem, where safety is enforced via hard constraints. For continuous input space, imposing safety restrictions on the exploration needed to deploying the deterministic policy gradient method poses some technical difficulties, which we address here. We will investigate in particular policy approximations based on robust Nonlinear Model Predictive Control (NMPC), where safety can be treated explicitly. For the sake of brevity, we will detail the construction of the safe scheme in the robust linear MPC context only. The extension to the nonlinear case is possible but more complex. We will additionally present a technique to maintain the system safety throughout the learning process in the context of robust linear MPC. This paper has a companion paper treating the stochastic policy gradient case.
Tractable safety-ensuring algorithms for cyber-physical systems are important in critical applications. Approaches based on Control Barrier Functions assume continuous enforcement, which is not possible in an online fashion. This paper presents two tractable algorithms to ensure forward invariance of discrete-time controlled cyber-physical systems. Both approaches are based on Control Barrier Functions to provide strict mathematical safety guarantees. The first algorithm exploits Lipschitz continuity and formulates the safety condition as a robust program which is subsequently relaxed to a set…
Reinforcement Learning (RL) has proven a stunning ability to learn optimal policies from data without any prior knowledge on the process. The main drawback of RL is that it is typically very difficult to guarantee stability and safety. On the other hand, Nonlinear Model Predictive Control (NMPC) is an advanced model-based control technique which does guarantee safety and stability, but only yields optimality for the nominal model. Therefore, it has been recently proposed to use NMPC as a function approximator within RL. While the ability of this approach to yield good performance has been demo…
A Step-by-step Guide on Nonlinear Model Predictive Control for Safe Mobile Robot Navigation
department of Cognitive Robotics, Delft University of Technology（认知机器人系，代尔夫特理工大学）；Institute for Dynamic Systems and Control, ETH Zürich（动态系统与控制研究所，苏黎世联邦理工学院）
Towards Safe Learning-Based Non-Linear Model Predictive Control through Recurrent Neural Network Modeling
Institute of Computer Engineering, TU Wien（计算机工程研究所，维也纳技术大学）；AIT Austrian Institute of Technology（奥地利技术研究所）；Institute of Mechanics and Mechatronics, TU Wien（机械与机电系统研究所，维也纳技术大学）
This paper extends the gap-based navigation technique in Potential Gap by guaranteeing safety for nonholonomic robots for all tiers of the local planner hierarchy, so called Safer Gap. The first tier generates a Bezier-based collision-free path through gaps. A subset of navigable free-space from the robot through a gap, called the keyhole, is defined to be the union of the largest collision-free disc centered on the robot and a trapezoidal region directed through the gap. It is encoded by a shallow neural network zeroing barrier function (ZBF). Nonlinear model predictive control (NMPC), with K…
Converse Barrier Certificates for Finite-time Safety Verification of Continuous-time Perturbed Deterministic Systems
Key Lab. of System Software (Chinese Academy of Sciences), Institute of Software, CAS（中国科学院软件研究所系统软件重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；School of Mathematics and Physics, University of Science and Technology Beijing（北京科技大学数理学院）
GRASP Lab, University of Pennsylvania, USA（宾夕法尼亚大学GRASP实验室）；Department of Computer and Information Science, University of Pennsylvania, USA（宾夕法尼亚大学计算机与信息科学系（注：根据机构粒度要求，这里提取到大学级别，即“宾夕法尼亚大学”，但原文明确写了具体部门，所以保留原文部门名+大学的组合））
This paper proposes a framework for safe reinforcement learning that can handle stochastic nonlinear dynamical systems. We focus on the setting where the nominal dynamics are known, and are subject to additive stochastic disturbances with known distribution. Our goal is to ensure the safety of a control policy trained using reinforcement learning, e.g., in a simulated environment. We build on the idea of model predictive shielding (MPS), where a backup controller is used to override the learned policy as needed to ensure safety. The key challenge is how to compute a backup policy in the contex…
DENSO Automotive Deutschland GmbH（电装德国汽车有限公司）；Carl von Ossietzky University（卡尔·冯·奥西茨基大学）
Ensuring safe behavior for automated vehicles in unregulated traffic areas poses a complex challenge for the industry. It is an open problem to provide scalable and certifiable solutions to this challenge. We derive a trajectory planner based on model predictive control which interoperates with a monitoring system for pedestrian safety based on cellular automata. The combined planner-monitor system is demonstrated on the example of a narrow indoor parking environment. The system features deterministic behavior, mitigating the immanent risk of black boxes and offering full certifiability. By us…
We propose Kernel Predictive Control (KPC), a learning-based predictive control strategy that enjoys deterministic guarantees of safety. Noise-corrupted samples of the unknown system dynamics are used to learn several models through the formalism of non-parametric kernel regression. By treating each prediction step individually, we dispense with the need of propagating sets through highly non-linear maps, a procedure that often involves multiple conservative approximation steps. Finite-sample error bounds are then used to enforce state-feasibility by employing an efficient robust formulation. …
In this paper, we propose a construction scheme for a Safe-visor architecture for sandboxing unverified controllers, e.g., artificial intelligence-based (a.k.a. AI-based) controllers, in two-players non-cooperative stochastic games. Concretely, we leverage abstraction-based approaches to construct a supervisor that checks and decides whether or not to accept the inputs provided by the unverified controller, and a safety advisor that provides fallback control inputs to ensure safety whenever the unverified controller is rejected. Moreover, by leveraging an ($ε,δ$)-approximate probabilistic rela…
Safe control methods are often intended to behave safely even in worst-case human uncertainties. However, humans may exploit such safety-first systems, which results in greater risk for everyone. Despite their significance, no prior work has investigated and accounted for such factors in safe control. In this paper, we leverage an interaction-based payoff structure from game theory to model humans' short-sighted, self-seeking behaviors and how humans change their strategies toward machines based on prior experience. We integrate such strategic human behaviors into a safe control architecture. …
Safety in terms of collision avoidance for multi-robot systems is a difficult challenge under uncertainty, non-determinism and lack of complete information. This paper aims to propose a collision avoidance method that accounts for both measurement uncertainty and motion uncertainty. In particular, we propose Probabilistic Safety Barrier Certificates (PrSBC) using Control Barrier Functions to define the space of admissible control actions that are probabilistically safe with formally provable theoretical guarantee. By formulating the chance constrained safety set into deterministic control cons…
This paper proposes a safe reinforcement learning algorithm for generation bidding decisions and unit maintenance scheduling in a competitive electricity market environment. In this problem, each unit aims to find a bidding strategy that maximizes its revenue while concurrently retaining its reliability by scheduling preventive maintenance. The maintenance scheduling provides some safety constraints which should be satisfied at all times. Satisfying the critical safety and reliability constraints while the generation units have an incomplete information of each others' bidding strategy is a ch…
