This paper proposes an on-policy reinforcement learning (RL) control algorithm that solves the optimal regulation problem for a class of uncertain continuous-time nonlinear systems under user-defined state constraints. We formulate the safe RL problem as the minimization of the Hamiltonian subject to a constraint on the time-derivative of a barrier Lyapunov function (BLF). We subsequently use the analytical solution of the optimization problem to modify the Actor-Critic-Identifier architecture to learn the optimal control policy safely. The proposed method does not require the presence of external backup controllers, and the RL policy ensures safety for the entire duration. The efficacy of the proposed controller is demonstrated on a class of Euler-Lagrange systems.
Uncertainty Estimators for Robust Backup Control Barrier Functions
California Institute of Technology（加州理工学院）；Illinois Institute of Technology（伊利诺伊理工大学）；Delft University of Technology（代尔夫特理工大学）；Georgia Institute of Technology（佐治亚理工学院）；Cleveland State University（克利夫兰州立大学）；Texas A&M University（德克萨斯农工大学）；The Aerospace Corporation（美国航空航天公司）
The weight, space, and power limitations of small aerial vehicles often prevent the application of modern control techniques without significant model simplifications. Moreover, high-speed agile behavior, such as that exhibited in drone racing, make these simplified models too unreliable for safety-critical control. In this work, we introduce the concept of time-varying backup controllers (TBCs): user-specified maneuvers combined with backup controllers that generate reference trajectories which guarantee the safety of nonlinear systems. TBCs reduce conservatism when compared to traditional ba…
This article proposes a new safety concept: backup plan safety. The backup plan safety is defined as the ability to complete one of the alternative missions in the case of primary mission abortion. To incorporate this new safety concept in control problems, we formulate a feasibility maximization problem that adopts additional (virtual) input horizons toward the alternative missions on top of the input horizon toward the primary mission. Cost functions for the primary and alternative missions construct multiple objectives, and multi-horizon inputs evaluate them. To address the feasibility maxi…
In large-scale networks of uncertain dynamical systems, where communication is limited and there is a strong interaction among subsystems, learning local models and control policies offers great potential for designing high-performance controllers. At the same time, the lack of safety guarantees, here considered in the form of constraint satisfaction, prevents the use of data-driven techniques to safety-critical distributed systems. This paper presents a safety framework that guarantees constraint satisfaction for uncertain distributed systems while learning. The framework considers linear sys…
This paper proposes a Robust Safe Control Architecture (RSCA) for safe-decision making. The system to be controlled is a vehicle in the presence of bounded disturbances. The RSCA consists of two parts: a Supervisor MPC and a Controller MPC. Both the Supervisor and the Controller are tube MPCs (TMPCs). The Supervisor MPC provides a safety certificate for an operating controller and a backup control input in every step. After an unsafe action by the operating controller is predicted, the Controller MPC takes over the system. In this paper, a method for the computation of a terminal set is propos…
Automated vehicles require efficient and safe planning to maneuver in uncertain environments. Largely this uncertainty is caused by other traffic participants, e.g., surrounding vehicles. Future motion of surrounding vehicles is often difficult to predict. Whereas robust control approaches achieve safe, yet conservative motion planning for automated vehicles, Stochastic Model Predictive Control (SMPC) provides efficient planning in the presence of uncertainty. Probabilistic constraints are applied to ensure that the maximal risk remains below a predefined level. However, safety cannot be ensur…
GRASP Lab, University of Pennsylvania, USA（宾夕法尼亚大学GRASP实验室）；Department of Computer and Information Science, University of Pennsylvania, USA（宾夕法尼亚大学计算机与信息科学系（注：根据机构粒度要求，这里提取到大学级别，即“宾夕法尼亚大学”，但原文明确写了具体部门，所以保留原文部门名+大学的组合））
This paper proposes a framework for safe reinforcement learning that can handle stochastic nonlinear dynamical systems. We focus on the setting where the nominal dynamics are known, and are subject to additive stochastic disturbances with known distribution. Our goal is to ensure the safety of a control policy trained using reinforcement learning, e.g., in a simulated environment. We build on the idea of model predictive shielding (MPS), where a backup controller is used to override the learned policy as needed to ensure safety. The key challenge is how to compute a backup policy in the contex…
Technical University of Munich（慕尼黑工业大学）；ETH Zurich（苏黎世联邦理工学院）
Ensuring safety is a crucial challenge when deploying reinforcement learning (RL) to real-world systems. We develop confidence-based safety filters, a control-theoretic approach for certifying state safety constraints for nominal policies learned via standard RL techniques, based on probabilistic dynamics models. Our approach is based on a reformulation of state constraints in terms of cost functions, reducing safety verification to a standard RL task. By exploiting the concept of hallucinating inputs, we extend this formulation to determine a "backup" policy that is safe for the unknown syste…
In this paper, a photovoltaic (PV) reconfigurable grid-tied inverter (RGTI) scheme is proposed. Unlike a conventional GTI that ceases operation during a power outage, the RGTI is designed to act as a regular GTI in the on-grid mode but it is reconfigured to function as a DC-DC charge-controller that continues operation during a grid outage. During this period, the RGTI is tied to the battery-bank of an external UPS based backup power system to augment it with solar power. Such an operation in off-grid mode without employing communication with the UPS is challenging, as the control of RGTI must…
Safety and performance are often two competing objectives in sequential decision-making problems. Existing performant controllers, such as controllers derived from reinforcement learning algorithms, often fall short of safety guarantees. On the contrary, controllers that guarantee safety, such as those derived from classical control theory, require restrictive assumptions and are often conservative in performance. Our goal is to blend a performant and a safe controller to generate a single controller that is safer than the performant and accumulates higher rewards than the safe controller. To …
In this paper, we propose a construction scheme for a Safe-visor architecture for sandboxing unverified controllers, e.g., artificial intelligence-based (a.k.a. AI-based) controllers, in two-players non-cooperative stochastic games. Concretely, we leverage abstraction-based approaches to construct a supervisor that checks and decides whether or not to accept the inputs provided by the unverified controller, and a safety advisor that provides fallback control inputs to ensure safety whenever the unverified controller is rejected. Moreover, by leveraging an ($ε,δ$)-approximate probabilistic rela…
High performance but unverified controllers, e.g., artificial intelligence-based (a.k.a. AI-based) controllers, are widely employed in cyber-physical systems (CPSs) to accomplish complex control missions. However, guaranteeing the safety and reliability of CPSs with this kind of controllers is currently very challenging, which is of vital importance in many real-life safety-critical applications. To cope with this difficulty, we propose in this work a Safe-visor architecture for sandboxing unverified controllers in CPSs operating in noisy environments (a.k.a. stochastic CPSs). The proposed arc…
We introduce a novel learning-based approach to synthesize safe and robust controllers for autonomous Cyber-Physical Systems and, at the same time, to generate challenging tests. This procedure combines formal methods for model verification with Generative Adversarial Networks. The method learns two Neural Networks: the first one aims at generating troubling scenarios for the controller, while the second one aims at enforcing the safety constraints. We test the proposed method on a variety of case studies.
To bring complex systems into real world environments in a safe manner, they will have to be robust to uncertainties - both in the environment and the system. This paper investigates the safety of control systems under input disturbances, wherein the disturbances can capture uncertainties in the system. Safety, framed as forward invariance of sets in the state space, is ensured with the framework of control barrier functions (CBFs). Concretely, the definition of input to state safety (ISSf) is generalized to allow the synthesis of non-conservative, tunable controllers that are provably safe un…
Guaranteeing safe behavior on complex autonomous systems -- from cars to walking robots -- is challenging due to the inherently high dimensional nature of these systems and the corresponding complex models that may be difficult to determine in practice. With this as motivation, this paper presents a safety-critical control framework that leverages reduced order models to ensure safety on the full order dynamics -- even when these models are subject to disturbances and bounded inputs (e.g., actuation limits). To handle input constraints, the backup set method is reformulated in the context of r…
Neural networks have been increasingly applied for control in learning-enabled cyber-physical systems (LE-CPSs) and demonstrated great promises in improving system performance and efficiency, as well as reducing the need for complex physical models. However, the lack of safety guarantees for such neural network based controllers has significantly impeded their adoption in safety-critical CPSs. In this work, we propose a controller adaptation approach that automatically switches among multiple controllers, including neural network controllers, to guarantee system safety and improve energy effic…
This paper investigates the car-following problem and proposes a nonlinear controller that considers driving comfort, safety concerns, steady-state response and transient response. This controller is designed based on the demands of lower cost, faster response, increased comfort, enhanced safety and elevated extendability from the automotive industry. Design insights and intuitions are provided in detail. Also, theoretical analysis are performed on plant stability, string stability and tracking performance of the closed-loop system. Conditions and guidelines are provided on the selection of co…
Switching control strategies that unite a potentially high-performance but uncertified controller and a stabilizing albeit conservative controller are shown to be able to balance safety with efficiency, but have been less studied under partial observation of state. To address this gap, we propose a switching control strategy for partially observed linear-Gaussian systems with provable performance guarantees. We show that the proposed switching strategy is both safe and efficient, in the sense that: (1) the linear-quadratic cost of the system is always bounded even if the original uncertified c…
We present a novel class of nonlinear controllers that interpolates among differently behaving linear controllers as a case study for recently proposed Linear and Nonlinear System Level Synthesis framework. The structure of the nonlinear controller allows for simultaneously satisfying performance and safety objectives defined for small- and large-disturbance regimes. The proposed controller is distributed, handles delays, sparse actuation, and localizes disturbances. We show our nonlinear controller always outperforms its linear counterpart for constrained LQR problems. We further demonstrate …
Learning Safe Control via On-the-Fly Bandit Exploration
Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；TUM School of Computation, Information and Technology, Technical University of Munich（慕尼黑技术大学计算、信息与技术学院）
In this paper, we propose a novel framework for approximating the explicit MPC policy for linear parameter-varying systems using supervised learning. Our learning scheme guarantees feasibility and near-optimality of the approximated MPC policy with high probability. Furthermore, in contrast to most existing approaches that only learn the MPC policy, we also learn the "dual policy", which enables us to keep a check on the approximated MPC's optimality online during the control process. If the check deems the control input from the approximated MPC policy safe and near-optimal, then it is applie…
In this paper we propose sufficient conditions to synthesizing reach-avoid controllers for deterministic systems modelled by ordinary differential equations and stochastic systems modeled by stochastic differential equations based on the notion of control guidance-barrier functions. We begin with considering deterministic systems. Given an open safe set, a target set and a nominal controller, we attempt to synthesize a reach-avoid controller, which modifies the nominal controller in a minimal way enforcing the reach-avoid objective, i.e., the system will enter the target set eventually while s…
Combining efficient and safe control for safety-critical systems is challenging. Robust methods may be overly conservative, whereas probabilistic controllers require a trade-off between efficiency and safety. In this work, we propose a safety algorithm that is compatible with any stochastic Model Predictive Control method for linear systems with additive uncertainty and polytopic constraints. This safety algorithm allows to use the optimistic control inputs of stochastic Model Predictive Control as long as a safe backup planner can ensure safety with respect to satisfying hard constraints subj…
Research Institute of Trustworthy Autonomous Systems, SUSTech（南方科技大学可信自主系统研究院）；School of Automation, Hangzhou Dianzi University（杭州电子科技大学自动化学院）；Verimag, Université Grenoble Alpes（格勒诺布尔阿尔卑斯大学Verimag实验室）
We design and experimentally evaluate a hybrid safe-by-construction collision avoidance controller for autonomous vehicles. The controller combines into a single architecture the respective advantages of an adaptive controller and a discrete safe controller. The adaptive controller relies on model predictive control to achieve optimal efficiency in nominal conditions. The safe controller avoids collision by applying two different policies, for nominal and out-of-nominal conditions, respectively. We present design principles for both the adaptive and the safe controller and show how each one ca…
Researchers have developed various cascaded controllers and non-cascaded controllers for the navigation and control of quadrotors in recent years. It is vital to ensure the safety of a quadrotor both in normal state and in abnormal state if a controller tends to make the quadrotor unsafe. To this end, this paper proposes a non-cascaded Control Barrier Function (CBF) for a quadrotor controlled by either cascaded controllers or a non-cascaded controller. Incorporated with a Quadratic Programming (QP), the non-cascaded CBF can simultaneously regulate the magnitude of the total thrust and the torq…
