In this paper, the issue of model uncertainty in safety-critical control is addressed with a data-driven approach. For this purpose, we utilize the structure of an input-ouput linearization controller based on a nominal model along with a Control Barrier Function and Control Lyapunov Function based Quadratic Program (CBF-CLF-QP). Specifically, we propose a novel reinforcement learning framework which learns the model uncertainty present in the CBF and CLF constraints, as well as other control-affine dynamic constraints in the quadratic program. The trained policy is combined with the nominal model-based CBF-CLF-QP, resulting in the Reinforcement Learning-based CBF-CLF-QP (RL-CBF-CLF-QP), which addresses the problem of model uncertainty in the safety constraints. The performance of the proposed method is validated by testing it on an underactuated nonlinear bipedal robot walking on randomly spaced stepping stones with one step preview, obtaining stable and safe walking under model uncertainty.
This paper addresses the co-design problem of control inputs and execution decisions for event- and self-triggered controls subject to constraints given by the control Lyapunov function and control barrier function. The proposed approach computes the control input in a way that allows for longer inter-execution intervals, which distinguishes it from many existing event- and self-triggered controllers or control Lyapunov-barrier function controllers. The proposed approach guarantees lower bounds on the minimum inter-execution times. The effectiveness of the proposed approach is demonstrated and…
Recent work has shown that stabilizing an affine control system to a desired state while optimizing a quadratic cost subject to state and control constraints can be reduced to a sequence of Quadratic Programs (QPs) by using Control Barrier Functions (CBFs) and Control Lyapunov Functions (CLFs). In our own recent work, we defined High Order CBFs (HOCBFs) for systems and constraints with arbitrary relative degrees. In this paper, in order to accommodate initial states that do not satisfy the state constraints and constraints with arbitrary relative degree, we generalize HOCBFs to High Order Cont…
Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions
Delft University of Technology（代尔夫特理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；Huawei Technologies（华为技术有限公司）；University College London（伦敦大学学院）；University of Manchester（曼彻斯特大学）
Norwegian University of Science and Technology（挪威科技大学）
This paper presents a novel task-priority control framework for redundant robotic systems based on a hierarchy of control Lyapunov function (CLF) and control barrier function (CBF) based quadratic programs (QPs). The proposed method guarantees strict priority among different groups of tasks such as safety-related, operational and optimization tasks. Moreover, a soft priority measure in the form of penalty parameters can be employed to prioritize tasks at the same priority level. As opposed to kinematic control schemes, the proposed framework is a holistic approach to control of redundant robot…
This paper introduces differentiable higher-order control barrier functions (CBF) that are end-to-end trainable together with learning systems. CBFs are usually overly conservative, while guaranteeing safety. Here, we address their conservativeness by softening their definitions using environmental dependencies without loosing safety guarantees, and embed them into differentiable quadratic programs. These novel safety layers, termed a BarrierNet, can be used in conjunction with any neural network-based controller, and can be trained by gradient descent. BarrierNet allows the safety constraints…
Control Lyapunov functions (CLFs) and control barrier functions (CBFs) are widely used tools for synthesizing controllers subject to stability and safety constraints. Paired with online optimization, they provide stabilizing control actions that satisfy input constraints and avoid unsafe regions of state-space. Designing CLFs and CBFs with rigorous performance guarantees is computationally challenging. To certify existence of control actions, current techniques not only design a CLF/CBF, but also a nominal controller. This can make the synthesis task more expensive, and performance estimation …
As the performance of autonomous systems increases, safety concerns arise, especially when operating in non-structured environments. To deal with these concerns, this work presents a safety layer for mechanical systems that detects and responds to unstable dynamics caused by external disturbances. The safety layer is implemented independently and on top of already present nominal controllers, like pose or wrench tracking, and limits power flow when the system's response would lead to instability. This approach is based on the computation of the Largest Lyapunov Exponent (LLE) of the system's e…
This paper studies the problem of utilizing data-driven adaptive control techniques to guarantee stability and safety of uncertain nonlinear systems with high relative degree. We first introduce the notion of a High Order Robust Adaptive Control Barrier Function (HO-RaCBF) as a means to compute control policies guaranteeing satisfaction of high relative degree safety constraints in the face of parametric model uncertainty. The developed approach guarantees safety by initially accounting for all possible parameter realizations but adaptively reduces uncertainty in the parameter estimates levera…
Control barrier functions (CBFs) have become a popular tool to enforce safety of a control system. CBFs are commonly utilized in a quadratic program formulation (CBF-QP) as safety-critical constraints. A class $\mathcal{K}$ function in CBFs usually needs to be tuned manually in order to balance the trade-off between performance and safety for each environment. However, this process is often heuristic and can become intractable for high relative-degree systems. Moreover, it prevents the CBF-QP from generalizing to different environments in the real world. By embedding the optimization procedure…
Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室）
Safety and robustness are two desired properties for any reinforcement learning algorithm. CMDPs can handle additional safety constraints and RMDPs can perform well under model uncertainties. In this paper, we propose to unite these two frameworks resulting in robust constrained MDPs (RCMDPs). The motivation is to develop a framework that can satisfy safety constraints while also simultaneously offer robustness to model uncertainties. We develop the RCMDP objective, derive gradient update formula to optimize this objective and then propose policy gradient based algorithms. We also independentl…
This report aims to compare two safety methods: control barrier function and Hamilton-Jacobi reachability analysis. We will consider the difference with a focus on the following aspects: generality of system dynamics, difficulty of construction and computation cost. A standard Dubins car model will be evaluated numerically to make the comparison more concrete.
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Massachusetts Institute of Technology（麻省理工学院）
This work applies universal adaptive control to control barrier functions to achieve forward invariance of a safe set despite the presence of unmatched parametric uncertainties. The approach combines two ideas. The first is to construct a family of control barrier functions that ensures the system is safe for all possible models. The second is to use online parameter adaptation to methodically select a control barrier function and corresponding safety controller from the allowable set. While such a combination does not necessarily yield forward invariance without additional requirements on the…
Deploying Reinforcement Learning (RL) agents in the real-world require that the agents satisfy safety constraints. Current RL agents explore the environment without considering these constraints, which can lead to damage to the hardware or even other agents in the environment. We propose a new method, LBPO, that uses a Lyapunov-based barrier function to restrict the policy update to a safe set for each training iteration. Our method also allows the user to control the conservativeness of the agent with respect to the constraints in the environment. LBPO significantly outperforms state-of-the-a…
Control Barrier Corridors: From Safety Functions to Safe Sets
Department of Mechanical Engineering, Eindhoven University of Technology（埃因霍温理工大学机械工程系）；Department of Electrical and Computer Engineering, University of California San Diego（加州大学圣地亚哥分校电气与计算机工程系）
The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）
Safety guarantee is essential in many engineering implementations. Reinforcement learning provides a useful way to strengthen safety. However, reinforcement learning algorithms cannot completely guarantee safety over realistic operations. To address this issue, this work adopts control barrier functions over reinforcement learning, and proposes a compensated algorithm to completely maintain safety. Specifically, a sum-of-squares programming has been exploited to search for the optimal controller, and tune the learning hyperparameters simultaneously. Thus, the control actions are pledged to be …
Pacific Northwest National Laboratory（太平洋西北国家实验室）；Battelle Memorial Institute（巴特尔纪念研究所）
We develop a novel form of differentiable predictive control (DPC) with safety and robustness guarantees based on control barrier functions. DPC is an unsupervised learning-based method for obtaining approximate solutions to explicit model predictive control (MPC) problems. In DPC, the predictive control policy parametrized by a neural network is optimized offline via direct policy gradients obtained by automatic differentiation of the MPC problem. The proposed approach exploits a new form of sampled-data barrier function to enforce offline and online safety requirements in DPC settings while …
Safety in terms of collision avoidance for multi-robot systems is a difficult challenge under uncertainty, non-determinism and lack of complete information. This paper aims to propose a collision avoidance method that accounts for both measurement uncertainty and motion uncertainty. In particular, we propose Probabilistic Safety Barrier Certificates (PrSBC) using Control Barrier Functions to define the space of admissible control actions that are probabilistically safe with formally provable theoretical guarantee. By formulating the chance constrained safety set into deterministic control cons…
Model-free Deep Reinforcement Learning (DRL) controllers have demonstrated promising results on various challenging non-linear control tasks. While a model-free DRL algorithm can solve unknown dynamics and high-dimensional problems, it lacks safety assurance. Although safety constraints can be encoded as part of a reward function, there still exists a large gap between an RL controller trained with this modified reward and a safe controller. In contrast, instead of implicitly encoding safety constraints with rewards, we explicitly co-learn a Twin Neural Lyapunov Function (TNLF) with the contro…
California Institute of Technology（加州理工学院）；Argo AI
Control Barrier Functions (CBFs) have been demonstrated to be a powerful tool for safety-critical controller design for nonlinear systems. Existing design paradigms do not address the gap between theory (controller design with continuous time models) and practice (the discrete time sampled implementation of the resulting controllers); this can lead to poor performance and violations of safety for hardware instantiations. We propose an approach to close this gap by synthesizing sampled-data counterparts to these CBF-based controllers using approximate discrete time models and Sampled-Data Contr…
Control Barrier Functions (CBFs) is an important tool used to address situations with multiple concurrent control objectives, such as safety and goal convergence. In this paper we investigate the similarities between CBFs and so-called Behavior Control Lyapunov Functions (BCLFs) that have been proposed to address the same type of problems in the aeronautics domain. The key results of both CBFs and BCLFs is the description of the set of controls that render a given set invariant. We compare the corresponding theorems, and show that if we restrict the general class K function in CBFs to be the g…
CNRS（法国国家科学研究中心）；University of Rennes（雷恩大学）；Inria（法国国家信息与自动化研究所）；IRISA（法国信息与随机系统研究所）；University of Innsbruck（因斯布鲁克大学）
This paper presents an approach to deal with safety of dynamical systems in presence of multiple non-convex unsafe sets. While optimal control and model predictive control strategies can be employed in these scenarios, they suffer from high computational complexity in case of general nonlinear systems. Leveraging control barrier functions, on the other hand, results in computationally efficient control algorithms. Nevertheless, when safety guarantees have to be enforced alongside stability objectives, undesired asymptotically stable equilibrium points have been shown to arise. We propose a com…
This paper details the theory and implementation behind practically ensuring safety of remotely piloted racing drones. We demonstrate robust and practical safety guarantees on a 7" racing drone at speeds exceeding 100 km/h, utilizing only online computations on a 10 gram micro-controller. To achieve this goal, we utilize the framework of control barrier functions (CBFs) which give guaranteed safety encoded as forward set invariance. To make this methodology practically applicable, we present an implicitly defined CBF which leverages backup controllers to enable gradient-free evaluations that e…
Massachusetts Institute of Technology（麻省理工学院）；University of California, San Diego（加州大学圣地亚哥分校）
Learning-enabled control systems have demonstrated impressive empirical performance on challenging control problems in robotics, but this performance comes at the cost of reduced transparency and lack of guarantees on the safety or stability of the learned controllers. In recent years, new techniques have emerged to provide these guarantees by learning certificates alongside control policies -- these certificates provide concise, data-driven proofs that guarantee the safety and stability of the learned control system. These methods not only allow the user to verify the safety of a learned cont…
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；California Institute of Technology（加州理工学院）；University of Michigan（密歇根大学）
This work presents a theoretical framework for the safety-critical control of time delay systems. The theory of control barrier functions, that provides formal safety guarantees for delay-free systems, is extended to systems with state delay. The notion of control barrier functionals is introduced to attain formal safety guarantees, by enforcing the forward invariance of safe sets defined in the infinite dimensional state space. The proposed framework is able to handle multiple delays and distributed delays both in the dynamics and in the safety condition, and provides an affine constraint on …
To navigate complex environments, robots must increasingly use high-dimensional visual feedback (e.g. images) for control. However, relying on high-dimensional image data to make control decisions raises important questions; particularly, how might we prove the safety of a visual-feedback controller? Control barrier functions (CBFs) are powerful tools for certifying the safety of feedback controllers in the state-feedback setting, but CBFs have traditionally been poorly-suited to visual feedback control due to the need to predict future observations in order to evaluate the barrier function. I…
University of Toronto Institute for Aerospace Studies（多伦多大学航空航天研究所）；Technical University of Munich（慕尼黑工业大学）；University of Toronto Robotics Institute（多伦多大学机器人研究所）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）
In this work, we consider the problem of designing a safety filter for a nonlinear uncertain control system. Our goal is to augment an arbitrary controller with a safety filter such that the overall closed-loop system is guaranteed to stay within a given state constraint set, referred to as being safe. For systems with known dynamics, control barrier functions (CBFs) provide a scalar condition for determining if a system is safe. For uncertain systems, robust or adaptive CBF certification approaches have been proposed. However, these approaches can be conservative or require the system to have…
University of Michigan（密歇根大学）；California Institute of Technology（加州理工学院）；Plus.ai Inc.（智加科技公司）
Balancing safety and performance is one of the predominant challenges in modern control system design. Moreover, it is crucial to robustly ensure safety without inducing unnecessary conservativeness that degrades performance. In this work we present a constructive approach for safety-critical control synthesis via Control Barrier Functions (CBF). By filtering a hand-designed controller via a CBF, we are able to attain performant behavior while providing rigorous guarantees of safety. In the face of disturbances, robust safety and performance are simultaneously achieved through the notion of In…
Formally Verifying Deep Reinforcement Learning Controllers with Lyapunov Barrier Certificates
Stanford University（斯坦福大学）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）；IT University of Copenhagen（哥本哈根信息技术大学）；Google（谷歌公司）；GE Aerospace Research（GE航空航天研究院）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zürich（苏黎世联邦理工学院）；Google Research（谷歌研究院）
Learning controllers merely based on a performance metric has been proven effective in many physical and non-physical tasks in both control theory and reinforcement learning. However, in practice, the controller must guarantee some notion of safety to ensure that it does not harm either the agent or the environment. Stability is a crucial notion of safety, whose violation can certainly cause unsafe behaviors. Lyapunov functions are effective tools to assess stability in nonlinear dynamical systems. In this paper, we combine an improving Lyapunov function with automatic controller synthesis in …
In many real-world reinforcement learning (RL) problems, besides optimizing the main objective function, an agent must concurrently avoid violating a number of constraints. In particular, besides optimizing performance it is crucial to guarantee the safety of an agent during training as well as deployment (e.g. a robot should avoid taking actions - exploratory or not - which irrevocably harm its hardware). To incorporate safety in RL, we derive algorithms under the framework of constrained Markov decision problems (CMDPs), an extension of the standard Markov decision problems (MDPs) augmented …
A Safety-Aware Shared Autonomy Framework with BarrierIK Using Control Barrier Functions
Technical University of Darmstadt（德累斯顿技术大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Robotics Institute Germany (RIG)（德国机器人研究所）；Centre for Cognitive Science（认知科学中心）；Honda Research Institute Europe GmbH（本田欧洲研究机构）
Reinforcement learning (RL) has shown a promising performance in learning optimal policies for a variety of sequential decision-making tasks. However, in many real-world RL problems, besides optimizing the main objectives, the agent is expected to satisfy a certain level of safety (e.g., avoiding collisions in autonomous driving). While RL problems are commonly formalized as Markov decision processes (MDPs), safety constraints are incorporated via constrained Markov decision processes (CMDPs). Although recent advances in safe RL have enabled learning safe policies in CMDPs, these safety requir…
This paper combines episodic learning and control barrier functions in the setting of bipedal locomotion. The safety guarantees that control barrier functions provide are only valid with perfect model knowledge; however, this assumption cannot be met on hardware platforms. To address this, we utilize the notion of projection-to-state safety paired with a machine learning framework in an attempt to learn the model uncertainty as it affects the barrier functions. The proposed approach is demonstrated both in simulation and on hardware for the AMBER-3M bipedal robot in the context of the stepping…
Microsoft Corporation（微软公司）；Technion - Israel Institute of Technology（以色列理工学院）
Recent advances in Deep Machine Learning have shown promise in solving complex perception and control loops via methods such as reinforcement and imitation learning. However, guaranteeing safety for such learned deep policies has been a challenge due to issues such as partial observability and difficulties in characterizing the behavior of the neural networks. While a lot of emphasis in safe learning has been placed during training, it is non-trivial to guarantee safety at deployment or test time. This paper extends how under mild assumptions, Safety Barrier Certificates can be used to guarant…
Massachusetts Institute of Technology（麻省理工学院）；University of California, San Diego（加利福尼亚大学圣地亚哥分校）
Safety and stability are common requirements for robotic control systems; however, designing safe, stable controllers remains difficult for nonlinear and uncertain models. We develop a model-based learning approach to synthesize robust feedback controllers with safety and stability guarantees. We take inspiration from robust convex optimization and Lyapunov theory to define robust control Lyapunov barrier functions that generalize despite model uncertainty. We demonstrate our approach in simulation on problems including car trajectory tracking, nonlinear control with obstacle avoidance, satell…
We introduce High-Relative Degree Stochastic Control Lyapunov functions and Barrier Functions as a means to ensure asymptotic stability of the system and incorporate state dependent high relative degree safety constraints on a non-linear stochastic systems. Our proposed formulation also provides a generalisation to the existing literature on control Lyapunov and barrier functions for stochastic systems. The control policies are evaluated using a constrained quadratic program that is based on control Lyapunov and barrier functions. Our proposed control design is validated via simulated experime…
