We propose a Safe Pontryagin Differentiable Programming (Safe PDP) methodology, which establishes a theoretical and algorithmic framework to solve a broad class of safety-critical learning and control tasks -- problems that require the guarantee of safety constraint satisfaction at any stage of the learning and control progress. In the spirit of interior-point methods, Safe PDP handles different types of system constraints on states and inputs by incorporating them into the cost or loss through barrier functions. We prove three fundamentals of the proposed Safe PDP: first, both the solution and its gradient in the backward pass can be approximated by solving their more efficient unconstrained counterparts; second, the approximation for both the solution and its gradient can be controlled for arbitrary accuracy by a barrier parameter; and third, importantly, all intermediate results throughout the approximation and optimization strictly respect the constraints, thus guaranteeing safety throughout the entire learning and control process. We demonstrate the capabilities of Safe PDP in solving various safety-critical tasks, including safe policy optimization, safe motion planning, and learning MPCs from demonstrations, on different challenging systems such as 6-DoF maneuvering quadrotor and 6-DoF rocket powered landing.
This paper develops a Pontryagin Differentiable Programming (PDP) methodology, which establishes a unified framework to solve a broad class of learning and control tasks. The PDP distinguishes from existing methods by two novel techniques: first, we differentiate through Pontryagin's Maximum Principle, and this allows to obtain the analytical derivative of a trajectory with respect to tunable parameters within an optimal control system, enabling end-to-end learning of dynamics, policies, or/and control objective functions; and second, we propose an auxiliary control system in the backward pass…
The objective of trajectory optimization algorithms is to achieve an optimal collision-free path between a start and goal state. In real-world scenarios where environments can be complex and non-homogeneous, a robot needs to be able to gauge whether a state will be in collision with various objects in order to meet some safety metrics. The collision detector should be computationally efficient and, ideally, analytically differentiable to facilitate stable and rapid gradient descent during optimization. However, methods today lack an elegant approach to detect collision differentiably, relying …
Ain Shams University（艾因沙姆斯大学）；Jacobs University（雅各布大学）；Stanford University（斯坦福大学）；KTH Royal Institute of Technology（KTH皇家理工学院）
Reinforcement learning (RL) is capable of sophisticated motion planning and control for robots in uncertain environments. However, state-of-the-art deep RL approaches typically lack safety guarantees, especially when the robot and environment models are unknown. To justify widespread deployment, robots must respect safety constraints without sacrificing performance. Thus, we propose a Black-box Reachability-based Safety Layer (BRSL) with three main components: (1) data-driven reachability analysis for a black-box robot model, (2) a trajectory rollout planner that predicts future actions and ob…
Self-Supervised Path Planning in Unstructured Environments via Global-Guided Differentiable Hard Constraint Projection
State Key Laboratory of Tribology in Advanced Equipment, Tsinghua University（摩擦学国家重点实验室（先进装备））；Beijing Key Laboratory of Transformative High-end Manufacturing Equipment and Technology, Department of Mechanical Engineering, Tsinghua University（北京高端制造装备与技术关键实验室（机械工程系））；Automotive Electronics Business Unit at Hirain Inc.（惠而得（中国）汽车电子业务部）
In this paper, a safe and learning-based control framework for model predictive control (MPC) is proposed to optimize nonlinear systems with a non-differentiable objective function under uncertain environmental disturbances. The control framework integrates a learning-based MPC with an auxiliary controller in a way of minimal intervention. The learning-based MPC augments the prior nominal model with incremental Gaussian Processes to learn the uncertain disturbances. The cross-entropy method (CEM) is utilized as the sampling-based optimizer for the MPC with a non-differentiable objective functi…
Institute for Robotics and Intelligent Machines (IRIM)（机器人与智能机器研究所（IRIM））；Georgia Institute of Technology（佐治亚理工学院）；School of Aerospace Engineering（航空航天工程学院）；Stanford University（斯坦福大学）
This paper introduces a new formulation for stochastic optimal control and stochastic dynamic optimization that ensures safety with respect to state and control constraints. The proposed methodology brings together concepts such as Forward-Backward Stochastic Differential Equations, Stochastic Barrier Functions, Differentiable Convex Optimization and Deep Learning. Using the aforementioned concepts, a Neural Network architecture is designed for safe trajectory optimization in which learning can be performed in an end-to-end fashion. Simulations are performed on three systems to show the effica…
University of Pennsylvania（宾夕法尼亚大学）；Northwestern University（西北大学）；Monash University（莫纳什大学）；Northrop Grumman Corporation（诺斯罗普·格鲁曼公司）；Purdue University（普渡大学）
This paper develops the method of Continuous Pontryagin Differentiable Programming (Continuous PDP), which enables a robot to learn an objective function from a few sparsely demonstrated keyframes. The keyframes, labeled with some time stamps, are the desired task-space outputs, which a robot is expected to follow sequentially. The time stamps of the keyframes can be different from the time of the robot's actual execution. The method jointly finds an objective function and a time-warping function such that the robot's resulting trajectory sequentially follows the keyframes with minimal discrep…
This paper proposes embedded Gaussian Process Barrier States (GP-BaS), a methodology to safely control unmodeled dynamics of nonlinear system using Bayesian learning. Gaussian Processes (GPs) are used to model the dynamics of the safety-critical system, which is subsequently used in the GP-BaS model. We derive the barrier state dynamics utilizing the GP posterior, which is used to construct a safety embedded Gaussian process dynamical model (GPDM). We show that the safety-critical system can be controlled to remain inside the safe region as long as we can design a controller that renders the B…
Uber ATG（Uber 先进技术集团）；University of Toronto（多伦多大学）
In this paper we propose a novel end-to-end learnable network that performs joint perception, prediction and motion planning for self-driving vehicles and produces interpretable intermediate representations. Unlike existing neural motion planners, our motion planning costs are consistent with our perception and prediction estimates. This is achieved by a novel differentiable semantic occupancy representation that is explicitly used as cost by the motion planning process. Our network is learned end-to-end from human demonstrations. The experiments in a large-scale manual-driving dataset and clo…
Control barrier functions (CBFs) have become a popular tool to enforce safety of a control system. CBFs are commonly utilized in a quadratic program formulation (CBF-QP) as safety-critical constraints. A class $\mathcal{K}$ function in CBFs usually needs to be tuned manually in order to balance the trade-off between performance and safety for each environment. However, this process is often heuristic and can become intractable for high relative-degree systems. Moreover, it prevents the CBF-QP from generalizing to different environments in the real world. By embedding the optimization procedure…
Learning-based controllers, such as neural network (NN) controllers, can show high empirical performance but lack formal safety guarantees. To address this issue, control barrier functions (CBFs) have been applied as a safety filter to monitor and modify the outputs of learning-based controllers in order to guarantee the safety of the closed-loop system. However, such modification can be myopic with unpredictable long-term effects. In this work, we propose a safe-by-construction NN controller which employs differentiable CBF-based safety layers, and investigate the performance of safe-by-const…
We present GLAS: Global-to-Local Autonomy Synthesis, a provably-safe, automated distributed policy generation for multi-robot motion planning. Our approach combines the advantage of centralized planning of avoiding local minima with the advantage of decentralized controllers of scalability and distributed computation. In particular, our synthesized policies only require relative state information of nearby neighbors and obstacles, and compute a provably-safe action. Our approach has three major components: i) we generate demonstration trajectories using a global planner and extract local obser…
Mechanical Engineering, University of Michigan（密歇根大学机械工程学院）；Robotics Institute, University of Michigan（密歇根大学机器人研究所）；Mechanical and Industrial Engineering, Louisiana State University（路易斯安那州立大学机械与工业工程学院）；Naval Architecture and Marine Engineering, University of Michigan（密歇根大学船舶与海洋工程学院）；Ford Motor Company（福特汽车公司）；Office of Naval Research（美国海军研究办公室）
For robotic arms to operate in arbitrary environments, especially near people, it is critical to certify the safety of their motion planning algorithms. However, there is often a trade-off between safety and real-time performance; one can either carefully design safe plans, or rapidly generate potentially-unsafe plans. This work presents a receding-horizon, real-time trajectory planner with safety guarantees, called ARMTD (Autonomous Reachability-based Manipulator Trajectory Design). The method first computes (offline) a reachable set of parameterized trajectories for each joint of an arm. Eac…
Pacific Northwest National Laboratory（太平洋西北国家实验室）；Battelle Memorial Institute（巴特尔纪念研究所）
We develop a novel form of differentiable predictive control (DPC) with safety and robustness guarantees based on control barrier functions. DPC is an unsupervised learning-based method for obtaining approximate solutions to explicit model predictive control (MPC) problems. In DPC, the predictive control policy parametrized by a neural network is optimized offline via direct policy gradients obtained by automatic differentiation of the MPC problem. The proposed approach exploits a new form of sampled-data barrier function to enforce offline and online safety requirements in DPC settings while …
Motion planning for safe autonomous driving requires learning how the environment around an ego-vehicle evolves with time. Ego-centric perception of driveable regions in a scene not only changes with the motion of actors in the environment, but also with the movement of the ego-vehicle itself. Self-supervised representations proposed for large-scale planning, such as ego-centric freespace, confound these two motions, making the representation difficult to use for downstream motion planners. In this paper, we use geometric occupancy as a natural alternative to view-dependent representations suc…
University of Luxembourg（卢森堡大学）；Interdisciplinary Centre for Security, Reliability and Trust（安全、可靠性与信任跨学科中心）；ispace Europe（ispace欧洲公司）
Uncertain dynamic obstacles, such as pedestrians or vehicles, pose a major challenge for optimal robot navigation with safety guarantees. Previous work on motion planning has followed two main strategies to provide a safe bound on an obstacle's space: a polyhedron, such as a cuboid, or a nonlinear differentiable surface, such as an ellipsoid. The former approach relies on disjunctive programming, which has a relatively high computational cost that grows exponentially with the number of obstacles. The latter approach needs to be linearized locally to find a tractable evaluation of the chance co…
Differentiable Compliant Contact Primitives for Estimation and Model Predictive Control
Fraunhofer IPK（弗劳恩霍夫生产设备与设计技术研究所）；DGIST（大邱庆北科学技术院）；Politecnico di Milano（米兰理工大学）；Istituto Dalle Molle di Studi sull’Intelligenza Artificiale (IDSIA)（达莱莫勒人工智能研究所（IDSIA））；Scuola Universitaria Professionale della Svizzera Italiana (SUPSI)（瑞士意大利语区应用科学大学（SUPSI））；Università della Svizzera Italiana (USI)（瑞士意大利语区大学（USI））
Whole-body motion planning and safety-critical control for aerial manipulation
School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore（机械与航空航天工程学院，南洋理工大学，新加坡）；Department of Aerospace Engineering, Seoul National University, South Korea（航空航天工程系，首尔国立大学，韩国）
CogDrive: Cognition-Driven Multimodal Prediction-Planning Fusion for Safe Autonomy
Singapore-MIT Alliance for Research and Technology (SMART), Singapore（新加坡-麻省理工联合研究技术联盟）；Department of Urban Studies and Planning, Massachusetts Institute of Technology, USA（麻省理工学院城市研究与规划系）；School of Vehicle and Mobility, Tsinghua University, China（清华大学车辆与移动系统学院）；Department of Mechanical Engineering, National University of Singapore, Singapore（新加坡国立大学机械工程系）；Key Laboratory of Road and Traffic Engineering, Ministry of Education, Tongji University, China（同济大学交通工程重点实验室）
This paper introduces differentiable higher-order control barrier functions (CBF) that are end-to-end trainable together with learning systems. CBFs are usually overly conservative, while guaranteeing safety. Here, we address their conservativeness by softening their definitions using environmental dependencies without loosing safety guarantees, and embed them into differentiable quadratic programs. These novel safety layers, termed a BarrierNet, can be used in conjunction with any neural network-based controller, and can be trained by gradient descent. BarrierNet allows the safety constraints…
Large Language Model Enhanced Differentiable Trajectory Planning for IoT-Enabled Autonomous Driving
State Key Laboratory of Automotive Simulation and Control, Jilin University（吉林大学汽车仿真与控制国家重点实验室）；Center of Research for Cyber Security and Network (CSNET), Faculty of Computer Science and Information Technology, Universiti Malaya（马来西亚大学计算机科学与信息技术学院网络安全与网络研究中心）；Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg（卢森堡大学安全、可靠性与信任跨学科中心）；IMT, Department of Humanities and Technology, Roskilde University（罗斯基勒大学人文与技术系IMT）；Center for Security, Theory and Algorithmic Research, International Institute of Information Technology（国际信息技术研究所安全、理论与算法研究中心）；Department of Computer Science and Engineering, College of Informatics, Korea University（韩国大学信息学院计算机科学与工程系）
Learning responsibility allocations for multi-agent interactions: A differentiable optimization approach with control barrier functions
University of Washington, Department of Aeronautics and Astronautics（华盛顿大学航空航天系）；University of Texas, Austin, Department of Aerospace Engineering and Engineering Mechanics（得克萨斯大学奥斯汀分校航空航天工程与工程力学系）；NVIDIA
Northwestern University（西北大学）；UC Berkeley（加州大学伯克利分校）；University of Liverpool（利物浦大学）；Yale University（耶鲁大学）
In model-based reinforcement learning for safety-critical control systems, it is important to formally certify system properties (e.g., safety, stability) under the learned controller. However, as existing methods typically apply formal verification \emph{after} the controller has been learned, it is sometimes difficult to obtain any certificate, even after many iterations between learning and verification. To address this challenge, we propose a framework that jointly conducts reinforcement learning and formal verification by formulating and solving a novel bilevel optimization problem, which…
We study the problem of learning worst-case-safe parameters for programs that use neural networks as well as symbolic, human-written code. Such neurosymbolic programs arise in many safety-critical domains. However, because they can use nondifferentiable operations, it is hard to learn their parameters using existing gradient-based approaches to safe learning. Our approach to this problem, Differentiable Symbolic Execution (DSE), samples control flow paths in a program, symbolically constructs worst-case "safety losses" along these paths, and backpropagates the gradients of these losses through…
Guaranteeing safety of perception-based learning systems is challenging due to the absence of ground-truth state information unlike in state-aware control scenarios. In this paper, we introduce a safety guaranteed learning framework for vision-based end-to-end autonomous driving. To this end, we design a learning system equipped with differentiable control barrier functions (dCBFs) that is trained end-to-end by gradient descent. Our models are composed of conventional neural network architectures and dCBFs. They are interpretable at scale, achieve great test performance under limited training …
Massachusetts Institute of Technology（麻省理工学院）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Control certificates based on barrier functions have been a powerful tool to generate probably safe control policies for dynamical systems. However, existing methods based on barrier certificates are normally for white-box systems with differentiable dynamics, which makes them inapplicable to many practical applications where the system is a black-box and cannot be accurately modeled. On the other side, model-free reinforcement learning (RL) methods for black-box systems suffer from lack of safety guarantees and low sampling efficiency. In this paper, we propose a novel method that can learn s…
This paper proposes a differentiable linear quadratic Model Predictive Control (MPC) framework for safe imitation learning. The infinite-horizon cost is enforced using a terminal cost function obtained from the discrete-time algebraic Riccati equation (DARE), so that the learned controller can be proven to be stabilizing in closed-loop. A central contribution is the derivation of the analytical derivative of the solution of the DARE, thereby allowing the use of differentiation-based learning methods. A further contribution is the structure of the MPC optimization problem: an augmented Lagrangi…
Georgia Institute of Technology（佐治亚理工学院）；University of Waterloo（滑铁卢大学）；Outrider（Outrider公司）；University of California, Irvine（加州大学欧文分校）
Reinforcement Learning (RL) has been shown to be effective in many scenarios. However, it typically requires the exploration of a sufficiently large number of state-action pairs, some of which may be unsafe. Consequently, its application to safety-critical systems remains a challenge. An increasingly common approach to address safety involves the addition of a safety layer that projects the RL actions onto a safe set of actions. In turn, a difficulty for such frameworks is how to effectively couple RL with the safety layer to improve the learning performance. In this paper, we frame safety as …
Carnegie Mellon University（卡内基梅隆大学）；University of Colorado, Boulder（科罗拉多大学博尔德分校）
While reinforcement learning (RL) is gaining popularity in energy systems control, its real-world applications are limited due to the fact that the actions from learned policies may not satisfy functional requirements or be feasible for the underlying physical system. In this work, we propose PROjected Feasibility (PROF), a method to enforce convex operational constraints within neural policies. Specifically, we incorporate a differentiable projection layer within a neural network-based policy to enforce that all learned actions are feasible. We then update the policy end-to-end by propagating…
In this work, we propose a novel safe and scalable decentralized solution for multi-agent control in the presence of stochastic disturbances. Safety is mathematically encoded using stochastic control barrier functions and safe controls are computed by solving quadratic programs. Decentralization is achieved by augmenting to each agent's optimization variables, copy variables, for its neighbors. This allows us to decouple the centralized multi-agent optimization problem. However, to ensure safety, neighboring agents must agree on "what is safe for both of us" and this creates a need for consens…
Prediction of the Most Fire-Sensitive Point in Building Structures with Differentiable Agents for Thermal Simulators
Shenzhen International Graduate School Tsinghua University（深圳国际研究生院清华大学）；Pacific Earthquake Engineering Research (PEER) Center University of California, Berkeley（太平洋地震工程研究中心加州大学伯克利分校）；University of California, Berkeley（加州大学伯克利分校）
Safe Reinforcement learning (Safe RL) aims at learning optimal policies while staying safe. A popular solution to Safe RL is shielding, which uses a logical safety specification to prevent an RL agent from taking unsafe actions. However, traditional shielding techniques are difficult to integrate with continuous, end-to-end deep RL methods. To this end, we introduce Probabilistic Logic Policy Gradient (PLPG). PLPG is a model-based Safe RL technique that uses probabilistic logic programming to model logical safety constraints as differentiable functions. Therefore, PLPG can be seamlessly applie…
Integrating Physics-Informed Neural Networks for Safe Reinforcement Learning in a 1-DoF Helicopter System
Josef Ressel Centre for Intelligent and Secure Industrial Automation, Salzburg University of Applied Sciences（约瑟夫·雷塞尔智能与安全工业自动化中心，萨尔茨堡应用科学大学）；Salzburg University of Applied Sciences（萨尔茨堡应用科学大学）；Paris Lodron University of Salzburg（萨尔茨堡帕里斯·洛德隆大学）
