This paper studies the control of safety-critical dynamical systems in the presence of adversarial disturbances. We seek to synthesize state-feedback controllers to minimize a cost incurred due to the disturbance, while respecting a safety constraint. The safety constraint is given by a bound on an H-inf norm, while the cost is specified as an upper bound on the H-2 norm of the system. We consider an online setting where costs at each time are revealed only after the controller at that time is chosen. We propose an iterative approach to the synthesis of the controller by solving a modified discrete-time Riccati equation. Solutions of this equation enforce the safety constraint. We compare the cost of this controller with that of the optimal controller when one has complete knowledge of disturbances and costs in hindsight. We show that the regret function, which is defined as the difference between these costs, varies logarithmically with the time horizon. We validate our approach on a process control setup that is subject to two kinds of adversarial attacks.
Speaker identification models are vulnerable to carefully designed adversarial perturbations of their input signals that induce misclassification. In this work, we propose a white-box steganography-inspired adversarial attack that generates imperceptible adversarial perturbations against a speaker identification model. Our approach, FoolHD, uses a Gated Convolutional Autoencoder that operates in the DCT domain and is trained with a multi-objective loss function, in order to generate and conceal the adversarial perturbation within the original audio files. In addition to hindering speaker ident…
City University of Hong Kong（香港城市大学）；University of Victoria（维多利亚大学）；University of Puerto Rico at Mayagüez（波多黎各大学马亚圭斯分校）
In recent years, many deep learning models have been adopted in autonomous driving. At the same time, these models introduce new vulnerabilities that may compromise the safety of autonomous vehicles. Specifically, recent studies have demonstrated that adversarial attacks can cause a significant decline in detection precision of deep learning-based 3D object detection models. Although driving safety is the ultimate concern for autonomous driving, there is no comprehensive study on the linkage between the performance of deep learning models and the driving safety of autonomous vehicles under adv…
State-of-the-art machine learning models can be vulnerable to very small input perturbations that are adversarially constructed. Adversarial training is an effective approach to defend against such examples. It is formulated as a min-max problem, searching for the best solution when the training data was corrupted by the worst-case attacks. For linear regression problems, adversarial training can be formulated as a convex problem. We use this reformulation to make two technical contributions: First, we formulate the training problem as an instance of robust regression to reveal its connection …
Adversarial Nibbler: A Data-Centric Challenge for Improving the Safety of Text-to-Image Models
Google（谷歌）；Oxford University（牛津大学）；Harvard University（哈佛大学）；CMU（卡内基梅隆大学）；Cohere（Cohere公司）；UCL（伦敦大学学院）；University of Zurich（苏黎世大学）；MLCommons（机器学习通用组织）；Kaggle（Kaggle平台）
University of Liverpool（利物浦大学）；University of Oxford（牛津大学）；Lancaster University（兰卡斯特大学）；Defence Science and Technology Laboratory (Dstl)（国防科学技术实验室）；Queen’s University Belfast（贝尔法斯特女王大学）
In the past few years, significant progress has been made on deep neural networks (DNNs) in achieving human-level performance on several long-standing tasks. With the broader deployment of DNNs on various applications, the concerns over their safety and trustworthiness have been raised in public, especially after the widely reported fatal incidents involving self-driving cars. Research to address these concerns is particularly active, with a significant number of papers released in the past few years. This survey paper conducts a review of the current research effort into making DNNs safe and …
This work provides formal safety guarantees for control systems with disturbance. A disturbance observer-based robust safety-critical controller is proposed, that estimates the effect of the disturbance on safety and utilizes this estimate with control barrier functions to attain provably safe dynamic behavior. The observer error bound - which consists of transient and steady-state parts - is quantified, and the system is endowed with robustness against this error via the proposed controller. An adaptive cruise control problem is used as illustrative example through simulations including real …
Google AI Princeton（谷歌AI普林斯顿）；Princeton University（普林斯顿大学）；University of Washington（华盛顿大学）；Allen School of Computer Science and Engineering（阿伦计算机科学与工程学院）
Princeton University（普林斯顿大学）；Google AI Princeton（谷歌人工智能普林斯顿分部）
We consider the problem of generating maximally adversarial disturbances for a given controller assuming only blackbox access to it. We propose an online learning approach to this problem that \emph{adaptively} generates disturbances based on control inputs chosen by the controller. The goal of the disturbance generator is to minimize \emph{regret} versus a benchmark disturbance-generating policy class, i.e., to maximize the cost incurred by the controller as well as possible compared to the best possible disturbance generator \emph{in hindsight} (chosen from a benchmark policy class). In the …
Resilient systems are able to recover quickly and easily from disturbed system states that might result from hazardous events or malicious attacks. In this paper a novel resilience metric for linear time invariant systems is proposed: the minimum control energy required to disturb the system is set into relation to the minimum control energy needed to recover. This definition extends known disturbance rejection metrics considering random effects to account for adversarial disturbances. The worst-case disturbance and the related resilience index can be computed efficiently via solving a general…
Recent developments in cyber-physical systems and event-triggered control have led to an increased interest in the impact of sparse disturbances on dynamical processes. We study Linear Quadratic Regulator (LQR) control under sparse disturbances by analyzing three distinct policies: the blind online policy, the disturbance-aware policy, and the optimal offline policy. We derive the two-dimensional recurrence structure of the optimal disturbance-aware policy, under the assumption that the controller has information about future disturbance values with only a probabilistic model of their location…
An approach to resilient planning and control of autonomous vehicles in multi-vehicle traffic scenarios is proposed. The proposed method is based on model predictive control (MPC), where alternative predictions of the surrounding traffic are determined automatically such that they are intentionally adversarial to the ego vehicle. This provides robustness against the inherent uncertainty in traffic predictions. To reduce conservatism, an assumption that other agents are of no ill intent is formalized. Simulation results from highway driving scenarios show that the proposed method in real-time n…
CBF-QP safety filters are pointwise minimizers of the control effort at a given state vector, i.e., myopically optimal at each time instant. But are they optimal over the entire infinite time horizon? What does it even mean for a controlled dynamic systems to be "optimally safe" as opposed to, conventionally "optimally stable"? When disturbances, deterministic and stochastic, have unknown upper bounds, how should safety be defined to allow a graceful degradation under disturbances? Can safety filters be designed to guarantee such weaker safety properties as well as the optimality of safety ove…
This paper presents a strategy for control of a spacecraft docking with a non-maneuvering target in the presence of safety constraints and bounded disturbances. The presence of disturbances prevents convergence to a unique docking state, so in our formulation, docking is defined as occurring within a set constructed using prescribed tolerances. Safety is ensured via application of Robust Control Barrier Functions to render a designated safe set forward invariant for any allowable disturbance. However, this safety strategy necessarily presumes a worst-case disturbance, and thus restricts trajec…
Safety for control systems is often posed as an invariance constraint; the system is said to be safe if state trajectories avoid some unsafe region of the statespace for all time. An assured controller is one that enforces safety online by filtering a desired control input at runtime, and control barrier functions (CBFs) provide an assured controller that renders a safe subset of the state-space forward invariant. Recent extensions propose CBF-based assured controllers that allow the system to leave a known safe set so long as a given backup control strategy eventually returns to the safe set,…
In a complex real-time operating environment, external disturbances and uncertainties adversely affect the safety, stability, and performance of dynamical systems. This paper presents a robust stabilizing safety-critical controller synthesis framework with control Lyapunov functions (CLFs) and control barrier functions (CBFs) in the presence of disturbance. A high-gain input observer method is adapted to estimate the time-varying unmodelled dynamics of the CBF with an error bound using the first-order time derivative of the CBF. This approach leads to an easily tunable low order disturbance es…
Dynamic Constraint Reconstruction Based Control Barrier Functions for Safety-Critical Control of High-Dimensional Manipulators
Yaowu (Shenzhen) Technology Co., Ltd.（耀武（深圳）科技有限公司）；National Engineering Research Center of Mobile Network Technologies, Beijing University of Posts and Telecommunications（北京邮电大学移动通信国家重点实验室）
This paper presents a safe robust policy iteration (SR-PI) algorithm to design controllers with satisficing (good enough) performance and safety guarantee. This is in contrast to standard PI-based control design methods with no safety certification. It also moves away from existing safe control design approaches that perform pointwise optimization and are thus myopic. Safety assurance requires satisfying a control barrier function (CBF), which might be in conflict with the performance-driven Lyapunov solution to the Bellman equation arising in each iteration of the PI. Therefore, a new develop…
To bring complex systems into real world environments in a safe manner, they will have to be robust to uncertainties - both in the environment and the system. This paper investigates the safety of control systems under input disturbances, wherein the disturbances can capture uncertainties in the system. Safety, framed as forward invariance of sets in the state space, is ensured with the framework of control barrier functions (CBFs). Concretely, the definition of input to state safety (ISSf) is generalized to allow the synthesis of non-conservative, tunable controllers that are provably safe un…
RTD-RAX: Fast, Safe Trajectory Planning for Systems under Unknown Disturbances
Georgia Institute of Technology（佐治亚理工学院）；School of Electrical and Computer Engineering（电子与计算机工程学院）；School of Mechanical Engineering（机械工程学院）；School of Civil and Environmental Engineering（土木与环境工程学院）
