We develop a method for obtaining safe initial policies for reinforcement learning via approximate dynamic programming (ADP) techniques for uncertain systems evolving with discrete-time dynamics. We employ kernelized Lipschitz estimation and semidefinite programming for computing admissible initial control policies with provably high probability. Such admissible controllers enable safe initialization and constraint enforcement while providing exponential stability of the equilibrium of the closed-loop system.
Wayne State University（韦恩州立大学）；Virginia Tech（弗吉尼亚理工大学）
We study a stochastic bandit problem with a general unknown reward function and a general unknown constraint function. Both functions can be non-linear (even non-convex) and are assumed to lie in a reproducing kernel Hilbert space (RKHS) with a bounded norm. This kernelized bandit setup strictly generalizes standard multi-armed bandits and linear bandits. In contrast to safety-type hard constraints studied in prior works, we consider soft constraints that may be violated in any round as long as the cumulative violations are small, which is motivated by various practical applications. Our ultim…
University of Pennsylvania（宾夕法尼亚大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）；Google Brain Robotics（谷歌大脑机器人）
Inspired by the success of imitation and inverse reinforcement learning in replicating expert behavior through optimal control, we propose a learning based approach to safe controller synthesis based on control barrier functions (CBFs). We consider the setting of a known nonlinear control affine dynamical system and assume that we have access to safe trajectories generated by an expert - a practical example of such a setting would be a kinematic model of a self-driving vehicle with safe trajectories (e.g., trajectories that avoid collisions with obstacles in the environment) generated by a hum…
We present a method for feedback motion planning of systems with unknown dynamics which provides probabilistic guarantees on safety, reachability, and goal stability. To find a domain in which a learned control-affine approximation of the true dynamics can be trusted, we estimate the Lipschitz constant of the difference between the true and learned dynamics, and ensure the estimate is valid with a given probability. Provided the system has at least as many controls as states, we also derive existence conditions for a one-step feedback law which can keep the real system within a small bound of …
Kernel Single-Index Bandits: Estimation, Inference, and Learning
Department of Mathematics, Applied Mathematics, and Statistics, Case Western Reserve University（数学、应用数学与统计学系，凯斯西储大学）；Department of Statistics, University of Florida（佛罗里达大学统计学系）；Department of Statistics, The Pennsylvania State University（宾夕法尼亚州立大学统计学系）
Inverter-based distributed energy resources provide the possibility for fast time-scale voltage control by quickly adjusting their reactive power. The power-electronic interfaces allow these resources to realize almost arbitrary control law, but designing these decentralized controllers is nontrivial. Reinforcement learning (RL) approaches are becoming increasingly popular to search for policy parameterized by neural networks. It is difficult, however, to enforce that the learned controllers are safe, in the sense that they may introduce instabilities into the system. This paper proposes a saf…
The control Barrier function approach has been widely used for safe controller synthesis. By solving an online convex quadratic programming problem, an optimal safe controller can be synthesized implicitly in state-space. Since the solution is unique, the mapping from state-space to control inputs is injective, thus enabling us to evaluate the underlying relationship. In this paper we aim at explicitly synthesizing a safe control law as a function of the state for nonlinear control-affine systems with limited control ability. We propose to transform the online quadratic programming problem int…
This paper studies the control problem for safety-critical multi-agent systems based on quadratic programming (QP). Each controlled agent is modeled as a cascade connection of an integrator and an uncertain nonlinear actuation system. In particular, the integrator represents the position-velocity relation, and the actuation system describes the dynamic response of the actual velocity to the velocity reference signal. The notion of input-to-output stability (IOS) is employed to characterize the essential velocity-tracking capability of the actuation system. The uncertain actuation dynamics may …
We present a method for contraction-based feedback motion planning of locally incrementally exponentially stabilizable systems with unknown dynamics that provides probabilistic safety and reachability guarantees. Given a dynamics dataset, our method learns a deep control-affine approximation of the dynamics. To find a trusted domain where this model can be used for planning, we obtain an estimate of the Lipschitz constant of the model error, which is valid with a given probability, in a region around the training data, providing a local, spatially-varying model error bound. We derive a traject…
