We propose a novel change point detection approach for online learning control with full information feedback (state, disturbance, and cost feedback) for unknown time-varying dynamical systems. We show that our algorithm can achieve a sub-linear regret with respect to the class of Disturbance Action Control (DAC) policies, which are a widely studied class of policies for online control of dynamical systems, for any sub-linear number of changes and very general class of systems: (i) matched disturbance system with general convex cost functions, (ii) general system with linear cost functions. Specifically, a (dynamic) regret of $Γ_T^{1/5}T^{4/5}$ can be achieved for these class of systems, where $Γ_T$ is the number of changes of the underlying system and $T$ is the duration of the control episode. That is, the change point detection approach achieves a sub-linear regret for any sub-linear number of changes, which other previous algorithms such as in \cite{minasyan2021online} cannot.
Numerically, we demonstrate that the change point detection approach is superior to a standard restart approach \cite{minasyan2021online} and to standard online learning approaches for time-invariant dynamical systems. Our work presents the first regret guarantee for unknown time-varying dynamical systems in terms of a stronger notion of variability like the number of changes in the underlying system. The extension of our work to state and output feedback controllers is a subject of future work.
We present a novel online learning algorithm for a class of unknown and uncertain dynamical environments that are fully observable. First, we obtain a novel probabilistic characterization of systems whose mean behavior is known but which are subject to additive, unknown subGaussian disturbances. This characterization relies on recent concentration of measure results and is given in terms of ambiguity sets. Second, we extend the results to environments whose mean behavior is also unknown but described by a parameterized class of possible mean behaviors. Our algorithm adapts the ambiguity set dy…
This paper addresses the problem of online inverse reinforcement learning for nonlinear systems with modeling uncertainties while in the presence of unknown disturbances. The developed approach observes state and input trajectories for an agent and identifies the unknown reward function online. Sub-optimality introduced in the observed trajectories by the unknown external disturbance is compensated for using a novel model-based inverse reinforcement learning approach. The observer estimates the external disturbances and uses the resulting estimates to learn the dynamic model of the demonstrato…
It is nontrivial to achieve exponential stability even for time-invariant nonlinear systems with matched uncertainties and persistent excitation (PE) condition. In this paper, without the need for PE condition, we address the problem of global exponential stabilization of strict-feedback systems with mismatched uncertainties and unknown yet time-varying control gains. The resultant control, embedded with time-varying feedback gains, is capable of ensuring global exponential stability of parametric-strict-feedback systems in the absence of persistence of excitation. By using the enhanced Nussba…
In this paper we are interested in the problem of adaptive state observation of linear time-varying (LTV) systems where the system and the input matrices depend on unknown time-varying parameters. It is assumed that these parameters satisfy some known LTV dynamics, but with unknown initial conditions. Moreover, the state equation is perturbed by an additive signal generated from an exosystem with uncertain constant parameters. Our main contribution is to propose a globally convergent state observer that requires only a weak excitation assumption on the system.
Motivated by the goal of learning controllers for complex systems whose dynamics change over time, we consider the problem of designing control laws for systems that switch among a finite set of unknown discrete-time linear subsystems under unknown switching signals. To this end, we propose a method that uses data to directly design a control mechanism without any explicit identification step. Our approach is online, meaning that the data are collected over time while the system is evolving in closed-loop, and are directly used to iteratively update the controller. A major benefit of the propo…
This work presents a notion of strong detectability for linear time varying systems affected by unknown inputs. It is shown that this notion is equivalent to detectability of an auxiliary system without unknown inputs. This allows a straightforward observer design for dependable state estimation in the presence of unknown inputs. The design reduces to a deterministic Kalman filter design problem, where the observer gains can be obtained from the solution of a differential Riccati equation. The efficacy of the proposed approach is demonstrated by means of a numerical example.
In this paper, a new yet indirect performance guaranteed framework is established to address the distributed tracking control problem for networked uncertain nonlinear strict-feedback systems with unknown time-varying gains under a directed interaction topology. The proposed framework involves two steps: In the first one, a fully distributed robust filter is constructed to estimate the desired trajectory for each agent with guaranteed observation performance that allows the directions among the agents to be non-identical. In the second one, by establishing a novel lemma regarding Nussbaum func…
We investigate the problem of stabilizing an unknown networked linear system under communication constraints and adversarial disturbances. We propose the first provably stabilizing algorithm for the problem. The algorithm uses a distributed version of nested convex body chasing to maintain a consistent estimate of the network dynamics and applies system level synthesis to determine a distributed controller based on this estimated model. Our approach avoids the need for system identification and accommodates a broad class of communication delay while being fully distributed and scaling favorabl…
In this paper, the model predictive control is designed for an interval type-2 Takagi-Sugeno (T-S) system with unknown time-varying delay in state and input vectors. The time-varying delay is a weird phenomenon that is appeared in almost all systems. It can make many problems and instability while the system is working. In this paper, the time-varying delay is considered in both states and input vectors and is the sensible difference between the proposed method here and previous algorithms, besides, it is unknown but bounded. To solve the problem, the Razumikhin approach is applied to the prop…
University of California at Berkeley（加州大学伯克利分校）；California Institute of Technology（加州理工学院）
We present a Model Predictive Control (MPC) strategy for unknown input-affine nonlinear dynamical systems. A non-parametric method is used to estimate the nonlinear dynamics from observed data. The estimated nonlinear dynamics are then linearized over time varying regions of the state space to construct an Affine Time Varying (ATV) model. Error bounds arising from the estimation and linearization procedure are computed by using sampling techniques. The ATV model and the uncertainty sets are used to design a robust Model Predictive Control (MPC) problem which guarantees safety for the unknown s…
Robust Adaptive Sliding Mode Control of Markovian Jump Systems with Uncertain Mode-dependent Time-varying Delays and Partly Unknown Transition Probabilities
Mississippi State University（密西西比州立大学）；University of Notre Dame（圣母大学）；Arak University（阿拉克大学）；Tarbiat Modares University（塔里亚特莫达雷斯大学）
A major limitation of online algorithms that track the optimizers of time-varying nonconvex optimization problems is that they focus on a specific local minimum trajectory, which may lead to poor spurious local solutions. In this paper, we show that the natural temporal variation may help simple online tracking methods find and track time-varying global minima. To this end, we investigate the properties of a time-varying projected gradient flow system with inertia, which can be regarded as the continuous-time limit of (1) the optimality conditions for a discretized sequential optimization prob…
Online optimization has recently opened avenues to study optimal control for time-varying cost functions that are unknown in advance. Inspired by this line of research, we study the distributed online linear quadratic regulator (LQR) problem for linear time-invariant (LTI) systems with unknown dynamics. Consider a multi-agent network where each agent is modeled as a LTI system. The network has a global time-varying quadratic cost, which may evolve adversarially and is only partially observed by each agent sequentially. The goal of the network is to collectively (i) estimate the unknown dynamic…
This paper considers a convex optimization problem with cost and constraints that evolve over time. The function to be minimized is strongly convex and possibly non-differentiable, and variables are coupled through linear constraints. In this setting, the paper proposes an online algorithm based on the alternating direction method of multipliers (ADMM), to track the optimal solution trajectory of the time-varying problem; in particular, the proposed algorithm consists of a primal proximal gradient descent step and an appropriately perturbed dual ascent step. The paper derives tracking results,…
A computationally efficient method for online joint state inference and dynamical model learning is presented. The dynamical model combines an a priori known, physically derived, state-space model with a radial basis function expansion representing unknown system dynamics and inherits properties from both physical and data-driven modeling. The method uses an extended Kalman filter approach to jointly estimate the state of the system and learn the unknown system dynamics, via the parameters of the basis function expansion. The key contribution is a computational complexity reduction compared to…
It is an interesting open problem to achieve adaptive prescribed-time control for strict-feedback systems with unknown and fast or even abrupt time-varying parameters. In this paper we present a solution with the aid of several design and analysis innovations. First, by using a spatiotemporal transformation, we convert the original system operational over finite time interval into one operational over infinite time interval, allowing for Lyapunov asymptotic design and recasting prescribed-time stabilization on finite time domain into asymptotic stabilization on infinite time domain. Second, to…
Voltage control generally requires accurate information about the grid's topology in order to guarantee network stability. However, accurate topology identification is a challenging problem for existing methods, especially as the grid is subject to increasingly frequent reconfiguration due to the adoption of renewable energy. Further, running existing control mechanisms with incorrect network information may lead to unstable control. In this work, we combine a nested convex body chasing algorithm with a robust predictive controller to achieve provably finite-time convergence to safe voltage li…
The paper investigates the problem of estimating the state of a time-varying system with a linear measurement model; in particular, the paper considers the case where the number of measurements available can be smaller than the number of states. In lieu of a batch linear least-squares (LS) approach -- well suited for static networks, where a sufficient number of measurements could be collected to obtain a full-rank design matrix -- the paper proposes an online algorithm to estimate the possibly time-varying state by processing measurements as and when available. The design of the algorithm hin…
Linear time-varying (LTV) systems are widely used for modeling real-world dynamical systems due to their generality and simplicity. Providing stability guarantees for LTV systems is one of the central problems in control theory. However, existing approaches that guarantee stability typically lead to significantly sub-optimal cumulative control cost in online settings where only current or short-term system information is available. In this work, we propose an efficient online control algorithm, COvariance Constrained Online Linear Quadratic (COCO-LQ) control, that guarantees input-to-state sta…
University of Technology Sydney（悉尼科技大学）；Centre for Autonomous Systems（自主系统中心）；Centre for Artificial Intelligence（人工智能中心）
This paper presents an observer-integrated Reinforcement Learning (RL) approach, called Disturbance OBserver Network (DOB-Net), for robots operating in environments where disturbances are unknown and time-varying, and may frequently exceed robot control capabilities. The DOB-Net integrates a disturbance dynamics observer network and a controller network. Originated from conventional DOB mechanisms, the observer is built and enhanced via Recurrent Neural Networks (RNNs), encoding estimation of past values and prediction of future values of unknown disturbances in RNN hidden state. Such encoding…
We present a method for incremental modeling and time-varying control of unknown nonlinear systems. The method combines elements of evolving intelligence, granular machine learning, and multi-variable control. We propose a State-Space Fuzzy-set-Based evolving Modeling (SS-FBeM) approach. The resulting fuzzy model is structurally and parametrically developed from a data stream with focus on memory and data coverage. The fuzzy controller also evolves, based on the data instances and fuzzy model parameters. Its local gains are redesigned in real-time -- whenever the corresponding local fuzzy mode…
This paper proposes a data-driven control framework to regulate an unknown, stochastic linear dynamical system to the solution of a (stochastic) convex optimization problem. Despite the centrality of this problem, most of the available methods critically rely on a precise knowledge of the system dynamics (thus requiring off-line system identification and model refinement). To this aim, in this paper we first show that the steady-state transfer function of a linear system can be computed directly from control experiments, bypassing explicit model identification. Then, we leverage the estimated …
This paper considers online switching control with a finite candidate controller pool, an unknown dynamical system, and unknown cost functions. The candidate controllers can be unstabilizing policies. We only require at least one candidate controller to satisfy certain stability properties, but we do not know which one is stabilizing. We design an online algorithm that guarantees finite-gain stability throughout the duration of its execution. We also provide a sublinear policy regret guarantee compared with the optimal stabilizing candidate controller. Lastly, we numerically test our algorithm…
Problem of an adaptive state observer design for nonlinear system with unknown time-varying parameters and under condition of delayed measurements is considered. State observation problem was raised by many researchers (see for example Sanx et al. (2019)). In this paper the results proposed in Bobtsov et al. (2021b), Bobtsov et al. (2021a), Bobtsov et al. (2022a), Bobtsov et al. (2022b) are developed. The problem is solved under assumption that the state matrix can be represented as sum of known and unknown parts. The output vector is measured with a known constant delay. An adaptive observer …
T2S-MPC: Time-Embedded Online Adaptive Model Predictive Control for Time-Varying Dynamics
JHU Department of Applied Mathematics and Statistics, Johns Hopkins University, MD, USA（约翰霍普金斯大学应用数学与统计学系）；CMU Department of Electrical and Computer Engineering, Carnegie Mellon University, PA, USA（卡内基梅隆大学电气与计算机工程系）
We propose an online detection procedure for cascading failures in the network from sequential data, which can be modeled as multiple correlated change-points happening during a short period. We consider a temporal diffusion network model to capture the temporal dynamic structure of multiple change-points and develop a sequential Shewhart procedure based on the generalized likelihood ratio statistics based on the diffusion network model assuming unknown post-change distribution parameters. We also tackle the computational complexity posed by the unknown propagation. Numerical experiments demon…
