Recently, policy optimization for control purposes has received renewed attention due to the increasing interest in reinforcement learning. In this paper, we investigate the global convergence of gradient-based policy optimization methods for quadratic optimal control of discrete-time Markovian jump linear systems (MJLS). First, we study the optimization landscape of direct policy optimization for MJLS, with static state feedback controllers and quadratic performance costs. Despite the non-convexity of the resultant problem, we are still able to identify several useful properties such as coercivity, gradient dominance, and almost smoothness. Based on these properties, we show global convergence of three types of policy optimization methods: the gradient descent method; the Gauss-Newton method; and the natural policy gradient method. We prove that all three methods converge to the optimal state feedback controller for MJLS at a linear rate if initialized at a controller which is mean-square stabilizing. Some numerical examples are presented to support the theory. This work brings new insights for understanding the performance of policy gradient methods on the Markovian jump linear quadratic control problem.
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Recently, policy optimization for control purposes has received renewed attention due to the increasing interest in reinforcement learning. In this paper, we investigate the convergence of policy optimization for quadratic control of Markovian jump linear systems (MJLS). First, we study the optimization landscape of direct policy optimization for MJLS, and, in particular, show that despite the non-convexity of the resultant problem the unique stationary point is the global optimal solution. Next, we prove that the Gauss-Newton method and the natural policy gradient method converge to the optim…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Markovian jump linear systems (MJLS) are an important class of dynamical systems that arise in many control applications. In this paper, we introduce the problem of controlling unknown (discrete-time) MJLS as a new benchmark for policy-based reinforcement learning of Markov decision processes (MDPs) with mixed continuous/discrete state variables. Compared with the traditional linear quadratic regulator (LQR), our proposed problem leads to a special hybrid MDP (with mixed continuous and discrete variables) and poses significant new challenges due to the appearance of an underlying Markov jump p…
Owing to the growth of interest in Reinforcement Learning in the last few years, gradient based policy control methods have been gaining popularity for Control problems as well. And rightly so, since gradient policy methods have the advantage of optimizing a metric of interest in an end-to-end manner, along with being relatively easy to implement without complete knowledge of the underlying system. In this paper, we study the global convergence of gradient-based policy optimization methods for quadratic control of discrete-time and model-free Markovian jump linear systems (MJLS). We surmount m…
This paper studies the mean stability of positive semi-Markovian jump linear systems. We show that their mean stability is characterized by the spectral radius of a matrix that is easy to compute. In deriving the condition we use a certain discretization of a semi-Markovian jump linear system that preserves stability. Also we show a characterization for the exponential mean stability of continuous-time positive Markovian jump linear systems. Numerical examples are given to illustrate the results.
This work focuses on stability analysis of numerical solutions to jump diffusions and jump diffusions with Markovian switching. Due to the use of Poisson processes, using asymptotic expansions as in the usual approach of treating diffusion processes does not work. Different from the existing treatments of Euler-Maurayama methods for solutions of stochastic differential equations, we use techniques from stochastic approximation. We analyze the almost sure exponential stability and exponential $p$-stability. The benchmark test model in numerical solutions, namely, one-dimensional linear scalar j…
We find a maximum principle for general non-Markovian semi-martingales. We do so by describing the adjoint processes with non-anticipating stochastic derivatives in a martingale random field setting. In the case of the Lévy processes this extends maximum principles with Malliavin derivatives, in the sense that we replace Malliavin differentiability conditions with weaker and simpler $L_2$-conditions. As an application we use the maximum principle to solve a portfolio optimization problem for assets with credit risk modeled by doubly stochastic Poisson processes.
We consider a non-Markovian optimal stopping problem on finite horizon. We prove that the value process can be represented by means of a backward stochastic differential equation (BSDE), defined on an enlarged probability space, containing a stochastic integral having a one-jump point process as integrator and an (unknown) process with a sign constraint as integrand. This provides an alternative representation with respect to the classical one given by a reflected BSDE. The connection between the two BSDEs is also clarified. Finally, we prove that the value of the optimal stopping problem is t…
This paper addresses the design of a state observer for networked systems with random delays and dropouts. The model of plant and network covers the cases of multiple sensors, out-of-sequence and buffered measurements. The measurement outcomes over a finite interval model the network measurement reception scenarios, which follow a Markov distribution. We present a tractable optimization problem to precalculate off-line a finite set of gains of jump observers. The proposed procedure allows us to trade the complexity of the observer implementation for achieved performance. Several examples illus…
University of Michigan（密歇根大学）；University of California, Riverside（加州大学河滨分校）
Real-world control applications often involve complex dynamics subject to abrupt changes or variations. Markov jump linear systems (MJS) provide a rich framework for modeling such dynamics. Despite an extensive history, theoretical understanding of parameter sensitivities of MJS control is somewhat lacking. Motivated by this, we investigate robustness aspects of certainty equivalent model-based optimal control for MJS with quadratic cost function. Given the uncertainty in the system matrices and in the Markov transition matrix is bounded by $ε$ and $η$ respectively, robustness results are esta…
We study a finite-dimensional continuous-time optimal control problem on finite horizon for a controlled diffusion driven by Brownian motion, in the linear-quadratic case. We admit stochastic coefficients, possibly depending on an underlying independent marked point process, so that our model is general enough to include controlled switching systems where the switching mechanism is not required to be Markovian. The problem is solved by means of a Riccati equation, which a backward stochastic differential equation driven by the Bronwian motion and by the random measure associated to the marked …
In this paper, we investigate the open-loop and weak closed-loop solvabilities of stochastic linear quadratic (LQ, for short) optimal control problem of Markovian regime switching system. Interestingly, these two solvabilities are equivalent. We first provide an alternative characterization of the open-loop solvability of the LQ problem using the perturbation approach. Then, we study the weak closed-loop solvability of the LQ problem of Markovian regime switching system, and establish the equivalent relationship between open-loop and weak closed-loop solvabilities. Finally, we present an examp…
This paper is concerned with the stochastic linear-quadratic optimal control problem with Poisson jumps. The coefficients in the state equation and the weighting matrices in the cost functional are all deterministic but are allowed indefinite. The notion of closed-loop strategies is introduced, and the optimal closed-loop strategy is characterized by a Riccati integral-differential equation and a backward stochastic differential equation with Poisson jumps.
In this paper, we investigate the solvability of matrix valued Backward stochastic Riccati equations with jumps (BSREJ), which is associated with a stochastic linear quadratic (SLQ) optimal control problem with random coefficients and driven by both Brownian motion and Poisson jumps. By dynamic programming principle, Doob-Meyer decomposition and inverse flow technique, the existence and uniqueness of the solution for the BSREJ is established. The difficulties addressed to this issue not only are brought from the high nonlinearity of the generator of the BSREJ like the case driven only by Brown…
This paper thoroughly investigates stochastic linear-quadratic optimal control problems with the Markovian regime switching system, where the coefficients of the state equation and the weighting matrices of the cost functional are random. We prove the solvability of the stochastic Riccati equation under the uniform convexity condition and obtain the closed-loop representation of the open-loop optimal control using the unique solvability of the corresponding stochastic Riccati equation. Moreover, by applying Itô's formula with jumps, we get a representation of the cost functional on a Hilbert s…
We study the discrete-time approximation for solutions of quadratic forward back- ward stochastic differential equations (FBSDEs) driven by a Brownian motion and a jump process which could be dependent. Assuming that the generator has a quadratic growth w.r.t. the variable z and the terminal condition is bounded, we prove the convergence of the scheme when the number of time steps n goes to infinity. Our approach is based on the companion paper [15] and allows to get a convergence rate similar to that of schemes of Brownian FBSDEs.
We provide an exhaustive treatment of Linear-Quadratic control problems for a class of stochastic Volterra equations of convolution type, whose kernels are Laplace transforms of certain signed matrix measures which are not necessarily finite. These equations are in general neither Markovian nor semimartingales, and include the fractional Brownian motion with Hurst index smaller than $1/2$ as a special case. We establish the correspondence of the initial problem with a possibly infinite dimensional Markovian one in a Banach space, which allows us to identify the Markovian controlled state varia…
This paper is concerned with a linear quadratic (LQ, for short) optimal control problem for mean-field backward stochastic differential equations (MF-BSDE, for short) driven by a Poisson random martingale measure and a Brownian motion. Firstly, by the classic convex variation principle, the existence and uniqueness of the optimal control is established. Secondly, the optimal control is characterized by the stochastic Hamilton system which turns out to be a linear fully coupled mean-field forward-backward stochastic differential equation with jumps by the duality method. Thirdly, in terms of a …
In this paper, we study a linear-quadratic optimal control problem for mean-field stochastic differential equations driven by a Poisson random martingale measure and a multidimensional Brownian motion. Firstly, the existence and uniqueness of the optimal control is obtained by the classic convex variation principle. Secondly, by the duality method, the optimality system, also called the stochastic Hamilton system which turns out to be a linear fully coupled mean-field forward-backward stochastic differential equation with jumps, is derived to characterize the optimal control. Thirdly, applying…
The aim of this paper is to propose a new numerical approximation of the Kalman-Bucy filter for semi-Markov jump linear systems. This approximation is based on the selection of typical trajectories of the driving semi-Markov chain of the process by using an optimal quantization technique. The main advantage of this approach is that it makes pre-computations possible. We derive a Lipschitz property for the solution of the Riccati equation and a general result on the convergence of perturbed solutions of semi-Markov switching Riccati equations when the perturbation comes from the driving semi-Ma…
We consider the diffusive limit of a typical pure-jump Markovian control problem as the intensity of the driving Poisson process tends to infinity. We show that the convergence speed is provided by the Hölder constant of the Hessian of the limit problem, and explain how correction terms can be constructed. This provides an alternative efficient method for the numerical approximation of the optimal control of a pure-jump problem in situations with very high intensity of jump. We illustrate this approach in the context of a display advertising auction problem.
In this paper, we study a stochastic linear-quadratic control problem with random coefficients and regime switching on a horizon $[0,T\wedgeτ]$, where $τ$ is a given random jump time for the underlying state process and $T$ is a constant. We obtain an explicit optimal state feedback control and explicit optimal cost value by solving a system of stochastic Riccati equations (SREs) with jumps on $[0,T\wedgeτ]$. By the decomposition approach stemming from filtration enlargement theory, we express the solution of the system of SREs with jumps in terms of another system of SREs involving only Brown…
Indefinite Stochastic Linear-Quadratic Optimal Control Problems with Random Coefficients and Poisson Jumps: Closed-Loop Representation of Open-Loop Optimal Controls
Southern University of Science and Technology（南方科技大学）；Southeast University（东南大学）
In this paper, we investigate infinite horizon jump-diffusion forward-backward stochastic differential equations under some monotonicity conditions. We establish an existence and uniqueness theorem, two stability results and a comparison theorem for solutions to such kind of equations. Then the theoretical results are applied to study a kind of infinite horizon backward stochastic linear-quadratic optimal control problems, and then differential game problems. The unique optimal controls for the control problems and the unique Nash equilibrium points for the game problems are obtained in closed…
In this paper, we consider a linear-quadratic optimal control problem of mean-field stochastic differential equation with jump diffusion, which is also called as an MF-LQJ problem. Here, cost functional is allowed to be indefinite. We use an equivalent cost functional method to deal with the MF-LQJ problem with indefinite weighting matrices. Some equivalent cost functionals enable us to establish a bridge between indefinite and positive-definite MF-LQJ problems. With such a bridge, solvabilities of stochastic Hamiltonian system and Riccati equations are further characterized. Optimal control o…
This paper mainly investigates the optimal control and stabilization problems for linear discrete-time Markov jump systems. The general case for the finite-horizon optimal controller is considered, where the input weighting matrix in the performance index is just required to be positive semi-definite. The necessary and sufficient condition for the existence of the optimal controller in finite-horizon is given explicitly from a set of coupled difference Riccati equations (CDRE). One of the key techniques is to solve the forward and backward stochastic difference equation (FDSDE) which is obtain…
The mean square stabilization problem for discrete-time networked control systems (NCSs) is investigated in this article. What the difference from most previous works is that input delay and packet losses occur simultaneously in the communication channel, moreover, the data packet dropout is modeled as a time-homogeneous Markov process which will bring some difficulties in solving the problem due to the temporal correlation. The contributions in this paper can be summarized as two points. Firstly, the equivalence condition for the solvability of linear quadratic optimal problem in finite horiz…
On explicit solutions to a class of quadratic BSDEJs driven by affine Volterra processes with jumps and applications
Laboratoire de Probabilités, Statistique et Modélisation, UMR 8001, Sorbonne Université and Université Paris Cité（概率、统计与建模实验室，UMR 8001，索邦大学与巴黎西岱大学）；BPCE SA（BPCE股份公司）
In this paper, we study state-feedback control of Markov jump linear systems with partial information. In particular, we assume that the controller can only access the mode signals according to a hidden-Markov observation process. Our formulation generalizes various relevant cases previously studied in the literature on Markov jump linear systems, such as the cases with perfect information, no information, and cluster observations of the mode signals. In this context, we propose a Linear Matrix Inequalities (LMI) formulation to design feedback control laws for (stochastic) stabilization, $H_2$…
This paper considers linear-quadratic (LQ) stochastic leader-follower Stackelberg differential games for jump-diffusion systems with random coefficients. We first solve the LQ problem of the follower using the stochastic maximum principle and obtain the state-feedback representation of the open-loop optimal solution in terms of the integro-stochastic Riccati differential equation (ISRDE), where the state-feedback type control is shown to be optimal via the completion of squares method. Next, we establish the stochastic maximum principle for the indefinite LQ stochastic optimal control problem …
JuMP is an open-source modeling language that allows users to express a wide range of optimization problems (linear, mixed-integer, quadratic, conic-quadratic, semidefinite, and nonlinear) in a high-level, algebraic syntax. JuMP takes advantage of advanced features of the Julia programming language to offer unique functionality while achieving performance on par with commercial modeling tools for standard tasks. In this work we will provide benchmarks, present the novel aspects of the implementation, and discuss how JuMP can be extended to new problem classes and composed with state-of-the-art…
We consider a couple of integrodifferential PDEs arising from a stochastic Markovian control problem subjected to initial-terminal conditions. These equations correspond to the MFG system for a controlled jump-diffusion process. We prove that for a specific choice of the control function the expectation of the jump-diffusion process can be found explicitly. The study is an extension of similar results known for the pure diffusion process. As an example, we show how this can be applied to the problem of investors evaluating the trend of an asset when choosing an optimal portfolio.
We develop a simple routine unifying the analysis of several important recently-developed stochastic optimization methods including SAGA, Finito, and stochastic dual coordinate ascent (SDCA). First, we show an intrinsic connection between stochastic optimization methods and dynamic jump systems, and propose a general jump system model for stochastic optimization methods. Our proposed model recovers SAGA, SDCA, Finito, and SAG as special cases. Then we combine jump system theory with several simple quadratic inequalities to derive sufficient conditions for convergence rate certifications of the…
Multidimensional indefinite stochastic Riccati equations and zero-sum stochastic linear-quadratic differential games with non-Markovian regime switching
