This paper thoroughly investigates stochastic linear-quadratic optimal control problems with the Markovian regime switching system, where the coefficients of the state equation and the weighting matrices of the cost functional are random. We prove the solvability of the stochastic Riccati equation under the uniform convexity condition and obtain the closed-loop representation of the open-loop optimal control using the unique solvability of the corresponding stochastic Riccati equation. Moreover, by applying Itô's formula with jumps, we get a representation of the cost functional on a Hilbert space, characterized as the adapted solutions of some forward-backward stochastic differential equations. We show that the necessary condition of the open-loop optimal control is the convexity of the cost functional, and the sufficient condition of the open-loop optimal control is the uniform convexity of the cost functional. In addition, we study the properties of the stochastic value flow of the stochastic linear-quadratic optimal control problem. Finally, as an application, we present a continuous-time mean-variance portfolio selection problem and prove its unique solvability.
In this paper, we investigate the distributed convex optimization problem over a multi-agent system with Markovian switching communication networks. The objective function is the sum of each agent's local objective function, which cannot be known by other agents. The communication network is assumed to switch over a set of weight-balanced directed graphs with a Markovian property.We propose a consensus sub-gradient algorithm with two time-scale step-sizes to handle the uncertainty due to the Markovian switching topologies and the absence of global gradient information. With a proper selection …
We study a finite-dimensional continuous-time optimal control problem on finite horizon for a controlled diffusion driven by Brownian motion, in the linear-quadratic case. We admit stochastic coefficients, possibly depending on an underlying independent marked point process, so that our model is general enough to include controlled switching systems where the switching mechanism is not required to be Markovian. The problem is solved by means of a Riccati equation, which a backward stochastic differential equation driven by the Bronwian motion and by the random measure associated to the marked …
The optimization of process economics within the model predictive control (MPC) formulation has given rise to a new control paradigm known as economic MPC (EMPC). Several authors have discussed the closed-loop properties of EMPC-controlled deterministic systems, however, little have uncertain systems been studied. In this paper we propose EMPC formulations for nonlinear Markovian switching systems which guarantee recursive feasibility, asymptotic performance bounds and constrained mean square (MS) stability.
In this paper we prove a weak necessary and sufficient maximum principle for Markovian regime switching stochastic optimal control problems. Instead of insisting on the maximum condition of the Hamiltonian, we show that 0 belongs to the sum of Clarke's generalized gradient of the Hamiltonian and Clarke's normal cone of the control constraint set at the optimal control. Under a joint concavity condition on the Hamiltonian and a convexity condition on the terminal objective function, the necessary condition becomes sufficient. We give four examples to demonstrate the weak stochastic maximum prin…
This work focuses on stability analysis of numerical solutions to jump diffusions and jump diffusions with Markovian switching. Due to the use of Poisson processes, using asymptotic expansions as in the usual approach of treating diffusion processes does not work. Different from the existing treatments of Euler-Maurayama methods for solutions of stochastic differential equations, we use techniques from stochastic approximation. We analyze the almost sure exponential stability and exponential $p$-stability. The benchmark test model in numerical solutions, namely, one-dimensional linear scalar j…
This paper first describes a class of uncertain stochastic control systems with Markovian switching, and derives an Itô-Liu formula for Markov-modulated processes. And we characterize an optimal control law, which satisfies the generalized Hamilton-Jacobi-Bellman (HJB) equation with Markovian switching. Then, by using the generalized HJB equation, we deduce the optimal consumption and portfolio policies under uncertain stochastic financial markets with Markovian switching. Finally, for constant relative risk-aversion (CRRA) felicity functions, we explicitly obtain the optimal consumption and p…
In this paper, we investigate the open-loop and weak closed-loop solvabilities of stochastic linear quadratic (LQ, for short) optimal control problem of Markovian regime switching system. Interestingly, these two solvabilities are equivalent. We first provide an alternative characterization of the open-loop solvability of the LQ problem using the perturbation approach. Then, we study the weak closed-loop solvability of the LQ problem of Markovian regime switching system, and establish the equivalent relationship between open-loop and weak closed-loop solvabilities. Finally, we present an examp…
In this paper, we consider a continuous-time mean-variance portfolio selection with regime-switching and random horizon. Unlike previous works, the dynamic of assets are described by non-Markovian regime-switching models in the sense that all the market parameters are predictable with respect to the filtration generated jointly by Markov chain and Brownian motion. We formulate this problem as a constrained stochastic linear-quadratic optimal control problem. The Markov chain is assumed to be independent of the Brownian motion. So the market is incomplete. We derive closed-form expressions for …
We present a data-driven model predictive control scheme for chance-constrained Markovian switching systems with unknown switching probabilities. Using samples of the underlying Markov chain, ambiguity sets of transition probabilities are estimated which include the true conditional probability distributions with high probability. These sets are updated online and used to formulate a time-varying, risk-averse optimal control problem. We prove recursive feasibility of the resulting MPC scheme and show that the original chance constraints remain satisfied at every time step. Furthermore, we show…
In this paper, stability of linearly coupled dynamical systems with feedback pinning is studied. Event-triggered rules are employed on both diffusion coupling and feedback pinning to reduce the updating load of the coupled system. Here, both the coupling matrix and the set of pinned-nodes vary with time are induced by a homogeneous Markov chain. For each node, the diffusion coupling and feedback pinning are set up from the observation of its neighbors' and target's (if pinned) information at the latest event time and the next event time is triggered by some specified criteria. Two event-trigge…
In this paper, we consider a linear quadratic (LQ) leader-follower stochastic differential game for regime switching diffusions with mean-field interactions. One of the salient features of this paper is that conditional mean-field terms are included in the state equation and cost functionals. Based on stochastic maximum principles (SMPs), the follower's problem and the leader's problem are solved sequentially and an open-loop Stackelberg equilibrium is obtained. Further, with the help of the so-called four-step scheme, the corresponding Hamiltonian systems for the two players are decoupled and…
Linear-quadratic mixed Stackelberg-zero-sum game for mean-field regime switching system
Shandong University of Finance and Economics（山东财经大学）；Dalian University of Technology（大连理工大学）；The Hong Kong Polytechnic University（香港理工大学）；Imperial College（帝国理工学院）
We consider quadratic optimization in variables $(x,y)$ where $0\le x\le y$, and $y\in\{0,1\}^n$. Such binary $y$ are commonly refered to as "indicator" or "switching" variables and occur commonly in applications. One approach to such problems is based on representing or approximating the convex hull of the set $\{ (x,xx^T, yy^T) : 0\le x\le y\in\{0,1\}^n\}$. A representation for the case $n=1$ is known and has been widely used. We give an exact representation for the case $n=2$ by starting with a disjunctive representation for the convex hull and then eliminating auxilliary variables and cons…
Multidimensional indefinite stochastic Riccati equations and zero-sum stochastic linear-quadratic differential games with non-Markovian regime switching
Owing to the growth of interest in Reinforcement Learning in the last few years, gradient based policy control methods have been gaining popularity for Control problems as well. And rightly so, since gradient policy methods have the advantage of optimizing a metric of interest in an end-to-end manner, along with being relatively easy to implement without complete knowledge of the underlying system. In this paper, we study the global convergence of gradient-based policy optimization methods for quadratic control of discrete-time and model-free Markovian jump linear systems (MJLS). We surmount m…
Risk-averse model predictive control (MPC) offers a control framework that allows one to account for ambiguity in the knowledge of the underlying probability distribution and unifies stochastic and worst-case MPC. In this paper we study risk-averse MPC problems for constrained nonlinear Markovian switching systems using generic cost functions, and derive Lyapunov-type risk-averse stability conditions by leveraging the properties of risk-averse dynamic programming operators. We propose a controller design procedure to design risk-averse stabilizing terminal conditions for constrained nonlinear …
This paper studies switching stabilization problems for continuous-time switched linear systems. We consider four types of switching stabilizability defined under different assumptions on the switching control input. The most general switching stabilizability is defined as the existence of a measurable switching signal under which the resulting time-varying system is asymptotically stable. Discrete switching stabilizability is defined similarly but requires the switching signal to be piecewise constant on intervals of uniform length. In addition, we define feedback stabilizability in Filippov …
In this paper, stability of linearly coupled dynamical systems with feedback pinning algorithm is studied. Here, both the coupling matrix and the set of pinned-nodes vary with time, induced by a continuous-time Markov chain with finite states. Event-triggered rules are employed on both diffusion coupling and feedback pinning terms, which can efficiently reduce the computation load, as well as communication load in some cases and be realized by the latest observations of the state information of its local neighborhood and the target trajectory. The next observation is triggered by certain crite…
We provide an exhaustive treatment of Linear-Quadratic control problems for a class of stochastic Volterra equations of convolution type, whose kernels are Laplace transforms of certain signed matrix measures which are not necessarily finite. These equations are in general neither Markovian nor semimartingales, and include the fractional Brownian motion with Hurst index smaller than $1/2$ as a special case. We establish the correspondence of the initial problem with a possibly infinite dimensional Markovian one in a Banach space, which allows us to identify the Markovian controlled state varia…
In this paper, we study a stochastic linear-quadratic control problem with random coefficients and regime switching on a horizon $[0,T\wedgeτ]$, where $τ$ is a given random jump time for the underlying state process and $T$ is a constant. We obtain an explicit optimal state feedback control and explicit optimal cost value by solving a system of stochastic Riccati equations (SREs) with jumps on $[0,T\wedgeτ]$. By the decomposition approach stemming from filtration enlargement theory, we express the solution of the system of SREs with jumps in terms of another system of SREs involving only Brown…
In this paper we show existence and uniqueness of a solution for a system of m variational partial differential inequalities with inter-connected obstacles. This system is the deterministic version of the Verification Theorem of the Markovian optimal m-states switching problem. The switching cost functions are arbitrary. This problem is in relation with the valuation of firms in a financial market.
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Recently, policy optimization for control purposes has received renewed attention due to the increasing interest in reinforcement learning. In this paper, we investigate the global convergence of gradient-based policy optimization methods for quadratic optimal control of discrete-time Markovian jump linear systems (MJLS). First, we study the optimization landscape of direct policy optimization for MJLS, with static state feedback controllers and quadratic performance costs. Despite the non-convexity of the resultant problem, we are still able to identify several useful properties such as coerc…
This paper investigates the stochastic linear quadratic (LQ, for short) optimal control problem of Markov regime switching system. The representation of the cost functional for the stochastic LQ optimal control problem of Markov regime switching system is derived using the technique of It{ô}'s formula. For the stochastic LQ optimal control problem of Markov regime switching system, we establish the equivalence between the open-loop (closed-loop) solvability and the existence of an adapted solution to the corresponding forward-backward stochastic differential equation with constraint (the exist…
This paper introduces a new class of optimal switching problems, where the player is allowed to switch at a sequence of exogenous Poisson arrival times, and the underlying switching system is governed by an infinite horizon backward stochastic differential equation system. The value function and the optimal switching strategy are characterized by the solution of the underlying switching system. In a Markovian setting, the paper gives a complete description of the structure of switching regions by means of the comparison principle.
This paper studies switching stabilization problems for general switched nonlinear systems. A piecewise smooth control-Lyapunov function (PSCLF) approach is proposed and a constructive way to design a stabilizing switching law is developed. The switching law is constructed via the directional derivatives of the PSCLF with a careful discussion on various technical issues that may occur on the nonsmooth surfaces. Sufficient conditions are derived to ensure stability of the closed-loop Filippov solutions including possible sliding motions. The proposed PSCLF approach contains many existing result…
We prove a sufficient stochastic maximum principle for the optimal control of a regime-switching diffusion model. We show the connection to dynamic programming and we apply the result to a quadratic loss minimization problem, which can be used to solve a mean-variance portfolio selection problem.
In this paper we provide a set of stability conditions for linear time-varying networked control systems with arbitrary topologies using a piecewise quadratic switching stabilization approach with multiple quadratic Lyapunov functions. We use this set of stability conditions to provide a novel iterative low-complexity algorithm that must be updated and optimized in discrete time for the design of a sparse observer-controller network, for a given plant network with an arbitrary topology. We employ distributed observers by utilizing the output of other subsystems to improve the stability of each…
Motivated by reduction of computational complexity, this work develops sign-error adaptive filtering algorithms for estimating time-varying system parameters. Different from the previous work on sign-error algorithms, the parameters are time-varying and their dynamics are modeled by a discrete-time Markov chain. A distinctive feature of the algorithms is the multi-time-scale framework for characterizing parameter varia- tions and algorithm updating speeds. This is realized by considering the stepsize of the estimation algorithms and a scaling parameter that defines the transition rates of the …
This paper is concerned with the linear quadratic optimal control problem for networked system simultaneously with input delay and Markovian dropout. Different from the results in the literature, we consider the hold-input strategy, which is much more computationally complicated than zero-input strategy, but much better in most cases especially in the transition phase. Necessary and sufficient conditions for the solvability of optimal control problem over a finite horizon are given by the coupled difference Riccati-type equations. Moreover, the networked control system is mean-square stability…
In this paper we study the optimal m-states switching problem in finite horizon as well as infinite horizon with risk of default. We allow the switching cost functionals and cost of default to be of polynomial growth and arbitrary. We show uniqueness of a solution for a system of m variational partial differential inequalities with inter-connected obstacles. This system is the deterministic version of the Verification Theorem of the Markovian optimal m-states switching problem with risk of default. This problem is connected with the valuation of a power plant in the energy market.
This paper studies the problem of the deterministic version of the Verification Theorem for the optimal m-states switching in infinite horizon under Markovian framework with arbitrary switching cost functions. The problem is formulated as an extended impulse control problem and solved by means of probabilistic tools such as the Snell envelop of processes and reflected backward stochastic differential equations. A viscosity solutions approach is employed to carry out a finne analysis on the associated system of m variational inequalities with inter-connected obstacles. We show that the vector o…
We solve non-Markovian optimal switching problems in discrete time on an infinite horizon, when the decision maker is risk aware and the filtration is general, and establish existence and uniqueness of solutions for the associated reflected backward stochastic difference equations. An example application to hydropower planning is provided.
We establish the existence (and in an appropriate sense uniqueness) of Markovian solutions for ergodic BSDEs under a novel monotonicity condition. Our monotonicity condition allows us to prove existence even when the driver f has arbitrary (in particular super-quadratic) growth in z, which reveals an interesting trade-off between monotonicity and growth for ergodic BSDEs. The technique of proof is to establish a probabilistic representation of the derivative of the Markovian solution, and then use this representation to obtain a-priori estimates. Our study is motivated by applications to ergod…
