Recently, policy optimization for control purposes has received renewed attention due to the increasing interest in reinforcement learning. In this paper, we investigate the convergence of policy optimization for quadratic control of Markovian jump linear systems (MJLS). First, we study the optimization landscape of direct policy optimization for MJLS, and, in particular, show that despite the non-convexity of the resultant problem the unique stationary point is the global optimal solution. Next, we prove that the Gauss-Newton method and the natural policy gradient method converge to the optimal state feedback controller for MJLS at a linear rate if initialized at a controller which stabilizes the closed-loop dynamics in the mean square sense. We propose a novel Lyapunov argument to fix a key stability issue in the convergence proof. Finally, we present a numerical example to support our theory. Our work brings new insights for understanding the performance of policy learning methods on controlling unknown MJLS.
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Recently, policy optimization for control purposes has received renewed attention due to the increasing interest in reinforcement learning. In this paper, we investigate the global convergence of gradient-based policy optimization methods for quadratic optimal control of discrete-time Markovian jump linear systems (MJLS). First, we study the optimization landscape of direct policy optimization for MJLS, with static state feedback controllers and quadratic performance costs. Despite the non-convexity of the resultant problem, we are still able to identify several useful properties such as coerc…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Markovian jump linear systems (MJLS) are an important class of dynamical systems that arise in many control applications. In this paper, we introduce the problem of controlling unknown (discrete-time) MJLS as a new benchmark for policy-based reinforcement learning of Markov decision processes (MDPs) with mixed continuous/discrete state variables. Compared with the traditional linear quadratic regulator (LQR), our proposed problem leads to a special hybrid MDP (with mixed continuous and discrete variables) and poses significant new challenges due to the appearance of an underlying Markov jump p…
Owing to the growth of interest in Reinforcement Learning in the last few years, gradient based policy control methods have been gaining popularity for Control problems as well. And rightly so, since gradient policy methods have the advantage of optimizing a metric of interest in an end-to-end manner, along with being relatively easy to implement without complete knowledge of the underlying system. In this paper, we study the global convergence of gradient-based policy optimization methods for quadratic control of discrete-time and model-free Markovian jump linear systems (MJLS). We surmount m…
Research School of Computer Science, ANU（澳大利亚国立大学计算机科学研究学院）
Most real-world problems have huge state and/or action spaces. Therefore, a naive application of existing tabular solution methods is not tractable on such problems. Nonetheless, these solution methods are quite useful if an agent has access to a relatively small state-action space homomorphism of the true environment and near-optimal performance is guaranteed by the map. A plethora of research is focused on the case when the homomorphism is a Markovian representation of the underlying process. However, we show that near-optimal performance is sometimes guaranteed even if the homomorphism is n…
This paper studies the mean stability of positive semi-Markovian jump linear systems. We show that their mean stability is characterized by the spectral radius of a matrix that is easy to compute. In deriving the condition we use a certain discretization of a semi-Markovian jump linear system that preserves stability. Also we show a characterization for the exponential mean stability of continuous-time positive Markovian jump linear systems. Numerical examples are given to illustrate the results.
This work focuses on stability analysis of numerical solutions to jump diffusions and jump diffusions with Markovian switching. Due to the use of Poisson processes, using asymptotic expansions as in the usual approach of treating diffusion processes does not work. Different from the existing treatments of Euler-Maurayama methods for solutions of stochastic differential equations, we use techniques from stochastic approximation. We analyze the almost sure exponential stability and exponential $p$-stability. The benchmark test model in numerical solutions, namely, one-dimensional linear scalar j…
We find a maximum principle for general non-Markovian semi-martingales. We do so by describing the adjoint processes with non-anticipating stochastic derivatives in a martingale random field setting. In the case of the Lévy processes this extends maximum principles with Malliavin derivatives, in the sense that we replace Malliavin differentiability conditions with weaker and simpler $L_2$-conditions. As an application we use the maximum principle to solve a portfolio optimization problem for assets with credit risk modeled by doubly stochastic Poisson processes.
We consider a non-Markovian optimal stopping problem on finite horizon. We prove that the value process can be represented by means of a backward stochastic differential equation (BSDE), defined on an enlarged probability space, containing a stochastic integral having a one-jump point process as integrator and an (unknown) process with a sign constraint as integrand. This provides an alternative representation with respect to the classical one given by a reflected BSDE. The connection between the two BSDEs is also clarified. Finally, we prove that the value of the optimal stopping problem is t…
This paper addresses the design of a state observer for networked systems with random delays and dropouts. The model of plant and network covers the cases of multiple sensors, out-of-sequence and buffered measurements. The measurement outcomes over a finite interval model the network measurement reception scenarios, which follow a Markov distribution. We present a tractable optimization problem to precalculate off-line a finite set of gains of jump observers. The proposed procedure allows us to trade the complexity of the observer implementation for achieved performance. Several examples illus…
We present a data-driven model predictive control scheme for chance-constrained Markovian switching systems with unknown switching probabilities. Using samples of the underlying Markov chain, ambiguity sets of transition probabilities are estimated which include the true conditional probability distributions with high probability. These sets are updated online and used to formulate a time-varying, risk-averse optimal control problem. We prove recursive feasibility of the resulting MPC scheme and show that the original chance constraints remain satisfied at every time step. Furthermore, we show…
We study the optimal stopping problem for a monotonous dynamic risk measure induced by a BSDE with jumps in the Markovian case. We show that the value function is a viscosity solution of an obstacle problem for a partial integro-differential variational inequality, and we provide an uniqueness result for this obstacle problem.
The aim of this paper is to propose a new numerical approximation of the Kalman-Bucy filter for semi-Markov jump linear systems. This approximation is based on the selection of typical trajectories of the driving semi-Markov chain of the process by using an optimal quantization technique. The main advantage of this approach is that it makes pre-computations possible. We derive a Lipschitz property for the solution of the Riccati equation and a general result on the convergence of perturbed solutions of semi-Markov switching Riccati equations when the perturbation comes from the driving semi-Ma…
We consider the diffusive limit of a typical pure-jump Markovian control problem as the intensity of the driving Poisson process tends to infinity. We show that the convergence speed is provided by the Hölder constant of the Hessian of the limit problem, and explain how correction terms can be constructed. This provides an alternative efficient method for the numerical approximation of the optimal control of a pure-jump problem in situations with very high intensity of jump. We illustrate this approach in the context of a display advertising auction problem.
The mean square stabilization problem for discrete-time networked control systems (NCSs) is investigated in this article. What the difference from most previous works is that input delay and packet losses occur simultaneously in the communication channel, moreover, the data packet dropout is modeled as a time-homogeneous Markov process which will bring some difficulties in solving the problem due to the temporal correlation. The contributions in this paper can be summarized as two points. Firstly, the equivalence condition for the solvability of linear quadratic optimal problem in finite horiz…
In this paper, we study state-feedback control of Markov jump linear systems with partial information. In particular, we assume that the controller can only access the mode signals according to a hidden-Markov observation process. Our formulation generalizes various relevant cases previously studied in the literature on Markov jump linear systems, such as the cases with perfect information, no information, and cluster observations of the mode signals. In this context, we propose a Linear Matrix Inequalities (LMI) formulation to design feedback control laws for (stochastic) stabilization, $H_2$…
We consider a couple of integrodifferential PDEs arising from a stochastic Markovian control problem subjected to initial-terminal conditions. These equations correspond to the MFG system for a controlled jump-diffusion process. We prove that for a specific choice of the control function the expectation of the jump-diffusion process can be found explicitly. The study is an extension of similar results known for the pure diffusion process. As an example, we show how this can be applied to the problem of investors evaluating the trend of an asset when choosing an optimal portfolio.
This paper is concerned with the linear quadratic optimal control problem for networked system simultaneously with input delay and Markovian dropout. Different from the results in the literature, we consider the hold-input strategy, which is much more computationally complicated than zero-input strategy, but much better in most cases especially in the transition phase. Necessary and sufficient conditions for the solvability of optimal control problem over a finite horizon are given by the coupled difference Riccati-type equations. Moreover, the networked control system is mean-square stability…
This paper establishes a verification theorem for impulse control problems involving conditional McKean-Vlasov jump diffusions. We obtain a Markovian system by combining the state equation of the problem with the stochastic Fokker-Planck equation for the conditional probability law of the state. We derive sufficient variational inequalities for a function to be the value function of the impulse control problem, and for an impulse control to be the optimal control. We illustrate our results by applying them to the study of an optimal stream of dividends under transaction costs. We obtain the so…
We introduce a mixed {\em generalized} Dynkin game/stochastic control with ${\cal E}^f$-expectation in a Markovian framework. We study both the case when the terminal reward function is supposed to be Borelian only and when it is continuous. We first establish a weak dynamic programming principle by using some refined results recently provided in \cite{DQS} and some properties of doubly reflected BSDEs with jumps (DRBSDEs). We then show a stronger dynamic programming principle in the continuous case, which cannot be derived from the weak one. In particular, we have to prove that the value func…
We study optimal stochastic control problems of general coupled systems of forward-backward stochastic differential equations with jumps. By means of the Itô-Ventzell formula the system is transformed to a controlled backward stochastic partial differential equation (BSPDE) with jumps. Using a comparison principle for such BSPDEs we obtain a general stochastic Hamilton-Jacobi- Bellman (HJB) equation for such control problems. In the classical Markovian case with optimal control of jump diffusions, the equation reduces to the classical HJB equation. The results are applied to study risk minimiz…
This paper studies the consensusability problem of multi-agent systems (MASs), where agents communicate with each other through Markovian packet loss channels. We try to determine conditions under which there exists a linear distributed consensus controller such that the MAS can achieve mean square consensus. We first provide a necessary and sufficient consensus condition for MASs with single input and i.i.d.\ channel losses, which complements existing results. Then we proceed to study the case with identical Markovian packet losses. A necessary and sufficient consensus condition is firstly de…
The optimization of process economics within the model predictive control (MPC) formulation has given rise to a new control paradigm known as economic MPC (EMPC). Several authors have discussed the closed-loop properties of EMPC-controlled deterministic systems, however, little have uncertain systems been studied. In this paper we propose EMPC formulations for nonlinear Markovian switching systems which guarantee recursive feasibility, asymptotic performance bounds and constrained mean square (MS) stability.
We study the problem of optimal stopping of conditional McKean-Vlasov (mean-field) stochastic differential equations with jumps (conditional McKean-Vlasov jump diffusions, for short). We obtain sufficient variational inequalities for a function to be the value function of such a problem and for a stopping time to be optimal. To achieve this, we combine the state equation for the conditional McKean-Vlasov equation with the associated stochastic Fokker-Planck equation for the conditional law of the solution of the state. This gives us a Markovian system which can be handled by using a version of…
The purpose of this paper is to study optimal control of conditional McKean-Vlasov (mean-field) stochastic differential equations with jumps (conditional McKean-Vlasov jump diffusions, for short). To this end, we first prove a stochastic Fokker-Planck equation for the conditional law of the solution of such equations. Combining this equation with the original state equation, we obtain a Markovian system for the state and its conditional law. Furthermore, we apply this to formulate an Hamilton-Jacobi-Bellman (HJB) equation for the optimal control of conditional McKean-Vlasov jump diffusions. Th…
New linear minimum mean square estimators are introduced in this paper by considering a cluster information structure in the filter design. The set of filters constructed in this way can be ordered in a lattice according to the refines of clusters of the Markov chain, including the linear Markovian estimator at one end (with only one cluster) and the Kalman filter at the other hand (with as many clusters as Markov states). The higher is the number of clusters, the heavier are pre-compuations and smaller is the estimation error, so that the cluster cardinality allows for a trade-off between per…
