We study a linear quadratic optimal control problem with stochastic coefficients and a terminal state constraint, which may be in force merely on a set with positive, but not necessarily full probability. Under such a partial terminal constraint, the usual approach via a coupled system of a backward stochastic Riccati equation and a linear backward equation breaks down. As a remedy, we introduce a family of auxiliary problems parametrized by the supersolutions to this Riccati equation alone. The target functional of these problems dominates the original constrained one and allows for an explicit description of both the optimal control policy and the auxiliary problem's value in terms of a suitably constructed optimal signal process. This suggests that, for the minimal supersolution of the Riccati equation, the minimizers of the auxiliary problem coincide with those of the original problem, a conjecture that we see confirmed in all examples understood so far.
This paper is concerned with a linear quadratic (LQ, for short) optimal control problem with fixed terminal states and integral quadratic constraints. A Riccati equation with infinite terminal value is introduced, which is uniquely solvable and whose solution can be approximated by the solution for a suitable unconstrained LQ problem with penalized terminal state. Using results from duality theory, the optimal control is explicitly derived by solving the Riccati equation together with an optimal parameter selection problem. It turns out that the optimal control is not only a feedback of the cu…
We discuss an open-loop backward Stackelberg differential game involving single leader and single follower. Unlike most Stackelberg game literature, the state to be controlled is characterized by a backward stochastic differential equation (BSDE) for which the terminal- instead initial-condition is specified as a priori; the decisions of leader consist of a static terminal-perturbation and a dynamic linear-quadratic control. In addition, the terminal control is subject to (convex-closed) pointwise and (affine) expectation constraints. Both constraints are arising from real applications such as…
A stochastic linear quadratic (LQ) optimal control problem with a pointwise linear equality constraint on the terminal state is considered. A strong Lagrangian duality theorem is proved under a uniform convexity condition on the cost functional and a surjectivity condition on the linear constraint mapping. Based on the Lagrangian duality, two approaches are proposed to solve the constrained stochastic LQ problem. First, a theoretical method is given to construct the closed-form solution by the strong duality. Second, an iterative algorithm, called augmented Lagrangian method (ALM), is proposed…
In this paper, we study the optimal control problem of a controlled time-symmetric forward-backward doubly stochastic differential equation with initial-terminal sate constraints. Applying the terminal perturbation method and Ekeland's variation principle, a necessary condition of the stochastic optimal control, i.e., stochastic maximum principle is derived. Applications to backward doubly stochastic linear-quadratic control models are investigated.
In this study, we propose a varying terminal time structure for the optimal control problem under state constraints, in which the terminal time follows the varying of the control via the constrained condition. Focusing on this new optimal control problem, we investigate a novel stochastic maximum principle, which differs from the traditional optimal control problem under state constraints. The optimal pair of the optimal control model can be verified via this new stochastic maximum principle.
The stability of stochastic Model Predictive Control (MPC) subject to additive disturbances is often demonstrated in the literature by constructing Lyapunov-like inequalities that guarantee closed-loop performance bounds and boundedness of the state, but convergence to a terminal control law is typically not shown. In this work we use results on general state space Markov chains to find conditions that guarantee convergence of disturbed nonlinear systems to terminal modes of operation, so that they converge in probability to a priori known terminal linear feedback laws and achieve time-average…
In this paper we study a general optimal liquidation problem with a control-dependent stopping time which is the first time the stock holding becomes zero or a fixed terminal time, whichever comes first. We prove a stochastic maximum principle (SMP) which is markedly different in its Hamiltonian condition from that of the standard SMP with fixed terminal time. We present a simple example in which the optimal solution satisfies the SMP in this paper but fails the standard SMP in the literature.
In the present paper we derive, via a backward induction technique, and ad hoc maximum principle for an optimal control problem with multiple random terminal times. Therefore we apply the aforementioned result to the case of a linear quadratic controller, providing solutions for the optimal control in terms of Riccati backward SDE with random terminal time. Eventually all the above results are applied to a system of interconnected banks.
We consider a stochastic control problem, where the control domain is convex and the system is governed by a nonlinear backward stochastic differential equation. With a L1 terminal data, we derive necessary optimality conditions in the form of stochastic maximum principle.
In this paper, we investigate an optimal control problem with terminal stochastic linear complementarity constraints (SLCC), and its discrete approximation using the relaxation, the sample average approximation (SAA) and the implicit Euler time-stepping scheme. We show the existence of feasible solutions and optimal solutions to the optimal control problem and its discrete approximation under the conditions that the expectation of the stochastic matrix in the SLCC is a Z-matrix or an adequate matrix. Moreover, we prove that the solution sequence generated by the discrete approximation converge…
We study the optimal liquidation problems in target zone models using dynamic programming methods. Such control problems allow for stochastic differential equations with reflections and random coefficients. The value function is characterized with a Neumann problem of backward stochastic partial differential equations (BSPDEs) with singular terminal conditions. The existence and the uniqueness of strong solution to such BSPDEs are addressed, which in turn yields the optimal feedback control. In addition, the unique existence of strong solution to Neumann problem of general semilinear BSPDEs in…
We investigate a stochastic optimal control problem where the controlled system is depicted as a stochastic differential delayed equation; however, at the terminal time, the state is constrained in a convex set. We firstly introduce an equivalent backward delayed system depicted as a time-delayed backward stochastic differential equation. Then a stochastic maximum principle is obtained by virtue of Ekeland's variational principle. Finally, applications to a state constrained stochastic delayed linear-quadratic control model and a production-consumption choice problem are studied to illustrate …
In this study, we consider an optimal control problem driven by a stochastic differential system with a stopping time terminal cost functional. We establish the stochastic maximum principle for this new kind of an optimal control problem by introducing a discrete terminal system. Finally, we provide an example to describe the main results of this study.
Incomplete financial markets are considered, defined by a multi-dimensional non-homogeneous diffusion process, being the direct sum of an Itô process (the price process), and another non-homogeneous diffusion process (the exogenous process, representing exogenous stochastic sources). The drift and the diffusion matrix of the price process are functions of the time, the price process itself and the exogenous process. In the context of such markets and for power utility functions, it is proved that the stochastic control problem consisting of optimizing the expected utility of the terminal wealt…
In this paper, we study the regularity of the value function associated with a stochastic control problem where two controls act simultaneously on a modulated multidimensional diffusion process. The first is a switching control modelling a random clock. Every time the random clock rings, the generator matrix is replaced by another, resulting in a different dynamic for the finite state Markov chain of the modulated diffusion process. The second is a singular stochastic control that is executed on the process within each regime.
Stochastic optimal control problems with constraints on the probability distribution of the final output are considered. Necessary conditions for optimality in the form of a coupled system of partial differential equations involving a forward Fokker-Planck equation and a backward Hamilton-Jacobi-Bellman equation are proved using convex duality techniques.
This paper is concerned with solutions to a one dimensional linear diffusion equation and their relation to some problems in stochastic control theory. A stochastic variational formula is obtained for the logarithm of the solution to the diffusion equation, with terminal data which is the characteristic function of a set. In this case the terminal data for the control problem is singular, and hence standard theory does not apply. The variational formula is used to prove convergence in the zero noise limit of the cost function for the stochastic control problem and its first derivatives, to the…
We consider a class of stochastic optimal control problems for discrete-time stochastic linear systems which seek for control policies that will steer the probability distribution of the terminal state of the system close to a desired Gaussian distribution. In our problem formulation, the closeness between the terminal state distribution and the desired (goal) distribution is measured in terms of the squared Wasserstein distance which is associated with a corresponding terminal cost term. We recast the stochastic optimal control problem as a finite-dimensional nonlinear program and we show tha…
We study a multi-dimensional optimal execution problem in illiquid markets with both instantaneous and persistent price impact and stochastic resilience. In our model the value function can be described by a multi-dimensional backward stochastic Riccati differential equations (BSRDE) with a singular terminal condition in one component. We prove the existence of a solution to the BSRDE system and characterise both the value function and the optimal strategy in terms of that solution. We prove that the solution to the liquidation problem can be approximated by the solutions to a sequence of unco…
Nonlinear Stochastic Optimal Control and Optimal Stopping using the Fokker-Planck Transformation
Daniel Guggenheim School of Aerospace, Georgia Institute of Technology（佐治亚理工学院丹尼尔·古根海姆航空学院）；Department of Mechanical Engineering, University of Alabama（阿拉巴马大学机械工程系）
We study the closed-loop solvability of a stochastic linear quadratic optimal control problem for systems governed by stochastic evolution equations. This solvability is established by means of solvability of the corresponding Riccati equation, which is implied by the uniform convexity of the quadratic cost functional. At last, conditions ensuring the uniform convexity of the cost functional are discussed.
In this paper, we generalise Pontryagin's stochastic maximum principle to controlled McKean-Vlasov equations with anticipating law. The associated new type of delayed backward equations with implicit terminal condition is studied.
In this paper, we consider linear quadratic optimal control with mean-field type for discrete-time stochastic systems with state and control dependent noise. An optimal control problem is studied for a linear mean-field stochastic differential equation with a quadratic cost functional. The coefficients and the weighting matrices in the cost functional are all assumed to be deterministic.
We show how a stochastic version of the Lagrange multiplier method can be combined with the stochastic maximum principle for jump diffusions to solve certain constrained stochastic optimal control problems. Two different terminal constraints are considered; one constraint holds in expectation and the other almost surely. As an application of this method, we study the effects of inflation- and wage risk on optimal consumption. To do this, we consider the optimal consumption problem for a budget constrained agent with a Lévy income process and stochastic inflation. The agent must choose a consum…
This paper considers the stochastic linear quadratic optimal control problem in which the control domain is nonconvex. By the functional analysis and convex perturbation methods, we establish a novel maximum principle. The application of the proposed maximum principle is illustrated through a work-out example.
One of the fundamental issues in Control Theory is to design feedback controls. It is well-known that, the purpose of introducing Riccati equations in the deterministic case is to provide the desired feedback controls for linear quadratic control problems. To date, the same problem in the stochastic setting is only partially well-understood. In this paper, we establish the equivalence between the existence of optimal feedback controls for the stochastic linear quadratic control problems with random coefficients and the solvability of the corresponding backward stochastic Riccati equations in a…
Efficient Riccati equation based techniques for the approximate solution of discrete time linear regulator problems are restricted in their application to problems with quadratic terminal payoffs. Where non-quadratic terminal payoffs are required, these techniques fail due to the attendant non-quadratic value functions involved. In order to compute these non-quadratic value functions, it is often necessary to appeal directly to dynamic programming in the form of grid- or element-based iterations for the value function. These iterations suffer from poor scalability with respect to problem dimen…
We establish existence, uniqueness and regularity of solution results for a class of backward stochastic partial differential equations with singular terminal condition. The equation describes the value function of non-Markovian stochastic optimal control problem in which the terminal state of the controlled process is pre-specified. The analysis of such control problems is motivated by models of optimal portfolio liquidation.
This paper studies a continuous-time stochastic linear-quadratic (SLQ) optimal control problem on infinite-horizon. A data-driven policy iteration algorithm is proposed to solve the SLQ problem. Without knowing three system coefficient matrices, this algorithm uses the collected data to iteratively approximate a solution of the corresponding stochastic algebraic Riccati equation (SARE). A simulation example is provided to illustrate the effectiveness and applicability of the algorithm.
This paper is concerned with the stochastic linear-quadratic optimal control problem with Poisson jumps. The coefficients in the state equation and the weighting matrices in the cost functional are all deterministic but are allowed indefinite. The notion of closed-loop strategies is introduced, and the optimal closed-loop strategy is characterized by a Riccati integral-differential equation and a backward stochastic differential equation with Poisson jumps.
Long-Time Behaviors of Stochastic Linear-Quadratic Optimal Control Problems
University of Michigan（密歇根大学）；California State University San Marcos（圣马科斯加州州立大学）；Worcester Polytechnic Institute（伍斯特理工学院）；University of Central Florida（中佛罗里达大学）
This paper is concerned with a linear quadratic optimal control problem of delayed backward stochastic differential equations. An explicit representation is derived for the optimal control, which is a linear feedback of the entire past history and the future state trajectory in a short period of time. This is one of the major distinctive features of the delayed backward stochastic linear quadratic optimal control problem. To obtain the optimal feedback, a new class of delayed Riccati equations is introduced and the unique solvability of their solutions are discussed in detail.
In this paper, we define and solve the Inverse Stochastic Optimal Control (ISOC) problem of the linear-quadratic Gaussian (LQG) and the linear-quadratic sensorimotor (LQS) control model. These Stochastic Optimal Control (SOC) models are state-of-the-art approaches describing human movements. The LQG ISOC problem consists of finding the unknown weighting matrices of the quadratic cost function and the covariance matrices of the additive Gaussian noise processes based on ground truth trajectories observed from the human in practice. The LQS ISOC problem aims at additionally finding the covarianc…
An extended quadratic function is a quadratic function plus the indicator function of an affine set, that is, a quadratic function with embedded linear equality constraints. We show that, under some technical conditions, random convex extended quadratic functions are closed under addition, composition with an affine function, expectation, and partial minimization, that is, minimizing over some of its arguments. These properties imply that dynamic programming can be tractably carried out for stochastic control problems with random affine dynamics and extended quadratic cost functions. While the…
In this paper, we consider a class of stochastic control problems for stochastic differential equations with random coefficients. The control domain need not to be convex but the control process is not allowed to enter in diffusion term. Moreover, the terminal cost involves a non linear term of the expected value of terminal state. Our purpose is to derive a new version of the Pontryagin's stochastic maximum principle by adopting an idea inspired from the work of Peng [S. Peng, Maximum Principle for Stochastic Optimal Control with Nonconvex Control Domain, Lecture Notes in Control & Informatio…
This paper is concerned with a linear quadratic stochastic two-person zero-sum differential game with constant coefficients in an infinite time horizon. Open-loop and closed-loop saddle points are introduced. The existence of closed-loop saddle points is characterized by the solvability of an algebraic Riccati equation with a certain stabilizing condition. A crucial result makes our approach work is the unique solvability of a class of linear backward stochastic differential equations in an infinite horizon.
This paper investigates an infinite-horizon linear quadratic stochastic (LQS) optimal control problem for a class of continuous-time stochastic systems. By employing the technique of adaptive dynamic programming (ADP), we propose a novel model-free policy iteration (PI) algorithm. Without needing all information of the system coefficient matrices, the proposed PI algorithm iterates by using the data of the input and system state collected on a fixed time interval. Finally, a numerical example is presented to demonstrate the feasibility of the obtained algorithm.
In this paper, a leader-follower stochastic differential game is studied for a linear stochastic differential equation with a quadratic cost functional. The coefficients in the state equation and the weighting matrices in the cost functionals are all deterministic. Closed-loop strategies are introduced, which require to be independent of initial states; and such a nature makes it very useful and convenient in applications. The follower first solves a stochastic linear quadratic optimal control problem, and his optimal closed-loop strategy is characterized by a Riccati equation, together with a…
