In this paper, we solve an open problem and obtain a general maximum principle for a stochastic optimal control problem where the control domain is an arbitrary non-empty set and all the coefficients (especially the diffusion term and the terminal cost) contain the control and state delay. In order to overcome the difficulty of dealing with the cross term of state and its delay in the variational inequality, we propose a new method: transform a delayed variational equation into a Volterra integral equation without delay, and introduce novel first-order, second-order adjoint equations via the backward stochastic Volterra integral equation theory. Finally we express these two kinds of adjoint equations in more compact anticipated backward stochastic differential equation types for several special yet typical control systems.
This paper deals with partially-observed optimal control problems for the state governed by stochastic differential equation with delay. We develop a stochastic maximum principle for this kind of optimal control problems using a variational method and a filtering technique. Also, we establish a sufficient condition without assumption of the concavity. Two examples shed light on the theoretical results are established in the paper. In particular, in the example of an optimal investment problem with delay, its numerical simulation shows the effect of delay via a discretization technique for forw…
We prove a maximum principle of optimal control of stochastic delay equations on infinite horizon. We establish first and second sufficient stochastic maximum principles as well as necessary conditions for that problem. We illustrate our results by an application to the optimal consumption rate from an economic quantity.
We solve a linear quadratic optimal control problem for sampled-data systems with stochastic delays. The delays are stochastically determined by the last few delays. The proposed optimal controller can be efficiently computed by iteratively solving a Riccati difference equation, provided that a discrete-time Markov jump system equivalent to the sampled-data system is stochastic stabilizable and detectable. Sufficient conditions for these notions are provided in the form of linear matrix inequalities, from which stabilizing controllers and state observers can be constructed.
We study methods for solving stochastic control problems of systems of forward-backward mean-field equations with delay, in finite or infinite horizon. Necessary and sufficient maximum principles under partial information are given. The results are applied to solve a recursive utility optimal problem
We consider a problem of optimal control of an infinite horizon system governed by forward-backward stochastic differential equations with delay. Sufficient and necessary maximum principles for optimal control under partial information in infinite horizon are derived. We illustrate our results by an application to a problem of optimal consumption with respect to recursive utility from a cash flow with delay.
This paper is concerned with a linear quadratic optimal control problem of delayed backward stochastic differential equations. An explicit representation is derived for the optimal control, which is a linear feedback of the entire past history and the future state trajectory in a short period of time. This is one of the major distinctive features of the delayed backward stochastic linear quadratic optimal control problem. To obtain the optimal feedback, a new class of delayed Riccati equations is introduced and the unique solvability of their solutions are discussed in detail.
In this paper, we first give the existence and uniqueness theorems for generalized mean-filed delay stochastic differential equations (GMFDSDEs) and mean-field anticipated backward stochastic differential equations (MFABSDEs). Then we study the stochastic maximum principle for generalized mean-filed delay control problem. Since the state is distribution-depending, we define the adjoint equation as a MFABSDE, in which, all the derivatives of coefficients are in Fréchet sense. We deduce the stochastic maximum principle, and also obtain, under some additional assumptions, a sufficient condition f…
This paper is concerned with a linear quadratic stochastic Stackelberg differential game with time delay. The model is general, in which the state delay and the control delay both appear in the state equation, moreover, they both enter into the diffusion term. By introducing two Pseudo-Riccati equations and a special matrix equation, the state feedback representation of the open-loop Stackelberg strategy is derived, under some assumptions. Finally, two examples are given to illustrate the applications of the theoretical results.
In this paper, we investigate the mean-square stabilization for discrete-time stochastic systems that endure both multiple input delays and multiplicative control-dependent noises. For such multi-delay stochastic systems, we for the first time put forward two stabilization criteria: Riccati type and Lyapunov type. On the one hand, we adopt a reduction method to reformulate the original multi-delay stochastic system to a delay-free auxiliary system and present their equivalent proposition for stabilization. Then, by introducing a delay-dependent algebraic Riccati equation (DDARE), we prove that…
In this paper, an open problem is solved, for the stochastic optimal control problem with delay where the control domain is nonconvex and the diffusion term contains both control and its delayed term. Inspired by previous results by Øksendal and Sulem [{\it A maximum principle for optimal control of stochastic systems with delay, with applications to finance. In J. M. Menaldi, E. Rofman, A. Sulem (Eds.), Optimal control and partial differential equations, ISO Press, Amsterdam, 64-79, 2000}] and Chen and Wu [{\it Maximum principle for the stochastic optimal control problem with delay and applic…
This paper investigates the linear-quadratic-Gaussian (LQG) mean-field game (MFG) for a class of stochastic delay systems. We consider a large population system in which the dynamics of each player satisfies some forward stochastic differential delay equation (SDDE). The consistency condition or Nash certainty equivalence (NCE) principle is established through an auxiliary mean-field system of anticipated forward-backward stochastic differential equation with delay (AFBSDDE). The wellposedness of such consistency condition system can be further established by some continuation method instead t…
In this paper, we consider optimal control problems derived by stochastic systems with delay, where control domains are non-convex and the diffusion coefficients depend on control variables. By an estimate of the integral of x_{1}(t)x_{1}(t-δ) term, we obtain a general maximum principle for the optimal control problems with a standard spike variational technique and duality method. The maximum principle is applied to study a delayed linear-quadratic optimal control problem with a non-convex control domain; an optimal solution is obtained.
This paper is concerned with the decoupling of delayed linear forward-backward stochastic differential equations (D-FBSDEs), which is much more involved than the delay-free case due to the infinite dimension caused by the delay. A new approach of `discretization' is proposed to obtain the explicit solution to the D-FBSDEs. Firstly, we transform the continuous-time D-FBSDEs into the discrete-time form by using discretization. Secondly, we derive the solution of the discrete-time D-FBSDEs by applying backward iterative induction. Finally the explicit solution of the continuous-time D-FBSDEs is o…
This paper is concerned with the stochastic recursive optimal control problem with mixed delay. The connection between Pontryagin's maximum principle and Bellman's dynamic programming principle is discussed. Without containing any derivatives of the value function, relations among the adjoint processes and the value function are investigated by employing the notions of super- and sub-jets introduced in defining the viscosity solutions. Stochastic verification theorem is also given to verify whether a given admissible control is really optimal.
In this paper we develop necessary conditions for optimality, in the form of the stochastic Pontryagin maximum principle, for controlled equations with pointwise delay in the state and with control dependent noise, in the general case of controls $u \in U$ with $U$ not necessarily convex. The maximum principle is formulated by means of first and second order adjoint BSDEs. We also outline how to deal with control problems with pointwise delay both in the state and in the control.
This paper is concerned with a non-zero sum differential game problem of an anticipated forward-backward stochastic differential delayed equation under partial information. We establish a necessary maximum principle and sufficient verification theorem of the game system by virtue of the duality and convex variational method. We apply the theoretical results and stochastic filtering theory to study a linear-quadratic game system and derive the explicit form of the Nash equilibrium point and discuss the existence and uniqueness in particular cases. As an application, we consider a time-delayed p…
The main contributions of this paper are three fold. First, our primary concern is to investigate a class of stochastic recursive delayed control problems which arise naturally with sound backgrounds but have not been well-studied yet. For illustration purpose, some concrete examples are also provided here. We derive the stochastic maximum principle of sufficient condition to the optimal control in both cases with and without control delay. Second, it is interesting that a new class of time-advanced stochastic differential equations (ASDEs) is introduced as the adjoint process via duality rela…
In this paper, we study the stabilization problem for the Ito systems with both multiplicative noise and multiple delays which exist widely in applications such as networked control systems. Sufficient and necessary conditions are obtained for the exponential stabilization problem of Ito stochastic systems with multiple delays. On one hand, we derive the solvability of the modified Riccati equation in case of the mean-square exponential stabilization. On the other hand, the mean-square exponential stabilization is guaranteed by the solvability of a modified Riccati equation. A novel stabilizin…
The purpose of this paper is to explore the necessary conditions for optimality of mean-field forward-backward delay control systems. A new estimate is proved, which is a powerfultool to deal with the optimal control problems of mean-field type with delay. Different from the classical situation, in our case the first-order adjoint system is an anticipated mean-field backward stochastic differential equation, and the second-order adjoint system is a system of matrix-valued process, not mean-field type.With the help of two adjoint systems, the second-order expansion of the variation of the state…
We consider a class of optimal control problems of stochastic delay differential equations (SDDE) that arise in connection with optimal advertising under uncertainty for the introduction of a new product to the market, generalizing classical work of Nerlove and Arrow (1962). In particular, we deal with controlled SDDE where the delay enters both the state and the control. Following ideas of Vinter and Kwong (1981) (which however hold only in the deterministic case), we reformulate the problem as an infinite dimensional stochastic control problem to which we associate, through the dynamic progr…
We investigate a stochastic optimal control problem where the controlled system is depicted as a stochastic differential delayed equation; however, at the terminal time, the state is constrained in a convex set. We firstly introduce an equivalent backward delayed system depicted as a time-delayed backward stochastic differential equation. Then a stochastic maximum principle is obtained by virtue of Ekeland's variational principle. Finally, applications to a state constrained stochastic delayed linear-quadratic control model and a production-consumption choice problem are studied to illustrate …
We study in this paper a weak approximation to stochastic variance reduced gradient Langevin dynamics by stochastic delay differential equations in Wasserstein-1 distance, and obtain a uniform error bound. Our approach is via a refined Lindeberg principle and Malliavin calculus.
Based on the classical probability, the stability criteria for stochastic differential delay equations (SDDEs) where their coefficients are either linear or nonlinear but bounded by linear functions have been investigated intensively. Moreover, the dependent stability of the highly nonlinear hybrid stochastic differential equations is recently studied. In this paper, by using the nonlinear expectation theory, we explore the dependent stability of a class of highly nonlinear hybrid stochastic differential delay equations driven by $G$-Brownian motion ($G$-SDDEs). Firstly, we give preliminaries …
In this paper, we prove both necessary and sufficient maximum principles for infinite horizon discounted control problems of stochastic Volterra integral equations with finite delay and a convex control domain. The corresponding adjoint equation is a novel class of infinite horizon anticipated backward stochastic Volterra integral equations. Our results can be applied to discounted control problems of stochastic delay differential equations and fractional stochastic delay differential equations. As an example, we consider a stochastic linear-quadratic regulator problem for a delayed fractional…
This work focuses on the well-posedness of abstract stochastic linear systems with boundary input delay and unbounded observation operators. We use product spaces and a semigroup approach to reformulate such delay systems into free-delay distributed stochastic systems with unbounded control and observation operators. This gives us the opportunity to use the concept of admissible control and observation operators as well as the concept of Yosida extensions to prove the existence and uniqueness of the solution process and provide an estimation of the observation process in relation to initial co…
In this article we give necessary and sufficient conditions providing regularity of solutions to stochastic Volterra equations with infinite delay on a $d$-dimensional torus. The harmonic analysis techniques and stochastic integration in function spaces are used. The work applies to both the stochastic heat and wave equations.
A novel approach to design the feedback control based on past states is proposed for hybrid stochastic differential equations (HSDEs). This new theorem builds up the connection between the delay feedback control and the control function without delay terms, which enables one to construct the delay feedback control using the existing results on stabilities of HSDEs. Methods to find the upper bound of the length of the time delay are also investigated. Numerical simulations are presented to demonstrate the new theorem.
We analyze a structure-preserving model order reduction technique for delay and stochastic delay equations based on the balanced truncation method and provide a system theoretic interpretation. Transferring error bounds based on Hankel operators to delay systems, we find error estimates for the difference between the dynamics of the full and reduced model. This analysis also yields new error bounds for bilinear systems and stochastic systems with multiplicative noise and non-zero initial states.
This paper is devoted to the stability analysis of an n species Lotka-Volterra system with discrete and distributed delays. Stochastic perturbations to the parameters of the model are allowed. Sufficient conditions for the almost sure global asymptotic stability of the positive equilibrium are derived in terms of LMIs. The efficiency of the proposed method is illustrated by numerical examples.
Google Research（谷歌研究院）；Hebrew University of Jerusalem（耶路撒冷希伯来大学）；Tel Aviv University（特拉维夫大学）
We consider stochastic optimization with delayed gradients where, at each time step $t$, the algorithm makes an update using a stale stochastic gradient from step $t - d_t$ for some arbitrary delay $d_t$. This setting abstracts asynchronous distributed optimization where a central server receives gradient updates computed by worker machines. These machines can experience computation and communication loads that might vary significantly over time. In the general non-convex smooth optimization setting, we give a simple and efficient algorithm that requires $O( σ^2/ε^4 + τ/ε^2 )$ steps for findin…
We analyze the convergence of gradient-based optimization algorithms that base their updates on delayed stochastic gradient information. The main application of our results is to the development of gradient-based distributed optimization algorithms where a master node performs parameter updates while worker nodes compute stochastic gradients based on local information in parallel, which may give rise to delays due to asynchrony. We take motivation from statistical problems where the size of the data is so large that it cannot fit on one computer; with the advent of huge datasets in biology, as…
Weizmann Institute of Science（魏茨曼科学研究所）；TTI Chicago（芝加哥丰田工业大学）
We provide tight finite-time convergence bounds for gradient descent and stochastic gradient descent on quadratic functions, when the gradients are delayed and reflect iterates from $τ$ rounds ago. First, we show that without stochastic noise, delays strongly affect the attainable optimization error: In fact, the error can be as bad as non-delayed gradient descent ran on only $1/τ$ of the gradients. In sharp contrast, we quantify how stochastic noise makes the effect of delays negligible, improving on previous work which only showed this phenomenon asymptotically or for much smaller delays. Al…
In this paper we develop necessary conditions for optimality, in the form of the Pontryagin maximum principle, for the optimal control problem of a class of infinite dimensional evolution equations with delay in the state. In the cost functional we allow the final cost to depend on the history of the state. To treat such kind of cost functionals we introduce a new form of anticipated backward stochastic differential equations which plays the role of dual equation associated to the control problem.
We consider a class of stochastic control problems with a delayed control, both in drift and diffusion, of the type dX t = $α$ t--d (bdt + $σ$dW t). We provide a new characterization of the solution in terms of a set of Riccati partial differential equations. Existence and uniqueness are obtained under a sufficient condition expressed directly as a relation between the horizon T and the quantity d(b/$σ$) 2. Furthermore, a deep learning scheme is designed and used to illustrate the effect of delay on the Markowitz portfolio allocation problem with execution delay.
Several real-world scenarios, such as remote control and sensing, are comprised of action and observation delays. The presence of delays degrades the performance of reinforcement learning (RL) algorithms, often to such an extent that algorithms fail to learn anything substantial. This paper formally describes the notion of Markov Decision Processes (MDPs) with stochastic delays and shows that delayed MDPs can be transformed into equivalent standard MDPs (without delays) with significantly simplified cost structure. We employ this equivalence to derive a model-free Delay-Resolved RL framework a…
We study the delay margin problem in the context of recent works by T. Qi, J. Zhu, and J. Chen, where a sufficient condition for the maximal delay margin is formulated in terms of an interpolation problem obtained after introducing a rational approximation. Instead we omit the approximation step and solve the same problem directly using techniques from function theory and analytic interpolation. Furthermore, we introduce a constant shift in the domain of the interpolation problem. In this way we are able to improve on their lower bound for the maximum delay margin.
Massachusetts Institute of Technology（麻省理工学院）；Carnegie Mellon University（卡内基梅隆大学）
We study distributed stochastic convex optimization under the delayed gradient model where the server nodes perform parameter updates, while the worker nodes compute stochastic gradients. We discuss, analyze, and experiment with a setup motivated by the behavior of real-world distributed computation networks, where the machines are differently slow at different time. Therefore, we allow the parameter updates to be sensitive to the actual delays experienced, rather than to worst-case bounds on the maximum delay. This sensitivity leads to larger stepsizes, that can help gain rapid initial conver…
Princeton University（普林斯顿大学）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
Stochastic control problems with delay are challenging due to the path-dependent feature of the system and thus its intrinsic high dimensions. In this paper, we propose and systematically study deep neural networks-based algorithms to solve stochastic control problems with delay features. Specifically, we employ neural networks for sequence modeling (\emph{e.g.}, recurrent neural networks such as long short-term memory) to parameterize the policy and optimize the objective function. The proposed algorithms are tested on three benchmark examples: a linear-quadratic problem, optimal consumption …
In this manuscript we consider a class optimal control problem for stochastic differential delay equations. First, we rewrite the problem in a suitable infinite-dimensional Hilbert space. Then, using the dynamic programming approach, we characterize the value function of the problem as the unique viscosity solution of the associated infinite-dimensional Hamilton-Jacobi-Bellman equation. Finally, we prove a $C^{1,α}$-partial regularity of the value function. We apply these results to path dependent financial and economic problems (Merton-like portfolio problem and optimal advertising).
