This paper investigates the optimal control problems for the finite-horizon continuous-time Markov decision processes with delay-dependent control policies. We develop compactification methods in decision processes, and show that the existence of optimal policies. Subsequently, through the dynamic programming principle of the delay-dependent control policies, the differential-difference Hamilton-Jacobi-Bellman (HJB) equation in the setting of discrete space was established. Under certain conditions, we give the comparison principle and further prove that the value function is the unique viscosity solution to this HJB equation. Based on this, we show that among the class of delay-dependent control policies, there is an optimal one which is Markovian.
It is known that Markovian forward-backward stochastic differential equations provide nonlinear Feynman-Kac representation formulae for semilinear parabolic PDEs. We show that non-Markovian forward-backward stochastic differential equations provide nonlinear Feynman-Kac formulae for semilinear path-dependent PDEs. This extends the result proved in Ekren, Keller, Touzi, and Zhang [4] to the case with a possibly degenerate diffusion coefficient in the forward dynamics.
In this paper we study a class of infinite horizon fully coupled forward-backward stochastic differential equations (FBSDEs), that are stimulated by various continuous time future expectations models with random coefficients. Under standard Lipschitz and monotonicity conditions, and by means of the contraction mapping principle, we establish existence, uniqueness, a comparison property and dependence on a parameter of adapted solutions. Making further the connection with infinite horizon quasilinear backward stochastic partial differential equations (BSPDEs) via a generalization of the well kn…
In this paper we aim to find the stationary stochastic viscosity solutions of a parabolic type SPDEs through the infinite horizon backward doubly stochastic differential equations (BDSDEs). For this, we study the existence, uniqueness and regularity of solutions of the corresponding infinite horizon BDSDEs as well as the "perfection procedure" applied to the solutions of BDSDEs. At last the "perfect" stationary stochastic viscosity solutions of SPDEs constructed by solutions of corresponding BDSDEs are obtained.
We consider stochastic impulse control problems when the impulses cost functions are arbitrary. We use the dynamic programming principle and viscosity solutions approach to show that the value function is a unique viscosity solution for the associated Hamilton-Jacobi-Bellman equation (HJB) partial differential equation (PDE) of stochastic impulse control problems
Path-dependent PDEs (PPDEs) are natural objects to study when one deals with non Markovian models. Recently, after the introduction of the so-called pathwise (or functional or Dupire) calculus (see [15]), in the case of finite-dimensional underlying space various papers have been devoted to studying the well-posedness of such kind of equations, both from the point of view of regular solutions (see e.g. [15, 9]) and viscosity solutions (see e.g. [16]). In this paper, motivated by the study of models driven by path-dependent stochastic PDEs, we give a first well-posedness result for viscosity so…
We establish the existence and uniqueness of pathwise strong solutions to the stochastic 3D primitive equations with only horizontal viscosity and diffusivity driven by transport noise on a cylindrical domain $M=(-h,0) \times G$, $G\subset \mathbb{R}^2$ bounded and smooth, with the physical Dirichlet boundary conditions on the lateral part of the boundary. Compared to the deterministic case where the uniqueness of $z$-weak solutions holds in $L^2$, more regular initial data are necessary to establish uniqueness in the anisotropic space $H^1_z L^2_{xy}$ so that the existence of local pathwise s…
We extend the notion of viscosity solutions for path-dependent PDEs introduced by Ekren et al. [Ann. Probab. 42 (2014), no. 1, 204-236] to path-dependent integro-differential equations and establish well-posedness, i.e., existence, uniqueness, and stability, for a class of semilinear path-dependent integro-differential equations with uniformly continuous data. Closely related are non-Markovian backward SDEs with jumps, which provide a probabilistic representation for solutions of our equations. The results are potentially useful for applications using non-Markovian jump-diffusion models.
In this paper we propose a notion of viscosity solutions for path dependent semi-linear parabolic PDEs. This can also be viewed as viscosity solutions of non-Markovian backward SDEs, and thus extends the well-known nonlinear Feynman-Kac formula to non-Markovian case. We shall prove the existence, uniqueness, stability and comparison principle for the viscosity solutions. The key ingredient of our approach is a functional Itô calculus recently introduced by Dupire [Functional Itô calculus (2009) Preprint].
We aim at characterizing viability, invariance and some reachability properties of controlled piecewise deterministic Markov processes (PDMPs). Using analytical methods from the theory of viscosity solutions, we establish criteria for viability and invariance in terms of the first order normal cone. We also investigate reachability of arbitrary open sets. The method is based on viscosity techniques and duality for some associated linearized problem. The theoretical results are applied to general On/Off systems, Cook's model for haploinssuficiency, and a stochastic model for bacteriophage lambd…
We study the Hamilton-Jacobi equation f - lambda Hf = h, where H f = e^{-f}Ae^f and where A is an operator that corresponds to a well-posed martingale problem. We identify an operator that gives viscosity solutions to the Hamilton-Jacobi equation, and which can therefore be interpreted as the resolvent of H. The operator is given in terms of optimization problem where the running cost is a path-space relative entropy. Finally, we use the resolvents to give a new proof of the abstract large deviation result of Feng and Kurtz.
In this paper, we consider the optimal stopping problem on semi-Markov processes (SMPs) with finite horizon, and aim to establish the existence and computation of optimal stopping times. To achieve the goal, we first develop the main results of finite horizon semi-Markov decision processes (SMDPs) to the case with additional terminal costs, introduce an explicit construction of SMDPs, and prove the equivalence between the optimal stopping problems on SMPs and SMDPs. Then, using the equivalence and the results on SMDPs developed here, we not only show the existence of optimal stopping time of S…
In this paper, we propose an extension of the forward improvement iteration algorithm, originally introduced in Irle (2006) and recently reconsidered in Miclo and Villeneuve (2021). The main new ingredient is a flexible window parameter describing the look-ahead distance in the improvement step. We consider the framework of a Markovian optimal stopping problem in discrete time with random discounting and infinite time horizon. We prove convergence and show that the additional flexibility may significantly reduce the runtime.
In this paper we study the optimal m-states switching problem in finite horizon as well as infinite horizon with risk of default. We allow the switching cost functionals and cost of default to be of polynomial growth and arbitrary. We show uniqueness of a solution for a system of m variational partial differential inequalities with inter-connected obstacles. This system is the deterministic version of the Verification Theorem of the Markovian optimal m-states switching problem with risk of default. This problem is connected with the valuation of a power plant in the energy market.
This paper introduces a notion of viscosity solutions for second order elliptic Hamilton-Jacobi-Bellman (HJB) equations with infinite delay associated with infinite-horizon optimal control problems for stochastic differential equations with infinite delay. We identify the value functional of optimal control problems as unique viscosity solution to associated second order elliptic HJB equation with infinite delay. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property.
We show stability and locality of the minimal supersolution of a forward backward stochastic differential equation with respect to the underlying forward process under weak assumptions on the generator. The forward process appears both in the generator and the terminal condition. Painlevé-Kuratowski and Convex Epi-convergence are used to establish the stability. For Markovian forward processes the minimal supersolution is shown to have the Markov property. Furthermore, it is related to a time-shifted problem and identified as the unique minimal viscosity supersolution of a corresponding PDE.
The present paper considers a stochastic optimal control problem, in which the cost function is defined through a backward stochastic differential equation with infinite horizon driven by G-Brownian motion. Then we study the regularities of the value function and establish the dynamic programming principle. Moreover, we prove that the value function is the uniqueness viscosity solution of the related HJBI equation.
In this paper, we consider the control problem with the Average-Value-at-Risk (AVaR) criteria of the possibly unbounded $L^{1}$-costs in infinite horizon on a Markov Decision Process (MDP). With a suitable state aggregation and by choosing a priori a global variable $s$ heuristically, we show that there exist optimal policies for the infinite horizon problem. To our knowledge, this is the first work of deriving dynamic programming equations with $L^1$-unbounded costs via AVaR-operator.
We study infinite horizon control of continuous-time non-linear branching processes with almost sure extinction for general (positive or negative) discount. Our main goal is to study the link between infinite horizon control of these processes and an optimization problem involving their quasi-stationary distributions and the corresponding extinction rates. More precisely, we obtain an equivalent of the value function when the discount parameter is close to the threshold where the value function becomes infinite, and we characterize the optimal Markov control in this limit. To achieve this, we …
Using probabilistic methods we study the existence of viscosity solutions to non-linear integro-differential equations $$\partial_t u(t,x) - \sup_{α\in I} \bigg( b_α(x) \cdot \nabla_x u(t,x) + \frac{1}{2} \text{tr}\left(Q_α(x) \cdot \nabla^2_x u(t,x)\right) +\int_{y \neq 0} \big(u(t,x+y)-u(t,x)-\nabla_x u(t,x) \cdot h(y) \big) \, ν_α(x,dy) \bigg) = 0$$ with initial condition $u(0,x)= φ(x)$; here $(b_α(x),Q_α(x),ν_α(x,dy))$, $α\in I$, $x \in \mathbb{R}^d$, is a family of Lévy triplets and $h$ is some truncation function. The solutions, which we construct, are of the form $u(t,x) = T_t φ(x)$ for…
We study optimal control problems in infinite horizon when the dynamics belong to a specific class of piecewise deterministic Markov processes constrained to star-shaped networks (inspired by traffic models). We adapt the results in [H. M. Soner. Optimal control with state-space constraint. II. SIAM J. Control Optim., 24(6):1110.1122, 1986] to prove the regularity of the value function and the dynamic programming principle. Extending the networks and Krylov's ''shaking the coefficients'' method, we prove that the value function can be seen as the solution to a linearized optimization problem s…
In this paper, we consider the (upper) semigroup envelope, i.e. the least upper bound, of a given family of linear Feller semigroups. We explicitly construct the semigroup envelope and show that, under suitable assumptions, it yields viscosity solutions to abstract Hamilton-Jacobi-Bellman-type partial differential equations related to stochastic optimal control problems arising in the field of Robust Finance. We further derive conditions for the existence of a Markov process under a nonlinear expectation related to the semigroup envelope for the case where the state space is locally compact. T…
Stochastic optimal control problems with delays in the state and in the control via viscosity solutions and applications to optimal advertising and optimal investment problems
The non-exponential Schilder-type theorem in Backhoff-Veraguas, Lacker and Tangpi [Ann. Appl. Probab., 30 (2020), pp. 1321-1367] is expressed as a convergence result for path-dependent partial differential equations with appropriate notions of generalized solutions. This entails a non-Markovian counterpart to the vanishing viscosity method. We show uniqueness of maximal subsolutions for path-dependent viscous Hamilton-Jacobi equations related to convex super-quadratic backward stochastic differential equations. We establish well-posedness for the Hamilton-Jacobi-Bellman equation associated to …
