The paper concerns the infinite dimensional Hamilton-Jacobi-Bellman equation related to optimal control problem regulated by a transport equation with boundary control. A suitable viscosity solution approach is needed in view of the presence of the unbounded control-related term in the Hilbertian state equation. An existence-and-uniqueness result is obtained.
In this article, the notion of viscosity solution is introduced for the path-dependent Hamilton-Jacobi-Bellman (PHJB) equations associated with the optimal control problems for path-dependent stochastic differential equations. We identify the value functional of the optimal control problems as unique viscosity solution to the associated PHJB equations. Applications to backward stochastic Hamilton-Jacobi-Bellman equations are also given.
In this article, a notion of viscosity solutions is introduced for first order path-dependent Hamilton-Jacobi-Bellman (HJB) equations associated with optimal control problems for path-dependent differential equations. We identify the value functional of the optimal control problems as a unique viscosity solution to the associated HJB equations. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property.
In this paper, we study the existence and uniqueness of viscosity solutions to a kind of Hamilton-Jacobi-Bellman (HJB) equations combined with algebra equations. This HJB equation is related to a stochastic optimal control problem for which the state equation is described by a fully coupled forward-backward stochastic differential equation. By extending Peng's backward semigroup approach to this problem, we obtain the dynamic programming principle and show that the value function is a viscosity solution to this HJB equation. As for the proof of the uniqueness of viscosity solution, the analysi…
In this article, a class of optimal control problems of differential equations with delays are investigated for which the associated Hamilton-Jacobi-Bellman (HJB) equations are nonlinear partial differential equations with delays. This type of HJB equation has not been previously studied and is difficult to solve because the state equations do not possess smoothing properties. We introduce a slightly different notion of viscosity solutions and identify the value function of the optimal control problems as a unique viscosity solution to the associated HJB equations.
In the present paper, we study a two-player zero-sum deterministic differential game with both players adopting impulse controls, in infinite time horizon, under rather weak assumptions on the cost functions. We prove by means of the dynamic programming principle (DPP) that the lower and upper value functions are continuous and viscosity solutions to the corresponding Hamilton-Jacobi-Bellman-Isaacs (HJBI) quasi-variational inequality (QVI). We define a new HJBI QVI for which, under a proportional property assumption on the maximizer cost, the value functions are the unique viscosity solution. …
This paper considers a formulation of a differential game with constrained dynamics, where one player selects the dynamics and the other selects the applicable cost. When the game is considered on a finite time horizon, its value satisfies an HJB equation with oblique Neumann boundary conditions. The first main result is uniqueness for viscosity solutions to this equation. This uniqueness is applied to obtain the second main result,i which is a unique characterization of the value function for a corresponding infinite time problem. The motivation comes from problems associated with queueing ne…
Motivated by a control problem of a certain queueing network we consider a control problem where the dynamics is constrained in the nonnegative orthant $\mathbb{R}_+$ of the $d$-dimensional Euclidean space and controlled by the reflections at the faces/boundaries. We define a discounted value function associated to this problem and show that the value function is a viscosity solution to a certain HJB equation in $\mathbb{R}_+$ with nonlinear Neumann type boundary condition. Under certain conditions, we also characterize this value function as the unique solution to this HJB equation.
For Hamilton-Jacobi-Bellman (HJB) equations, with the standard definitions of viscosity super-solution and sub-solution, it is known that there is a comparison between any (viscosity) super-solutions and sub-solutions. This should be the same for HJB type quasi-variational inequalities (QVIs) arising from optimal impulse control problems. However, according to a natural adoption of the definition found in Barles 1985, Barles 1985b, the uniqueness of the viscosity solution could be guaranteed, but the comparison between viscosity super- and sub-solutions could not be guaranteed. This paper intr…
We investigate feedback control for infinite horizon optimal control problems for partial differential equations. The method is based on the coupling between Hamilton-Jacobi-Bellman (HJB) equations and model reduction techniques. It is well-known that HJB equations suffer the so called curse of dimensionality and, therefore, a reduction of the dimension of the system is mandatory. In this report we focus on the infinite horizon optimal control problem with quadratic cost functionals. We compare several model reduction methods such as Proper Orthogonal Decomposition, Balanced Truncation and a n…
General theorems for existence and uniqueness of viscosity solutions for Hamilton-Jacobi-Bellman quasi-variational inequalities (HJBQVI) with integral term are established. Such nonlinear partial integro-differential equations (PIDE) arise in the study of combined impulse and stochastic control for jump-diffusion processes. The HJBQVI consists of an HJB part (for stochastic control) combined with a nonlocal impulse intervention term. Existence results are proved via stochastic means, whereas our uniqueness (comparison) results adapt techniques from viscosity solution theory. This paper is to o…
A general continuous mean-variance problem is considered for a diffusion controlled process where the reward functional has an integral and a terminal-time component. The problem is transformed into a superposition of a static and a dynamic optimization problem. The value function of the latter can be considered as the solution to a degenerate HJB equation either in viscosity or in Sobolev sense (after a regularization) under suitable assumptions and with implications with regards to the optimality of strategies. There is a useful interplay between the two approaches -- viscosity and Sobolev.
This paper introduces a notion of viscosity solutions for second order elliptic Hamilton-Jacobi-Bellman (HJB) equations with infinite delay associated with infinite-horizon optimal control problems for stochastic differential equations with infinite delay. We identify the value functional of optimal control problems as unique viscosity solution to associated second order elliptic HJB equation with infinite delay. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property.
In this paper we consider nonautonomous optimal control problems of infinite horizon type, whose control actions are given by $L^1$-functions. We verify that the value function is locally Lipschitz. The equivalence between dynamic programming inequalities and Hamilton-Jacobi-Bellman (HJB) inequalities for proximal sub (super) gradients is proven. Using this result we show that the value function is a Dini solution of the HJB equation. We obtain a verification result for the class of Dini sub-solutions of the HJB equation and also prove a minimax property of the value function with respect to t…
In this paper we study the fully nonlinear stochastic Hamilton-Jacobi-Bellman (HJB) equation for the optimal stochastic control problem of stochastic differential equations with random coefficients. The notion of viscosity solution is introduced, and we prove that the value function of the optimal stochastic control problem is the maximal viscosity solution of the associated stochastic HJB equation. For the superparabolic cases when the diffusion coefficients are deterministic functions of time, states and controls, the uniqueness is addressed as well.
IIn this paper we show that some HJB equations arising from both finite and infinite horizon stochastic optimal control problems have a regular singular point at the origin. This makes them amenable to solution by power series techniques. This extends the work of Al'brecht who showed that the HJB equations of an infinite horizon deterministic optimal control problem can have a regular singular point at the origin, Al'brekht solved the HJB equations by power series, degree by degree. In particular, we show that the infinite horizon stochastic optimal control problem with linear dynamics, quadra…
This paper focuses on zero-sum stochastic differential games in the framework of forward-backward stochastic differential equations on a finite time horizon with both players adopting impulse controls. By means of BSDE methods, in particular that of the notion from Peng's stochastic \textit{% backward semigroups}, we prove a dynamic programming principle for both the upper and the lower value functions of the game. The upper and the lower value functions are then shown to be the unique viscosity solutions of the Hamilton-Jacobi-Bellman-Isaacs equations with a double-obstacle. As a consequence,…
In this paper, we consider the stochastic optimal control problem for jump diffusion systems with state constraints. In general, the value function of such problems is a discontinuous viscosity solution of the Hamilton-Jacobi-Bellman (HJB) equation, since the regularity cannot be guaranteed at the boundary of the state constraint. By adapting approaches of \cite{Bokanowski_SICON_2016} and the stochastic target theory, we obtain an equivalent representation of the original value function as the backward reachable set. We then show that this backward reachable can be characterized by the zero-le…
In this paper, we study a stochastic recursive optimal control problem in which the system is governed by a functional forward-backward stochastic differential equation. Under standard assumptions, we establish the dynamic programming principle and the related Path-dependent Hamilton-Jacobi-Bellman (HJB) equation in the framework of functional Itô calculus. The stochastic verification theorem for the smooth case is proved. Finally, we show that the value function is the viscosity solution of the Path-dependent HJB equation.
We study a class of optimal control problems with state constraints where the state equation is a differential equation with delays. This class includes some problems arising in economics, in particular the so-called models with time to build. We embed the problem in a suitable Hilbert space H and consider the associated Hamilton-Jacobi-Bellman (HJB) equation. This kind of infinite-dimensional HJB equation has not been previously studied and is difficult due to the presence of state constraints and the lack of smoothing properties of the state equation. Our main result on the regularity of sol…
In this paper, we study a stochastic recursive optimal control problem in which the objective functional is described by the solution of a backward stochastic differential equation driven by G-Brownian motion. Under standard assumptions, we establish the dynamic programming principle and the related Hamilton-Jacobi-Bellman (HJB) equation in the framework of G-expectation. Finally, we show that the value function is the viscosity solution of the obtained HJB equation.
In this paper infinite horizon optimal control problems for nonlinear high-dimensional dynamical systems are studied. Nonlinear feedback laws can be computed via the value function characterized as the unique viscosity solution to the corresponding Hamilton-Jacobi-Bellman (HJB) equation which stems from the dynamic programming approach. However, the bottleneck is mainly due to the curse of dimensionality and HJB equations are only solvable in a relatively small dimension. Therefore, a reduced-order model is derived for the dynamical system and for this purpose the method of proper orthogonal d…
In this paper, we study a stochastic recursive optimal control problem in which the cost functional is described by the solution of a backward stochastic differential equation driven by G-Brownian motion. Under standard assumptions, we establish the dynamic programming principle and the related fully nonlinear HJB equation in the framework of G-expectation. Finally, we show that the value function is the viscosity solution of the obtained HJB equation.
We consider stochastic impulse control problems when the impulses cost functions are arbitrary. We use the dynamic programming principle and viscosity solutions approach to show that the value function is a unique viscosity solution for the associated Hamilton-Jacobi-Bellman equation (HJB) partial differential equation (PDE) of stochastic impulse control problems
This paper analyses a stochastic differential game of control and stopping in which one of the players modifies a diffusion process using impulse controls, an adversary then chooses a stopping time to end the game. The paper firstly establishes the regularity and boundedness of the upper and lower value functions from which an appropriate variant of the dynamic programming principle (DPP) is derived. It is then proven that the upper and lower value functions coincide so that the game admits a value and that the value of the game is a unique viscosity solution to a HJBI equation described by a …
Path Integral Control methods were developed for stochastic optimal control covering a wide class of finite horizon formulations with control affine nonlinear dynamics. Characteristic for this class is that the HJB equation is linear and consequently the value function can be expressed as a conditional expectation of the exponentially weighted cost-to-go evaluated over trajectories with uncontrolled system dynamics, hence the name. Subsequently it was shown that under the same assumptions Path Integral Control generalises to finite horizon risk sensitive stochastic optimal control problems. He…
This paper is devoted to a stochastic differential game of functional forward-backward stochastic differential equation (FBSDE, for short). The associated upper and lower value functions of the stochastic differential game are defined by controlled functional backward stochastic differential equations (BSDEs, for short). Applying the Girsanov transformation method introduced by Buckdahn and Li [1], the upper and the lower value functions are shown to be deterministic. We also generalize the Hamilton-Jacobi-Bellman-Isaacs (HJBI, for short) equations to the path-dependent ones. By establishing t…
In this paper, we introduce Hamilton-Jacobi-Bellman (HJB) equations for Q-functions in continuous time optimal control problems with Lipschitz continuous controls. The standard Q-function used in reinforcement learning is shown to be the unique viscosity solution of the HJB equation. A necessary and sufficient condition for optimality is provided using the viscosity solution framework. By using the HJB equation, we develop a Q-learning method for continuous-time dynamical systems. A DQN-like algorithm is also proposed for high-dimensional state and control spaces. The performance of the propos…
The purpose of this paper is to describe the numerical solution of the Hamilton-Jacobi-Bellman (HJB) for an optimal control problem for quantum spin systems. This HJB equation is a first order nonlinear partial differential equation defined on a Lie group. We employ recent extensions of the theory of viscosity solutions from Euclidean space to Riemannian manifolds to interpret possibly non-differentiable solutions to this equation. Results from differential topology on the triangulation of manifolds are then used to develop a finite difference approximation method, which is shown to converge u…
We study an optimal stopping problem when the state process is governed by a general Feller process. In particular, we examine viscosity properties of the associated value function with no a priori assumption on the stochastic differential equation satisfied by the state process. Our approach relies on properties of the Feller semigroup. We present conditions on the state process under which the value function is the unique viscosity solution to an Hamilton-Jacobi-Bellman (HJB) equation associated with a particular operator. More specifically, assuming that the state process is a Feller proces…
This paper concerns the continuous time mean-variance portfolio selection problem with a special nonlinear wealth equation. This nonlinear wealth equation has a nonsmooth coefficient and the dual method developed in [6] does not work. We invoke the HJB equation of this problem and give an explicit viscosity solution of the HJB equation. Furthermore, via this explicit viscosity solution, we obtain explicitly the efficient portfolio strategy and efficient frontier for this problem. Finally, we show that our nonlinear wealth equation can cover three important cases.
In this paper we study stochastic optimal control problems of fully coupled forward-backward stochastic differential equations (FBSDEs). The recursive cost functionals are defined by controlled fully coupled FBSDEs. We study two cases of diffusion coefficients $σ$ of FSDEs. We use a new method to prove that the value functions are deterministic, satisfy the dynamic programming principle (DPP), and are viscosity solutions to the associated generalized Hamilton-Jacobi-Bellman (HJB) equations. The associated generalized HJB equations are related with algebraic equations when $σ$ depends on the se…
We consider a utility maximization problem for an investment-consumption portfolio when the current utility depends also on the wealth process. Such kind of problems arise, e.g., in portfolio optimization with random horizon or with random trading times. To overcome the difficulties of the problem we use the dual approach. We define a dual problem and treat it by means of dynamic programming, showing that the viscosity solutions of the associated Hamilton-Jacobi-Bellman equation belong to a suitable class of smooth functions. This allows to define a smooth solution of the primal Hamilton-Jacob…
We apply the Stochastic Perron method, created by Bayraktar and Sîrbu, to a stochastic exit time control problem. Our main assumption is the validity of the Strong Comparison Result for the related Hamilton-Jacobi-Bellman (HJB) equation. Without relying on Bellman's optimality principle we prove that inside the domain the value function is continuous and coincides with a viscosity solution of the Dirichlet boundary value problem for the HJB equation.
In this paper, a stochastic optimal control problem is investigated in which the system is governed by a stochastic functional differential equation. In the framework of functional Itô calculus, we build the dynamic programming principle and the related Path-dependent Hamilton-Jacobi-Bellman (HJB) equation. We prove that the value function is the viscosity solution of the Path-dependent HJB equation.
In this work we investigate regularity properties of a large class of Hamilton-Jacobi-Bellman (HJB) equations with or without obstacles, which can be stochastically interpreted in form of a stochastic control system which nonlinear cost functional is defined with the help of a backward stochastic differential equation (BSDE) or a reflected BSDE (RBSDE). More precisely, we prove that, firstly, the unique viscosity solution $V(t,x)$ of such a HJB equation over the time interval $[0,T],$ with or without an obstacle, and with terminal condition at time $T$, is jointly Lipschitz in $(t,x)$, for $t$…
This paper, which is the natural continuation of a previous paper by the same authors, studies a class of optimal control problems with state constraints where the state equation is a differential equation with delays. This class includes some problems arising in economics, in particular the so-called models with time to build. The problem is embedded in a suitable Hilbert space H and the regularity of the associated Hamilton-Jacobi-Bellman (HJB) equation is studied. Therein the main result is that the value function V solves the HJB equation and has continuous classical derivative in the dire…
