This paper considers a formulation of a differential game with constrained dynamics, where one player selects the dynamics and the other selects the applicable cost. When the game is considered on a finite time horizon, its value satisfies an HJB equation with oblique Neumann boundary conditions. The first main result is uniqueness for viscosity solutions to this equation. This uniqueness is applied to obtain the second main result,i which is a unique characterization of the value function for a corresponding infinite time problem. The motivation comes from problems associated with queueing networks, where the games appear in several contexts, including a robust approach to network modeling and optimization and risk-sensitive control.
In this article, a notion of viscosity solutions is introduced for first order path-dependent Hamilton-Jacobi-Bellman (HJB) equations associated with optimal control problems for path-dependent differential equations. We identify the value functional of the optimal control problems as a unique viscosity solution to the associated HJB equations. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property.
In this article, the notion of viscosity solution is introduced for the path-dependent Hamilton-Jacobi-Bellman (PHJB) equations associated with the optimal control problems for path-dependent stochastic differential equations. We identify the value functional of the optimal control problems as unique viscosity solution to the associated PHJB equations. Applications to backward stochastic Hamilton-Jacobi-Bellman equations are also given.
The paper concerns the infinite dimensional Hamilton-Jacobi-Bellman equation related to optimal control problem regulated by a transport equation with boundary control. A suitable viscosity solution approach is needed in view of the presence of the unbounded control-related term in the Hilbertian state equation. An existence-and-uniqueness result is obtained.
Motivated by a control problem of a certain queueing network we consider a control problem where the dynamics is constrained in the nonnegative orthant $\mathbb{R}_+$ of the $d$-dimensional Euclidean space and controlled by the reflections at the faces/boundaries. We define a discounted value function associated to this problem and show that the value function is a viscosity solution to a certain HJB equation in $\mathbb{R}_+$ with nonlinear Neumann type boundary condition. Under certain conditions, we also characterize this value function as the unique solution to this HJB equation.
In this paper, we study the existence and uniqueness of viscosity solutions to a kind of Hamilton-Jacobi-Bellman (HJB) equations combined with algebra equations. This HJB equation is related to a stochastic optimal control problem for which the state equation is described by a fully coupled forward-backward stochastic differential equation. By extending Peng's backward semigroup approach to this problem, we obtain the dynamic programming principle and show that the value function is a viscosity solution to this HJB equation. As for the proof of the uniqueness of viscosity solution, the analysi…
In this article, a class of optimal control problems of differential equations with delays are investigated for which the associated Hamilton-Jacobi-Bellman (HJB) equations are nonlinear partial differential equations with delays. This type of HJB equation has not been previously studied and is difficult to solve because the state equations do not possess smoothing properties. We introduce a slightly different notion of viscosity solutions and identify the value function of the optimal control problems as a unique viscosity solution to the associated HJB equations.
In the present paper, we study a two-player zero-sum deterministic differential game with both players adopting impulse controls, in infinite time horizon, under rather weak assumptions on the cost functions. We prove by means of the dynamic programming principle (DPP) that the lower and upper value functions are continuous and viscosity solutions to the corresponding Hamilton-Jacobi-Bellman-Isaacs (HJBI) quasi-variational inequality (QVI). We define a new HJBI QVI for which, under a proportional property assumption on the maximizer cost, the value functions are the unique viscosity solution. …
For Hamilton-Jacobi-Bellman (HJB) equations, with the standard definitions of viscosity super-solution and sub-solution, it is known that there is a comparison between any (viscosity) super-solutions and sub-solutions. This should be the same for HJB type quasi-variational inequalities (QVIs) arising from optimal impulse control problems. However, according to a natural adoption of the definition found in Barles 1985, Barles 1985b, the uniqueness of the viscosity solution could be guaranteed, but the comparison between viscosity super- and sub-solutions could not be guaranteed. This paper intr…
This paper focuses on zero-sum stochastic differential games in the framework of forward-backward stochastic differential equations on a finite time horizon with both players adopting impulse controls. By means of BSDE methods, in particular that of the notion from Peng's stochastic \textit{% backward semigroups}, we prove a dynamic programming principle for both the upper and the lower value functions of the game. The upper and the lower value functions are then shown to be the unique viscosity solutions of the Hamilton-Jacobi-Bellman-Isaacs equations with a double-obstacle. As a consequence,…
This paper is devoted to a stochastic differential game of functional forward-backward stochastic differential equation (FBSDE, for short). The associated upper and lower value functions of the stochastic differential game are defined by controlled functional backward stochastic differential equations (BSDEs, for short). Applying the Girsanov transformation method introduced by Buckdahn and Li [1], the upper and the lower value functions are shown to be deterministic. We also generalize the Hamilton-Jacobi-Bellman-Isaacs (HJBI, for short) equations to the path-dependent ones. By establishing t…
Control problems not admitting the dynamic programming principle are known as time-inconsistent. The game-theoretic approach is to interpret such problems as intrapersonal dynamic games and look for subgame perfect Nash equilibria. A fundamental result of time-inconsistent stochastic control is a verification theorem saying that solving the extended HJB system is a sufficient condition for equilibrium. We show that solving the extended HJB system is a necessary condition for equilibrium, under regularity assumptions. The controlled process is a general Itô diffusion.
A general continuous mean-variance problem is considered for a diffusion controlled process where the reward functional has an integral and a terminal-time component. The problem is transformed into a superposition of a static and a dynamic optimization problem. The value function of the latter can be considered as the solution to a degenerate HJB equation either in viscosity or in Sobolev sense (after a regularization) under suitable assumptions and with implications with regards to the optimality of strategies. There is a useful interplay between the two approaches -- viscosity and Sobolev.
This paper introduces a notion of viscosity solutions for second order elliptic Hamilton-Jacobi-Bellman (HJB) equations with infinite delay associated with infinite-horizon optimal control problems for stochastic differential equations with infinite delay. We identify the value functional of optimal control problems as unique viscosity solution to associated second order elliptic HJB equation with infinite delay. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property.
In this paper we study the fully nonlinear stochastic Hamilton-Jacobi-Bellman (HJB) equation for the optimal stochastic control problem of stochastic differential equations with random coefficients. The notion of viscosity solution is introduced, and we prove that the value function of the optimal stochastic control problem is the maximal viscosity solution of the associated stochastic HJB equation. For the superparabolic cases when the diffusion coefficients are deterministic functions of time, states and controls, the uniqueness is addressed as well.
In this paper, we study a stochastic recursive optimal control problem in which the system is governed by a functional forward-backward stochastic differential equation. Under standard assumptions, we establish the dynamic programming principle and the related Path-dependent Hamilton-Jacobi-Bellman (HJB) equation in the framework of functional Itô calculus. The stochastic verification theorem for the smooth case is proved. Finally, we show that the value function is the viscosity solution of the Path-dependent HJB equation.
We study nonzero-sum stochastic games for continuous time Markov decision processes on a denumerable state space with risk-sensitive ergodic cost criterion. Transition rates and cost rates are allowed to be unbounded. Under a Lyapunov type stability assumption, we show that the corresponding system of coupled HJB equations admits a solution which leads to the existence of a Nash equilibrium in stationary strategies. We establish this using an approach involving principal eigenvalues associated with the HJB equations. Furthermore, exploiting appropriate stochastic representation of principal ei…
We consider stochastic impulse control problems when the impulses cost functions are arbitrary. We use the dynamic programming principle and viscosity solutions approach to show that the value function is a unique viscosity solution for the associated Hamilton-Jacobi-Bellman equation (HJB) partial differential equation (PDE) of stochastic impulse control problems
We study nonzero-sum stochastic games for continuous time Markov chains on a denumerable state space with risk sensitive discounted and ergodic cost criteria. For the discounted cost criterion we first show that the corresponding system of coupled HJB equations has an appropriate solution. Then under an additional additive structure on the transition rate matrix and payoff functions, we establish the existence of a Nash equilibrium in Markov strategies. For the ergodic cost criterion we assume a Lyapunov type stability assumption and a small cost condition. Under these assumptions we show that…
In this paper, we introduce Hamilton-Jacobi-Bellman (HJB) equations for Q-functions in continuous time optimal control problems with Lipschitz continuous controls. The standard Q-function used in reinforcement learning is shown to be the unique viscosity solution of the HJB equation. A necessary and sufficient condition for optimality is provided using the viscosity solution framework. By using the HJB equation, we develop a Q-learning method for continuous-time dynamical systems. A DQN-like algorithm is also proposed for high-dimensional state and control spaces. The performance of the propos…
In this paper, we consider the stochastic optimal control problem for jump diffusion systems with state constraints. In general, the value function of such problems is a discontinuous viscosity solution of the Hamilton-Jacobi-Bellman (HJB) equation, since the regularity cannot be guaranteed at the boundary of the state constraint. By adapting approaches of \cite{Bokanowski_SICON_2016} and the stochastic target theory, we obtain an equivalent representation of the original value function as the backward reachable set. We then show that this backward reachable can be characterized by the zero-le…
The Hamilton-Jacobi-Bellman equation (HJB) associated with the time inhomogeneous singular control problem is a parabolic partial differential equation, and the existence of a classical solution is usually difficult to prove. In this paper, a class of finite horizon stochastic singular control problems of one dimensional diffusion is solved via a time inhomogeneous zero-sum game (Dynkin game). The regularity of the value function of the Dynkin game is investigated, and its integrated form coincides with the value function of the singular control problem. We provide conditions under which a cla…
Gaussian process policy iteration with additive Schwarz acceleration for forward and inverse HJB and mean field game problems
Department of Computing and Mathematical Sciences, California Institute of Technology, CA, USA（计算与数学科学系，加州理工学院，CA，美国）；Department of Mathematics and Risk Management Institute, National University of Singapore, Singapore（数学与风险管理研究所，新加坡国立大学，新加坡）
In this paper we consider nonautonomous optimal control problems of infinite horizon type, whose control actions are given by $L^1$-functions. We verify that the value function is locally Lipschitz. The equivalence between dynamic programming inequalities and Hamilton-Jacobi-Bellman (HJB) inequalities for proximal sub (super) gradients is proven. Using this result we show that the value function is a Dini solution of the HJB equation. We obtain a verification result for the class of Dini sub-solutions of the HJB equation and also prove a minimax property of the value function with respect to t…
We study a class of optimal control problems with state constraints where the state equation is a differential equation with delays. This class includes some problems arising in economics, in particular the so-called models with time to build. We embed the problem in a suitable Hilbert space H and consider the associated Hamilton-Jacobi-Bellman (HJB) equation. This kind of infinite-dimensional HJB equation has not been previously studied and is difficult due to the presence of state constraints and the lack of smoothing properties of the state equation. Our main result on the regularity of sol…
This paper concerns the continuous time mean-variance portfolio selection problem with a special nonlinear wealth equation. This nonlinear wealth equation has a nonsmooth coefficient and the dual method developed in [6] does not work. We invoke the HJB equation of this problem and give an explicit viscosity solution of the HJB equation. Furthermore, via this explicit viscosity solution, we obtain explicitly the efficient portfolio strategy and efficient frontier for this problem. Finally, we show that our nonlinear wealth equation can cover three important cases.
In this paper, we study a stochastic recursive optimal control problem in which the objective functional is described by the solution of a backward stochastic differential equation driven by G-Brownian motion. Under standard assumptions, we establish the dynamic programming principle and the related Hamilton-Jacobi-Bellman (HJB) equation in the framework of G-expectation. Finally, we show that the value function is the viscosity solution of the obtained HJB equation.
The purpose of this paper is to describe the numerical solution of the Hamilton-Jacobi-Bellman (HJB) for an optimal control problem for quantum spin systems. This HJB equation is a first order nonlinear partial differential equation defined on a Lie group. We employ recent extensions of the theory of viscosity solutions from Euclidean space to Riemannian manifolds to interpret possibly non-differentiable solutions to this equation. Results from differential topology on the triangulation of manifolds are then used to develop a finite difference approximation method, which is shown to converge u…
Controlled Interacting Branching Diffusion Processes: A Viscosity Approach
Centre de Recherche en Économie et de Statistiques (CREST), Groupe ENSAE-ENSAI, ENSAE Paris, Institut Polytechnique de Paris（经济与统计研究中心（CREST），ENSAE-ENSAI小组，巴黎ENSAE，巴黎理工学院）
In this paper, we study a stochastic recursive optimal control problem in which the cost functional is described by the solution of a backward stochastic differential equation driven by G-Brownian motion. Under standard assumptions, we establish the dynamic programming principle and the related fully nonlinear HJB equation in the framework of G-expectation. Finally, we show that the value function is the viscosity solution of the obtained HJB equation.
In this paper we study stochastic optimal control problems of fully coupled forward-backward stochastic differential equations (FBSDEs). The recursive cost functionals are defined by controlled fully coupled FBSDEs. We study two cases of diffusion coefficients $σ$ of FSDEs. We use a new method to prove that the value functions are deterministic, satisfy the dynamic programming principle (DPP), and are viscosity solutions to the associated generalized Hamilton-Jacobi-Bellman (HJB) equations. The associated generalized HJB equations are related with algebraic equations when $σ$ depends on the se…
In this work we investigate regularity properties of a large class of Hamilton-Jacobi-Bellman (HJB) equations with or without obstacles, which can be stochastically interpreted in form of a stochastic control system which nonlinear cost functional is defined with the help of a backward stochastic differential equation (BSDE) or a reflected BSDE (RBSDE). More precisely, we prove that, firstly, the unique viscosity solution $V(t,x)$ of such a HJB equation over the time interval $[0,T],$ with or without an obstacle, and with terminal condition at time $T$, is jointly Lipschitz in $(t,x)$, for $t$…
We consider a utility maximization problem for an investment-consumption portfolio when the current utility depends also on the wealth process. Such kind of problems arise, e.g., in portfolio optimization with random horizon or with random trading times. To overcome the difficulties of the problem we use the dual approach. We define a dual problem and treat it by means of dynamic programming, showing that the viscosity solutions of the associated Hamilton-Jacobi-Bellman equation belong to a suitable class of smooth functions. This allows to define a smooth solution of the primal Hamilton-Jacob…
We study two person nonzero-sum stochastic differential games with risk-sensitive discounted and ergodic cost criteria. Under certain conditions we establish a Nash equilibrium in Markov strategies for the discounted cost criterion and a Nash equilibrium in stationary strategies for the ergodic cost criterion. We achieve our results by studying the relevant systems of coupled HJB equations.
In this paper infinite horizon optimal control problems for nonlinear high-dimensional dynamical systems are studied. Nonlinear feedback laws can be computed via the value function characterized as the unique viscosity solution to the corresponding Hamilton-Jacobi-Bellman (HJB) equation which stems from the dynamic programming approach. However, the bottleneck is mainly due to the curse of dimensionality and HJB equations are only solvable in a relatively small dimension. Therefore, a reduced-order model is derived for the dynamical system and for this purpose the method of proper orthogonal d…
This paper analyses a stochastic differential game of control and stopping in which one of the players modifies a diffusion process using impulse controls, an adversary then chooses a stopping time to end the game. The paper firstly establishes the regularity and boundedness of the upper and lower value functions from which an appropriate variant of the dynamic programming principle (DPP) is derived. It is then proven that the upper and lower value functions coincide so that the game admits a value and that the value of the game is a unique viscosity solution to a HJBI equation described by a …
Maximum entropy reinforcement learning (RL) methods have been successfully applied to a range of challenging sequential decision-making and control tasks. However, most of existing techniques are designed for discrete-time systems. As a first step toward their extension to continuous-time systems, this paper considers continuous-time deterministic optimal control problems with entropy regularization. Applying the dynamic programming principle, we derive a novel class of Hamilton-Jacobi-Bellman (HJB) equations and prove that the optimal value function of the maximum entropy control problem corr…
We extend the stochastic Perron method to analyze the framework of stochastic target games, in which one player tries to find a strategy such that the state process almost surely reaches a given target no matter which action is chosen by the other player. Within this framework, our method produces a viscosity sub-solution (super-solution) of a Hamilton-Jacobi-Bellman (HJB) equation. We then characterize the value function as a viscosity solution to the HJB equation using a comparison result and a byproduct to obtain the dynamic programming principle.
We apply the Stochastic Perron method, created by Bayraktar and Sîrbu, to a stochastic exit time control problem. Our main assumption is the validity of the Strong Comparison Result for the related Hamilton-Jacobi-Bellman (HJB) equation. Without relying on Bellman's optimality principle we prove that inside the domain the value function is continuous and coincides with a viscosity solution of the Dirichlet boundary value problem for the HJB equation.
In this paper, a stochastic optimal control problem is investigated in which the system is governed by a stochastic functional differential equation. In the framework of functional Itô calculus, we build the dynamic programming principle and the related Path-dependent Hamilton-Jacobi-Bellman (HJB) equation. We prove that the value function is the viscosity solution of the Path-dependent HJB equation.
In this paper, we study a time-inconsistent stochastic optimal control problem with a recursive cost functional by a multi-person hierarchical differential game approach. An equilibrium strategy of this problem is constructed and a corresponding equilibrium Hamilton-Jacobi-Bellman (HJB, for short) equation is established to characterize the associated equilibrium value function. Moreover, a well-posedness result of the equilibrium HJB equation is established under certain conditions.
