The paper deals with a zero-sum differential game for a dynamical system which motion is described by a nonlinear delay differential equation under an initial condition defined by a piecewise continuous function. The corresponding Cauchy problem for Hamilton-Jacobi-Bellman-Isaacs equation with coinvariant derivatives is derived and the definition of a viscosity solution of this problem is considered. It is proved that the differential game has the value that is the unique viscosity solution. Moreover, based on notions of sub- and superdifferentials corresponding to coinvariant derivatives, the infinitesimal description of the viscosity solution is obtained. The example of applying these results is given.
In this paper we study a two person zero sum stochastic differential game in weak formulation. Unlike standard literature which uses strategy type of controls, the weak formulation allows us to consider the game with control against control. We shall prove the existence of game value under natural conditions. Another main feature of the paper is that we allow for non-Markovian structure, and thus the game value is a random process. We characterize the value process as the unique viscosity solution of the corresponding path dependent Bellman-Isaacs equation, a notion recently introduced by Ekre…
We introduce some approximation schemes for linear and fully non-linear diffusion equations of Bellman-Isaacs type. Although they are not monotone one can prove their convergence to the viscosity solution of the problem. Effective implementation of these scheme is discussed and they are extensively tested.
Policy iteration for Hamilton-Jacobi-Isaacs equations with control constraints and comparison with Hamilton-Jacobi-Bellman equations
Rajiv Gandhi Institute of Petroleum Technology（拉吉夫·甘地石油技术学院）；University of Graz（格拉茨大学）；Radon Institute for Computational and Applied Mathematics (RICAM)（雷登计算与应用数学研究所）
A new stochastic control model for the long-run environmental management of rivers is mathematically and numerically analyzed, focusing on a modern sediment replenishment problem with unique nonsmooth and nonlinear properties. Rational inattention as a novel adaptive strategy to collect information and intervene against the target system is modeled using Erlangization. The system dynamics containing the river discharge following a continuous-state branching with an immigration-type process and the controlled sediment storage dynamics lead to a nonsmooth and nonlocal infinitesimal generator. Mo…
This paper derives the Hamilton-Jacobi-Bellman equation of nonlinear optimal control problems for cost functions with fractional discount rate from the Bellman's principle of optimality. The fractional discount rate is described by Mittag-Leffler function that can be considered as a generalized exponential function.
In this paper we study controlled continuous time random walks (CTRWs) and heuristically derive pay-off function dynamic programming (DP) equations which turn in the limit of standard scaling to fractional Hamilton Jacobi Bellman type equations. This paper aims to extend results from [1] in a controlled setting.
For a non-cooperative m-persons differential game, the value functions ofthe various players satisfy a system of Hamilton-Jacobi-Bellman equations.Nashequilibrium solutions in feedback form can be obtained by studying a related system of P.D.E's.A new approach, which is proposed in this paper allows one to construct the feedback optimal control and cost functions J_i(t,x),i=1,...,m directly,i.e.,without any reference to the corresponding Hamilton-Jacobi-Bellman equations.
We extend the theory of Barles Jakobsen to develop numerical schemes for Hamilton Jacobi Bellman equations. We show that the monotonicity of the schemes can be relaxed still leading to the convergence to the viscosity solution of the equation. We give some examples of such numerical schemes and show that the bounds obtained by the framework developed are not tight. At last we test some numerical schemes.
We consider a two-player zero-sum deterministic differential game where each player uses both continuous and impulse controls in infinite-time horizon. We assume that the impulses supposed to be of general term and the costs depend on the state of the system. We use the dynamic programming principle and viscosity solutions approach to show existence and uniqueness of a solution for the Hamilton-Jacobi-Bellman-Isaacs (HJBI) partial differential equations (PDEs) of the game. We prove under Isaacs condition that the upper and lower value functions coincide.
We study an optimal control problem in Bolza form and we consider the value function associated to this problem. We prove two verification theorems which ensure that, if a function $W$ satisfies some suitable weak continuity assumptions and a Hamilton-Jacobi-Bellman inequality outside a countably $\mathcal H^n$-rectifiable set, then it is lower or equal to the value function. These results can be used for optimal synthesis approach.
In this paper, we consider state and control path-dependent stochastic zero-sum differential games, where the dynamics and the running cost include both state and control paths of the players. Using the notion of nonanticipative strategies, we define lower and upper value functionals, which are functions of the initial state and control paths of the players. We prove that the value functionals satisfy the dynamic programming principle. The associated lower and upper Hamilton-Jacobi-Isaacs (HJI) equations from the dynamic programming principle are state and control path-dependent nonlinear seco…
We introduce some sparse grids interpolations used in Semi-Lagrangian schemes for linear and fully non-linear diffusion Hamilton Jacobi Bellman equations arising in stochastic control. We prove that the method introduced converges toward the viscosity solution of the problem and we show that some potentially high order schemes can be efficiently implemented. Numerical test in dimension $2$ to $5$ are achieved and show that deterministic methods can be used efficiently in stochastic control in moderate dimension.
In this paper, we study the optimal singular controls for stochastic recursive systems, in which the control has two components: the regular control, and the singular control. Under certain assumptions, we establish the dynamic programming principle for this kind of optimal singular controls problem, and prove that the value function is a unique viscosity solution of the corresponding Hamilton-Jacobi-Bellman inequality, in a given class of bounded and continuous functions. At last, an example is given for illustration.
In the present paper, it is provided a representation result for the weak solutions of a class of evolutionary Hamilton-Jacobi-Bellman equations on infinite horizon, with Hamiltonians measurable in time and fiber convex. Such Hamiltonians are associated with a - faithful - representation, namely involving two functions measurable in time and locally Lipschitz in the state and control. Our results concern the recovering of a representation of convex Hamiltonians under a relaxed assumption on the Fenchel transform of the Hamiltonian with respect to the fiber. We apply them to investigate the uni…
This article is devoted to the study of lower semicontinuous solutions of Hamilton-Jacobi equations with convex Hamiltonians in a gradient variable. Such Hamiltonians appear in the optimal control theory. We present a necessary and sufficient condition for a reduction of a Hamiltonian satisfying optimality conditions to the case when the Hamiltonian is positively homogeneous and also satisfies optimality conditions. It allows us to reduce some uniqueness problems of lower semicontinuous solutions to Barron-Jensen and Frankowska theorems. For Hamiltonians, which cannot be reduced in that way, w…
In this article, a class of optimal control problems of differential equations with delays are investigated for which the associated Hamilton-Jacobi-Bellman (HJB) equations are nonlinear partial differential equations with delays. This type of HJB equation has not been previously studied and is difficult to solve because the state equations do not possess smoothing properties. We introduce a slightly different notion of viscosity solutions and identify the value function of the optimal control problems as a unique viscosity solution to the associated HJB equations.
The paper deals with a Bolza optimal control problem for a dynamical system which motion is described by a delay differential equation under an initial condition defined by a piecewise continuous function. For the value functional in this problem, the Cauchy problem for the Hamilton-Jacobi-Bellman equation with coinvariant derivatives is considered. Minimax and viscosity solutions of this problem are studied. It is proved that both of these solutions exist, are unique and coincide with the value functional.
An example of a nonunique solution of the Cauchy problem of Hamilton-Jacobi-Bellman (HJB) equation with surprisingly regular Hamiltonian is presented. The Hamiltonian H(t,x,p) is locally Lipschitz continuous with respect to all variables, convex in p and with linear growth with respect to p and x. The HJB equation possesses two distinct lower semicontinuous solutions with the same final conditions; moreover, one of them is the value function of the corresponding Bolza problem. The definition of lower semicontinuous solution was proposed by Barron-Jensen (1990) and Frankowska (1993). Using the …
This paper is concerned with the stochastic Hamilton-Jacobi-Bellman equation with controlled leading coefficients, which is a type of fully nonlinear backward stochastic partial differential equation (BSPDE for short). In order to formulate the weak solution for such kind of BSPDEs, the classical potential theory is generalized in the backward stochastic framework. The existence and uniqueness of the weak solution is proved, and for the partially non-Markovian case, we obtain the associated gradient estimate. As a byproduct, the existence and uniqueness of solution for a class of degenerate re…
We present a method for determining optimal walking paths in steep terrain using the level set method and an optimal control formulation. By viewing the walking direction as a control variable, we can determine the optimal control by solving a Hamilton-Jacobi-Bellman equation. We then calculate the optimal walking path by solving an ordinary differential equation. We demonstrate the effectiveness of our method by computing optimal paths which travel throughout mountainous regions of Yosemite National Park. We include details regarding the numerical implementation of our model and address a spe…
We present a novel method to approximate optimal feedback laws for nonlinear optimal control based on low-rank tensor train (TT) decompositions. The approach is based on the Dirac-Frenkel variational principle with the modification that the optimisation uses an empirical risk. Compared to current state-of-the-art TT methods, our approach exhibits a greatly reduced computational burden while achieving comparable results. A rigorous description of the numerical scheme and demonstrations of its performance are provided.
In this paper, we study a stochastic recursive optimal control problem in which the value functional is defined by the solution of a backward stochastic differential equation (BSDE) under $\tilde{G}$-expectation. Under standard assumptions, we establish the comparison theorem for this kind of BSDE and give a novel and simple method to obtain the dynamic programming principle. Finally, we prove that the value function is the unique viscosity solution of a type of fully nonlinear HJB equation.
In this work we study the stochastic recursive control problem, in which the aggregator (or called generator) of the backward stochastic differential equation describing the running cost is continuous but not necessarily Lipschitz with respect to the first unknown variable and the control, and monotonic with respect to the first unknown variable. The dynamic programming principle and the connection between the value function and the viscosity solution of the associated Hamilton-Jacobi-Bellman equation are established in this setting by the generalized comparison theorem of backward stochastic …
The value of a zero-sum differential games is known to exist, under Isaacs' condition, as the unique viscosity solution of a Hamilton-Jacobi-Bellman equation. In this note we provide a self-contained proof based on the construction of $\ep$-optimal strategies, which is inspired by the "extremal aiming" method from Krasovskii and Subbotin.
We show that necessary and sufficient conditions of optimality in periodic optimization problems can be stated in terms of a solution of the corresponding HJB inequality, the latter being equivalent to a max-min type variational problem considered on the space of continuously differentiable functions. We approximate the latter with a maximin problem on a finite dimensional subspace of the space of continuously differentiable functions and show that a solution of this problem (existing under natural controllability conditions) can be used for construction of near optimal controls. We illustrate…
This article is devoted to the optimal control of state equations with memory of the form: ?[x(t) = F(x(t),u(t), \int_0^{+\infty} A(s) x(t-s) ds), t>0, with initial conditions x(0)=x, x(-s)=z(s), s>0.]Denoting by $y_{x,z,u}$ the solution of the previous Cauchy problem and: \[v(x,z):=\inf_{u\in V} \{\int_0^{+\infty} e^{-λs} L(y_{x,z,u}(s), u(s))ds \}\] where $V$ is a class of admissible controls, we prove that $v$ is the only viscosity solution of an Hamilton-Jacobi-Bellman equation of the form: \[λv(x,z)+H(x,z,\nabla_x v(x,z))+D_z v(x,z), \dot{z} >=0\] in the sense of the theory of viscosity s…
In this paper we investigate zero-sum two-player stochastic differential games whose cost functionals are given by doubly controlled reflected backward stochastic differential equations (RBSDEs) with two barriers. For admissible controls which can depend on the whole past and so include, in particular, information occurring before the beginning of the game, the games are interpreted as games of the type "admissible strategy" against "admissible control", and the associated lower and upper value functions are studied. A priori random, they are shown to be deterministic, and it is proved that th…
We consider fully nonlinear Hamilton-Jacobi-Bellman equations associated to diffusion control problems involving a finite set-valued (or switching) control and possibly a continuum-valued control. In previous works (Akian, Fodjo, 2016 and 2017), we introduced a lower complexity probabilistic numerical algorithm for such equations by combining max-plus and numerical probabilistic approaches. The max-plus approach is in the spirit of the one of McEneaney, Kaise and Han (2011), and is based on the distributivity of monotone operators with respect to suprema. The numerical probabilistic approach i…
We investigate a two-player zero-sum differential game with asymmetric information on the payoff and without Isaacs condition. The dynamics is an ordinary differential equation parametrised by two controls chosen by the players. Each player has a private information on the payoff of the game, while his opponent knows only the probability distribution on the information of the other player. We show that a suitable definition of random strategies allows to prove the existence of a value in mixed strategies. Moreover, the value function can be characterised in term of the unique viscosity solutio…
For a zero-sum stochastic game which does not satisfy the Isaacs condition, we provide a value function representation for an Isaacs-type equation whose Hamiltonian lies in between the lower and upper Hamiltonians, as a convex combination of the two. For the general case (i.e. the convex combination is time and state dependent) our representation amounts to a random change of the rules of the game, to allow each player at any moment to see the other player's action or not, according to a coin toss with probabilities of heads and tails given by the convex combination appearing in the PDE. If th…
We consider the problem of overbounding and underbounding both the backward and forward reachable set for a given polynomial vector field, nonlinear in both state and input, with a given semialgebriac set of initial conditions and with inputs constrained pointwise to lie in a semialgebraic set. Specifically, we represent the forward reachable set using the value function which gives the optimal cost to go of an optimal control problems and if smooth satisfies the Hamilton-Jacobi- Bellman PDE. We then show that there exist polynomial upper and lower bounds to this value function and furthermore…
In this paper, we study the existence and uniqueness of viscosity solutions to a kind of Hamilton-Jacobi-Bellman (HJB) equations combined with algebra equations. This HJB equation is related to a stochastic optimal control problem for which the state equation is described by a fully coupled forward-backward stochastic differential equation. By extending Peng's backward semigroup approach to this problem, we obtain the dynamic programming principle and show that the value function is a viscosity solution to this HJB equation. As for the proof of the uniqueness of viscosity solution, the analysi…
In a previous work (Akian, Fodjo, 2016), we introduced a lower complexity probabilistic max-plus numerical method for solving fully nonlinear Hamilton-Jacobi-Bellman equations associated to diffusion control problems involving a finite set-valued (or switching) control and possibly a continuum-valued control. This method was based on the idempotent expansion properties obtained by McEneaney, Kaise and Han (2011) and on the numerical probabilistic method proposed by Fahim, Touzi and Warin (2011) for solving some fully nonlinear parabolic partial differential equations. A difficulty of the algor…
Many optimal control problems are formulated as two point boundary value problems (TPBVPs) with conditions of optimality derived from the Hamilton-Jacobi-Bellman (HJB) equations. In most cases, it is challenging to solve HJBs due to the difficulty of guessing the adjoint variables. This paper proposes two learning-based approaches to find the initial guess of adjoint variables in real-time, which can be applied to solve general TPBVPs. For cases with database of solutions and corresponding adjoint variables of a TPBVP under varying boundary conditions, a supervised learning method is applied t…
In this article, a notion of viscosity solutions is introduced for second order path-dependent Hamilton-Jacobi-Bellman (PHJB) equations associated with optimal control problems for path-dependent stochastic differential equations. We identify the value functional of optimal control problems as unique viscosity solution to the associated PHJB equations. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property. Applications to backward stochastic Hamilton-Jacobi-Bellman equations are also given.
In this paper, a stochastic optimal control problem is investigated in which the system is governed by a stochastic functional differential equation. In the framework of functional Itô calculus, we build the dynamic programming principle and the related Path-dependent Hamilton-Jacobi-Bellman (HJB) equation. We prove that the value function is the viscosity solution of the Path-dependent HJB equation.
In this paper, we study backward doubly stochastic recursive optimal control problem where the cost function is described by the solution of a backward doubly stochastic differential equation. We give the dynamical programming principle for this kind of optimal control problem and show that the value function is the unique Sobolev weak solution for the corresponding stochastic Hamilton-Jacobi-Bellman equation.
This paper proposes a new framework to model control systems in which a dynamic friction occurs. The model consists in a controlled differential inclusion with a discontinuous right hand side, which still preserves existence and uniqueness of the solution for each given input function $u(t)$. Under general hypotheses, we are able to derive the Hamilton-Jacobi-Bellman equation for the related free time optimal control problem and to characterise the value function as the unique, locally Lipschitz continuous viscosity solution.
In this paper, we study one kind of stochastic recursive optimal control problem with the obstacle constraints for the cost function where the cost function is described by the solution of one reflected backward stochastic differential equations. We will give the dynamic programming principle for this kind of optimal control problem and show that the value function is the unique viscosity solution of the obstacle problem for the corresponding Hamilton-Jacobi-Bellman equations.
In this paper we first investigate zero-sum two-player stochastic differential games with reflection with the help of theory of Reflected Backward Stochastic Differential Equations (RBSDEs). We will establish the dynamic programming principle for the upper and the lower value functions of this kind of stochastic differential games with reflection in a straight-forward way. Then the upper and the lower value functions are proved to be the unique viscosity solutions of the associated upper and the lower Hamilton-Jacobi-Bellman-Isaacs equations with obstacles, respectively. The method differs hea…
In this article, a notion of viscosity solutions is introduced for first order path-dependent Hamilton-Jacobi-Bellman (PHJB) equations associated with optimal control problems for path-dependent evolution equations in Hilbert space. We identify the value functional of optimal control problems as unique viscosity solution to the associated PHJB equations. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property.
We investigate the long time behavior of weakly dissipative semilinear Hamilton-Jacobi-Bellman (HJB) equations and the turnpike property for the corresponding stochastic control problems. To this aim, we develop a probabilistic approach based on a variant of coupling by reflection adapted to the study of controlled diffusion processes. We prove existence and uniqueness of solutions for the ergodic Hamilton-Jacobi-Bellman equation and different kind of quantitative exponential convergence results at the level of the value function, of the optimal controls and of the optimal processes. Moreover,…
We show strong uniform convergence of monotone P1 finite element methods to the viscosity solution of isotropic parabolic Hamilton-Jacobi-Bellman equations with mixed boundary conditions on unstructured meshes and for possibly degenerate diffusions. Boundary operators can generally be discontinuous across face-boundaries and type changes. Robin-type boundary conditions are discretised via a lower Dini derivative. In time the Bellman equation is approximated through IMEX schemes. Existence and uniqueness of numerical solutions follows through Howard's algorithm. Keywords: Finite element method,…
This paper introduces a notion of viscosity solutions for second order elliptic Hamilton-Jacobi-Bellman (HJB) equations with infinite delay associated with infinite-horizon optimal control problems for stochastic differential equations with infinite delay. We identify the value functional of optimal control problems as unique viscosity solution to associated second order elliptic HJB equation with infinite delay. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property.
In this article, a notion of viscosity solutions is introduced for second order path-dependent Hamilton-Jacobi-Bellman (PHJB) equations associated with optimal control problems for path-dependent stochastic evolution equations in Hilbert spaces. We identify the value functional of optimal control problems as unique viscosity solution to the associated PHJB equations. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property.
In this paper we show a uniqueness result for weak epigraphical solutions of Hamilton-Jacobi-Bellman (HJB) equations on infinite horizon for a class of lower semicontinuous functions vanishing at infinity. Weak epigraphical solutions of HJB equations, with time-measurable data and fiber-convex, turn out to be viscosity solutions - in the classical sense - whenever they are locally Lipschitz continuous. Here we extend the notion of locally absolutely continuous tubes to set-valued maps with continuous epigraph of locally bounded variations. This new notion fits with the lack of uniform lower bo…
A novel method for computing reachable sets is proposed in this paper. In the proposed method, a Hamilton-Jacobi-Bellman equation with running cost functionis numerically solved and the reachable sets of different time horizons are characterized by a family of non-zero level sets of the solution of the Hamilton-Jacobi-Bellman equation. In addition to the classical reachable set, by setting different running cost functions and terminal conditionsof the Hamilton-Jacobi-Bellman equation, the proposed method allows to compute more generalized reachable sets, which are referred to as cost-limited r…
