A procedure for the numerical approximation of high-dimensional Hamilton-Jacobi-Bellman (HJB) equations associated to optimal feedback control problems for semilinear parabolic equations is proposed. Its main ingredients are a pseudospectral collocation approximation of the PDE dynamics, and an iterative method for the nonlinear HJB equation associated to the feedback synthesis. The latter is known as the Successive Galerkin Approximation. It can also be interpreted as Newton iteration for the HJB equation. At every step, the associated linear Generalized HJB equation is approximated via a separable polynomial approximation ansatz. Stabilizing feedback controls are obtained from solutions to the HJB equations for systems of dimension up to fourteen.
An optimized variant of the State Dependent Riccati Equations (SDREs) approach for nonlinear optimal feedback stabilization is presented. The proposed method is based on the construction of equivalent semilinear representations associated to the dynamics and their affine combination. The optimal combination is chosen to minimize the discrepancy between the SDRE control and the optimal feedback law stemming from the solution of the corresponding Hamilton Jacobi Bellman (HJB) equation. Numerical experiments assess effectiveness of the method in terms of stability of the closed-loop with near-to-…
We show strong uniform convergence of monotone P1 finite element methods to the viscosity solution of isotropic parabolic Hamilton-Jacobi-Bellman equations with mixed boundary conditions on unstructured meshes and for possibly degenerate diffusions. Boundary operators can generally be discontinuous across face-boundaries and type changes. Robin-type boundary conditions are discretised via a lower Dini derivative. In time the Bellman equation is approximated through IMEX schemes. Existence and uniqueness of numerical solutions follows through Howard's algorithm. Keywords: Finite element method,…
Policy iteration is a widely used technique to solve the Hamilton Jacobi Bellman (HJB) equation, which arises from nonlinear optimal feedback control theory. Its convergence analysis has attracted much attention in the unconstrained case. Here we analyze the case with control constraints both for the HJB equations which arise in deterministic and in stochastic control cases. The linear equations in each iteration step are solved by an implicit upwind scheme. Numerical examples are conducted to solve the HJB equation with control constraints and comparisons are shown with the unconstrained case…
Policy iteration for Hamilton-Jacobi-Isaacs equations with control constraints and comparison with Hamilton-Jacobi-Bellman equations
Rajiv Gandhi Institute of Petroleum Technology（拉吉夫·甘地石油技术学院）；University of Graz（格拉茨大学）；Radon Institute for Computational and Applied Mathematics (RICAM)（雷登计算与应用数学研究所）
A tensor decomposition approach for the solution of high-dimensional, fully nonlinear Hamilton-Jacobi-Bellman equations arising in optimal feedback control of nonlinear dynamics is presented. The method combines a tensor train approximation for the value function together with a Newton-like iterative method for the solution of the resulting nonlinear system. The tensor approximation leads to a polynomial scaling with respect to the dimension, partially circumventing the curse of dimensionality. A convergence analysis for the linear-quadratic case is presented. For nonlinear dynamics, the effec…
A gradient-enhanced functional tensor train cross approximation method for the resolution of the Hamilton-Jacobi-Bellman (HJB) equations associated to optimal feedback control of nonlinear dynamics is presented. The procedure uses samples of both the solution of the HJB equation and its gradient to obtain a tensor train approximation of the value function. The collection of the data for the algorithm is based on two possible techniques: Pontryagin Maximum Principle and State Dependent Riccati Equations. Several numerical tests are presented in low and high dimension showing the effectiveness o…
We propose an approximation scheme for a class of semilinear parabolic equations that are convex and coercive in their gradients. Such equations arise often in pricing and portfolio management in incomplete markets and, more broadly, are directly connected to the representation of solutions to backward stochastic differential equations. The proposed scheme is based on splitting the equation in two parts, the first corresponding to a linear parabolic equation and the second to a Hamilton-Jacobi equation. The solutions of these two equations are approximated using, respectively, the Feynman-Kac …
We propose a novel numerical method for high dimensional Hamilton--Jacobi--Bellman (HJB) type elliptic partial differential equations (PDEs). The HJB PDEs, reformulated as optimal control problems, are tackled by the actor-critic framework inspired by reinforcement learning, based on neural network parametrization of the value and control functions. Within the actor-critic framework, we employ a policy gradient approach to improve the control, while for the value function, we derive a variance reduced least-squares temporal difference method using stochastic calculus. To numerically discretize…
This paper derives the Hamilton-Jacobi-Bellman equation of nonlinear optimal control problems for cost functions with fractional discount rate from the Bellman's principle of optimality. The fractional discount rate is described by Mittag-Leffler function that can be considered as a generalized exponential function.
In this paper we study controlled continuous time random walks (CTRWs) and heuristically derive pay-off function dynamic programming (DP) equations which turn in the limit of standard scaling to fractional Hamilton Jacobi Bellman type equations. This paper aims to extend results from [1] in a controlled setting.
For a non-cooperative m-persons differential game, the value functions ofthe various players satisfy a system of Hamilton-Jacobi-Bellman equations.Nashequilibrium solutions in feedback form can be obtained by studying a related system of P.D.E's.A new approach, which is proposed in this paper allows one to construct the feedback optimal control and cost functions J_i(t,x),i=1,...,m directly,i.e.,without any reference to the corresponding Hamilton-Jacobi-Bellman equations.
This paper introduces the Hamilton-Jacobi-Bellman Proximal Policy Optimization (HJBPPO) algorithm into reinforcement learning. The Hamilton-Jacobi-Bellman (HJB) equation is used in control theory to evaluate the optimality of the value function. Our work combines the HJB equation with reinforcement learning in continuous state and action spaces to improve the training of the value network. We treat the value network as a Physics-Informed Neural Network (PINN) to solve for the HJB equation by computing its derivatives with respect to its inputs exactly. The Proximal Policy Optimization (PPO)-Cl…
We introduce some approximation schemes for linear and fully non-linear diffusion equations of Bellman-Isaacs type. Although they are not monotone one can prove their convergence to the viscosity solution of the problem. Effective implementation of these scheme is discussed and they are extensively tested.
We extend the theory of Barles Jakobsen to develop numerical schemes for Hamilton Jacobi Bellman equations. We show that the monotonicity of the schemes can be relaxed still leading to the convergence to the viscosity solution of the equation. We give some examples of such numerical schemes and show that the bounds obtained by the framework developed are not tight. At last we test some numerical schemes.
We study an optimal control problem in Bolza form and we consider the value function associated to this problem. We prove two verification theorems which ensure that, if a function $W$ satisfies some suitable weak continuity assumptions and a Hamilton-Jacobi-Bellman inequality outside a countably $\mathcal H^n$-rectifiable set, then it is lower or equal to the value function. These results can be used for optimal synthesis approach.
We investigate the long time behavior of weakly dissipative semilinear Hamilton-Jacobi-Bellman (HJB) equations and the turnpike property for the corresponding stochastic control problems. To this aim, we develop a probabilistic approach based on a variant of coupling by reflection adapted to the study of controlled diffusion processes. We prove existence and uniqueness of solutions for the ergodic Hamilton-Jacobi-Bellman equation and different kind of quantitative exponential convergence results at the level of the value function, of the optimal controls and of the optimal processes. Moreover,…
We deal with a class of semilinear parabolic PDEs on the space of continuous functions that arise, for example, as Kolmogorov equations associated to the infinite-dimensional lifting of path-dependent SDEs. We investigate existence of smooth solutions through their representation via forward-backward stochastic systems, for which we provide the necessary regularity theory. Because of the lack of smoothing properties of the parabolic operators at hand, solutions in general will only share the same regularity as the coefficients of the equation. To conclude we exhibit an application to Hamilton-…
We introduce some sparse grids interpolations used in Semi-Lagrangian schemes for linear and fully non-linear diffusion Hamilton Jacobi Bellman equations arising in stochastic control. We prove that the method introduced converges toward the viscosity solution of the problem and we show that some potentially high order schemes can be efficiently implemented. Numerical test in dimension $2$ to $5$ are achieved and show that deterministic methods can be used efficiently in stochastic control in moderate dimension.
In this paper, we study the optimal singular controls for stochastic recursive systems, in which the control has two components: the regular control, and the singular control. Under certain assumptions, we establish the dynamic programming principle for this kind of optimal singular controls problem, and prove that the value function is a unique viscosity solution of the corresponding Hamilton-Jacobi-Bellman inequality, in a given class of bounded and continuous functions. At last, an example is given for illustration.
In the present paper, it is provided a representation result for the weak solutions of a class of evolutionary Hamilton-Jacobi-Bellman equations on infinite horizon, with Hamiltonians measurable in time and fiber convex. Such Hamiltonians are associated with a - faithful - representation, namely involving two functions measurable in time and locally Lipschitz in the state and control. Our results concern the recovering of a representation of convex Hamiltonians under a relaxed assumption on the Fenchel transform of the Hamiltonian with respect to the fiber. We apply them to investigate the uni…
In this article, a notion of viscosity solutions is introduced for second order path-dependent Hamilton-Jacobi-Bellman (PHJB) equations associated with optimal control problems for path-dependent stochastic differential equations. We identify the value functional of optimal control problems as unique viscosity solution to the associated PHJB equations. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property. Applications to backward stochastic Hamilton-Jacobi-Bellman equations are also given.
This article is devoted to the study of lower semicontinuous solutions of Hamilton-Jacobi equations with convex Hamiltonians in a gradient variable. Such Hamiltonians appear in the optimal control theory. We present a necessary and sufficient condition for a reduction of a Hamiltonian satisfying optimality conditions to the case when the Hamiltonian is positively homogeneous and also satisfies optimality conditions. It allows us to reduce some uniqueness problems of lower semicontinuous solutions to Barron-Jensen and Frankowska theorems. For Hamiltonians, which cannot be reduced in that way, w…
In this article, a class of optimal control problems of differential equations with delays are investigated for which the associated Hamilton-Jacobi-Bellman (HJB) equations are nonlinear partial differential equations with delays. This type of HJB equation has not been previously studied and is difficult to solve because the state equations do not possess smoothing properties. We introduce a slightly different notion of viscosity solutions and identify the value function of the optimal control problems as a unique viscosity solution to the associated HJB equations.
The paper deals with a Bolza optimal control problem for a dynamical system which motion is described by a delay differential equation under an initial condition defined by a piecewise continuous function. For the value functional in this problem, the Cauchy problem for the Hamilton-Jacobi-Bellman equation with coinvariant derivatives is considered. Minimax and viscosity solutions of this problem are studied. It is proved that both of these solutions exist, are unique and coincide with the value functional.
A novel method for computing reachable sets is proposed in this paper. In the proposed method, a Hamilton-Jacobi-Bellman equation with running cost functionis numerically solved and the reachable sets of different time horizons are characterized by a family of non-zero level sets of the solution of the Hamilton-Jacobi-Bellman equation. In addition to the classical reachable set, by setting different running cost functions and terminal conditionsof the Hamilton-Jacobi-Bellman equation, the proposed method allows to compute more generalized reachable sets, which are referred to as cost-limited r…
In this paper, a stochastic optimal control problem is investigated in which the system is governed by a stochastic functional differential equation. In the framework of functional Itô calculus, we build the dynamic programming principle and the related Path-dependent Hamilton-Jacobi-Bellman (HJB) equation. We prove that the value function is the viscosity solution of the Path-dependent HJB equation.
An example of a nonunique solution of the Cauchy problem of Hamilton-Jacobi-Bellman (HJB) equation with surprisingly regular Hamiltonian is presented. The Hamiltonian H(t,x,p) is locally Lipschitz continuous with respect to all variables, convex in p and with linear growth with respect to p and x. The HJB equation possesses two distinct lower semicontinuous solutions with the same final conditions; moreover, one of them is the value function of the corresponding Bolza problem. The definition of lower semicontinuous solution was proposed by Barron-Jensen (1990) and Frankowska (1993). Using the …
In this paper, we study backward doubly stochastic recursive optimal control problem where the cost function is described by the solution of a backward doubly stochastic differential equation. We give the dynamical programming principle for this kind of optimal control problem and show that the value function is the unique Sobolev weak solution for the corresponding stochastic Hamilton-Jacobi-Bellman equation.
This paper is concerned with the stochastic Hamilton-Jacobi-Bellman equation with controlled leading coefficients, which is a type of fully nonlinear backward stochastic partial differential equation (BSPDE for short). In order to formulate the weak solution for such kind of BSPDEs, the classical potential theory is generalized in the backward stochastic framework. The existence and uniqueness of the weak solution is proved, and for the partially non-Markovian case, we obtain the associated gradient estimate. As a byproduct, the existence and uniqueness of solution for a class of degenerate re…
This paper proposes a new framework to model control systems in which a dynamic friction occurs. The model consists in a controlled differential inclusion with a discontinuous right hand side, which still preserves existence and uniqueness of the solution for each given input function $u(t)$. Under general hypotheses, we are able to derive the Hamilton-Jacobi-Bellman equation for the related free time optimal control problem and to characterise the value function as the unique, locally Lipschitz continuous viscosity solution.
In this paper, we study one kind of stochastic recursive optimal control problem with the obstacle constraints for the cost function where the cost function is described by the solution of one reflected backward stochastic differential equations. We will give the dynamic programming principle for this kind of optimal control problem and show that the value function is the unique viscosity solution of the obstacle problem for the corresponding Hamilton-Jacobi-Bellman equations.
The paper deals with a zero-sum differential game for a dynamical system which motion is described by a nonlinear delay differential equation under an initial condition defined by a piecewise continuous function. The corresponding Cauchy problem for Hamilton-Jacobi-Bellman-Isaacs equation with coinvariant derivatives is derived and the definition of a viscosity solution of this problem is considered. It is proved that the differential game has the value that is the unique viscosity solution. Moreover, based on notions of sub- and superdifferentials corresponding to coinvariant derivatives, the…
In this paper we study the fully nonlinear stochastic Hamilton-Jacobi-Bellman (HJB) equation for the optimal stochastic control problem of stochastic differential equations with random coefficients. The notion of viscosity solution is introduced, and we prove that the value function of the optimal stochastic control problem is the maximal viscosity solution of the associated stochastic HJB equation. For the superparabolic cases when the diffusion coefficients are deterministic functions of time, states and controls, the uniqueness is addressed as well.
In this article, a notion of viscosity solutions is introduced for first order path-dependent Hamilton-Jacobi-Bellman (PHJB) equations associated with optimal control problems for path-dependent evolution equations in Hilbert space. We identify the value functional of optimal control problems as unique viscosity solution to the associated PHJB equations. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property.
This paper introduces a notion of viscosity solutions for second order elliptic Hamilton-Jacobi-Bellman (HJB) equations with infinite delay associated with infinite-horizon optimal control problems for stochastic differential equations with infinite delay. We identify the value functional of optimal control problems as unique viscosity solution to associated second order elliptic HJB equation with infinite delay. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property.
We present a method for determining optimal walking paths in steep terrain using the level set method and an optimal control formulation. By viewing the walking direction as a control variable, we can determine the optimal control by solving a Hamilton-Jacobi-Bellman equation. We then calculate the optimal walking path by solving an ordinary differential equation. We demonstrate the effectiveness of our method by computing optimal paths which travel throughout mountainous regions of Yosemite National Park. We include details regarding the numerical implementation of our model and address a spe…
In this article, a notion of viscosity solutions is introduced for second order path-dependent Hamilton-Jacobi-Bellman (PHJB) equations associated with optimal control problems for path-dependent stochastic evolution equations in Hilbert spaces. We identify the value functional of optimal control problems as unique viscosity solution to the associated PHJB equations. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property.
We present a novel method to approximate optimal feedback laws for nonlinear optimal control based on low-rank tensor train (TT) decompositions. The approach is based on the Dirac-Frenkel variational principle with the modification that the optimisation uses an empirical risk. Compared to current state-of-the-art TT methods, our approach exhibits a greatly reduced computational burden while achieving comparable results. A rigorous description of the numerical scheme and demonstrations of its performance are provided.
