We obtain non-symmetric upper and lower bounds on the rate of convergence of general monotone approximation/numerical schemes for parabolic Hamilton Jacobi Bellman Equations by introducing a new notion of consistency. We apply our general results to various schemes including finite difference schemes, splitting methods and the classical approximation by piecewise constant controls.
In this paper we conduct a priori and a posteriori error analysis of the $C^0$ interior penalty method for Hamilton-Jacobi-Bellman equations, with coefficients that satisfy the Cordes condition. These estimates show the quasi-optimality of the method, and provide one with an adaptive finite element method. In accordance with the proven regularity theory, we only assume that the solution of the Hamilton-Jacobi-Bellman equation belongs to $H^2$.
This paper is concerned with the asymptotic analysis of infinite systems of weakly coupled stationary Hamilton-Jacobi-Bellman equations as the discount factor tends to zero. With a specific Hamiltonian, we show the convergence of the solution and prove the solvability of the corresponding ergodic problem.
In this paper we provide a rate of convergence for periodic homogenization of Hamilton-Jacobi-Bellman equations with nonlocal diffusion. The result is based on the regularity of the associated effective problem, where the convexity plays a crucial role. Such regularity estimates are possible from the available literature once we provide a representation formula for the effective Hamiltonian, a result that has an independent interest.
This paper introduces the Hamilton-Jacobi-Bellman Proximal Policy Optimization (HJBPPO) algorithm into reinforcement learning. The Hamilton-Jacobi-Bellman (HJB) equation is used in control theory to evaluate the optimality of the value function. Our work combines the HJB equation with reinforcement learning in continuous state and action spaces to improve the training of the value network. We treat the value network as a Physics-Informed Neural Network (PINN) to solve for the HJB equation by computing its derivatives with respect to its inputs exactly. The Proximal Policy Optimization (PPO)-Cl…
The aim of this paper is to construct and analyze solutions to a class of Hamilton-Jacobi-Bellman equations with range bounds on the optimal response variable. Using the Riccati transformation we derive and analyze a fully nonlinear parabolic partial differential equation for the optimal response function. We construct monotone traveling wave solutions and identify parametric regions for which the traveling wave solution has a positive or negative wave speed.
We study the partial differential equation max{Lu - f, H(Du)}=0 where u is the unknown function, L is a second-order elliptic operator, f is a given smooth function and H is a convex function. This is a model equation for Hamilton-Jacobi-Bellman equations arising in stochastic singular control. We establish the existence of a unique viscosity solution of the Dirichlet problem that has a Holder continuous gradient. We also show that if H is uniformly convex, the gradient of this solution is Lipschitz continuous.
We obtain weighted uniform estimates for the gradient of the solutions to a class of linear parabolic Cauchy problems with unbounded coefficients. Such estimates are then used to prove existence and uniqueness of the mild solution to a semi-linear backward parabolic Cauchy problem, where the differential equation is the Hamilton-Jacobi-Bellman equation of a suitable optimal control problem. Via backward stochastic differential equations, we show that the mild solution is indeed the Value Function of the controlled equation and that the feedback law is verified.
In this paper we study the fully nonlinear stochastic Hamilton-Jacobi-Bellman (HJB) equation for the optimal stochastic control problem of stochastic differential equations with random coefficients. The notion of viscosity solution is introduced, and we prove that the value function of the optimal stochastic control problem is the maximal viscosity solution of the associated stochastic HJB equation. For the superparabolic cases when the diffusion coefficients are deterministic functions of time, states and controls, the uniqueness is addressed as well.
In this paper we establish periodic homogenization for Hamilton-Jacobi-Bellman (HJB) equations, associated to nonlocal operators of integro-differential type. We consider the case when the fractional diffusion has the same order as the drift term, and is weakly elliptic. The outcome of the paper is twofold. One one hand, we provide Lipschitz regularity results for weakly elliptic non-local HJB, extending the results previously obtained in [8]. On the other hand, we establish a convergence result, based on half relaxed limits and a comparison principle for the effective problem. The latter stro…
The quasi-periodic homogenization for some classes of first-order Hamilton-Jaconi-Bellman equation is studied in this paper. The cell problem of the quasi-periodic homogenization satisfies the non-resonance condition, under which the corresponding deterministic system is ergodic. The almost periodic homogenization for the same classes of equations is also solved, as a limit of a sequence of quasi-periodic homogenizations. Here, the almost periodicity is in the sense of H. Bohr. This result has been cited by some authors, for example: by H. Ishii, "Almost periodic homogenization of Hamilton-Jac…
In this paper we show a uniqueness result for weak epigraphical solutions of Hamilton-Jacobi-Bellman (HJB) equations on infinite horizon for a class of lower semicontinuous functions vanishing at infinity. Weak epigraphical solutions of HJB equations, with time-measurable data and fiber-convex, turn out to be viscosity solutions - in the classical sense - whenever they are locally Lipschitz continuous. Here we extend the notion of locally absolutely continuous tubes to set-valued maps with continuous epigraph of locally bounded variations. This new notion fits with the lack of uniform lower bo…
We study the well-posedness of Hamilton-Jacobi-Bellman equations on subsets of $\mathbb{R}^d$ in a context without boundary conditions. The Hamiltonian is given as the supremum over two parts: an internal Hamiltonian depending on an external control variable and a cost functional penalizing the control. The key feature in this paper is that the control function can be unbounded and discontinuous. This way we can treat functionals that appear e.g. in the Donsker-Varadhan theory of large deviations for occupation time measures. To allow for this flexibility, we assume that the internal Hamiltoni…
We derive the long time asymptotic of solutions to an evolutive Hamilton-Jacobi-Bellman equation in a bounded smooth domain, in connection with ergodic problems recently studied in \cite{bcr}. Our main assumption is an appropriate degeneracy condition on the operator at the boundary. This condition is related to the characteristic boundary points for linear operators as well as to the irrelevant points for the generalized Dirichlet problem, and implies in particular that no boundary datum has to be imposed. We prove that there exists a constant $c$ such that the solutions of the evolutive prob…
Here, we study the large-time limit of viscosity solutions of the Cauchy problem for second-order Hamilton--Jacobi--Bellman equations with convex Hamiltonians in the torus. This large-time limit solves the corresponding stationary problem, sometimes called the ergodic problem. This problem, however, has multiple viscosity solutions and, thus, a key question is which of these solutions is selected by the limit. Here, we provide a representation for the viscosity solution to the Cauchy problem in terms of generalized holonomic measures. Then, we use this representation to characterize the large-…
We consider the homogenization of Hamilton-Jacobi equations and degenerate Bellman equations in stationary, ergodic, unbounded environments. We prove that, as the microscopic scale tends to zero, the equation averages to a deterministic Hamilton-Jacobi equation and study some properties of the effective Hamiltonian. We discover a connection between the effective Hamiltonian and an eikonal-type equation in exterior domains. In particular, we obtain a new formula for the effective Hamiltonian. To prove the results we introduce a new strategy to obtain almost sure homogenization, completing a pro…
We study the regularity properties of integro-partial differential equations of Hamilton-Jocobi-Bellman type with terminal condition, which can be interpreted through a stochastic control system, composed of a forward and a backward stochastic differential equation, both driven by a Brownian motion and a compensated Poisson random measure. More precisely, we prove that, under appropriate assumptions, the viscosity solution of such equations is jointly Lipschitz and jointly semiconcave in $(t,x)\inΔ\times\R^d$, for all compact time intervals $Δ$ excluding the terminal time. Our approach is base…
We consider a Bolza-type optimal control problem for a dynamical system described by a fractional differential equation with the Caputo derivative of an order $α\in (0, 1)$. The value of this problem is introduced as a functional in a suitable space of histories of motions. We prove that this functional satisfies the dynamic programming principle. Based on a new notion of coinvariant derivatives of the order $α$, we associate the considered optimal control problem with a Hamilton-Jacobi-Bellman equation. Under certain smoothness assumptions, we establish a connection between the value function…
We analyze the consequences that the so-called turnpike property has on the long-time behavior of the value function corresponding to a finite-dimensional linear-quadratic optimal control problem with general terminal cost and constrained controls. We prove that, when the time horizon $T$ tends to infinity, the value function asymptotically behaves as $W(x) + c\, T + λ$, and we provide a control interpretation of each of these three terms, making clear the link with the turnpike property. As a by-product, we obtain the long-time behavior of the solution to the associated Hamilton-Jacobi-Bellma…
In this paper, we propose and study the stochastic path-dependent Hamilton-Jacobi-Bellman (SPHJB) equation that arises naturally from the optimal stochastic control problem of stochastic differential equations with path-dependence and measurable randomness. Both the notions of viscosity solution and classical solution are proposed, and the value function of the optimal stochastic control problem is proved to be the viscosity solution to the associated SPHJB equation. A uniqueness result about viscosity solutions is also given for certain superparabolic cases, while the uniqueness of classical …
We study PDE of the form $\max\{F(D^2u,x)-f(x), H(Du)\}=0$ where $F$ is uniformly elliptic and convex in its first argument, $H$ is convex, $f$ is a given function and $u$ is the unknown. These equations are derived from dynamic programming in a wide class of stochastic singular control problems. In particular, examples of these equations arise in mathematical finance models involving transaction costs, in queuing theory, and spacecraft control problems. The main aspects of this work are to identify conditions under which solutions are uniquely defined and have Lipschitz continuous gradients. …
We study in this series of articles the Kardar-Parisi-Zhang (KPZ) equation $$ \partial_t h(t,x)=νΔh(t,x)+λV(|\nabla h(t,x)|) +\sqrt{D}\, η(t,x), \qquad x\in{\mathbb{R}}^d $$ in $d\ge 1$ dimensions. The forcing term $η$ in the right-hand side is a regularized white noise. The deposition rate $V$ is assumed to be isotropic and convex. Assuming $V(0)\ge 0$, one finds $V(|\nabla h|)\ltimes |\nabla h|^2$ for small gradients, yielding the equation which is most commonly used in the literature. The present article, a continuation of [24], is dedicated to a generalization of the PDE estimates obtained…
In this paper we study homogenization for a class of monotone systems of first-order time-dependent periodic Hamilton-Jacobi equations. We characterize the Hamiltonians of the limit problem by appropriate cell problems. Hence we show the uniform convergence of the solution of the oscillating systems to the bounded uniformly continuous solution of the homogenized system.
We consider the homogenization of monotone systems of viscous Hamilton-Jacobi equations with convex nonlinearities set in the stationary, ergodic setting. The primary focus of this paper is on collapsing systems which, as the microscopic scale tends to zero, average to a deterministic scalar Hamilton-Jacobi equation. However, our methods also apply to systems which do not collapse and, as the microscopic scale tends to zero, average to a deterministic system of Hamilton-Jacobi equations.
We study the vanishing discount problem for a nonlinear monotone system of Hamilton-Jacobi equations. This continues the first author's investigation on the vanishing discount problem for a monotone system of Hamilton-Jacobi equations. As in Part 1, we introduce by the convex duality Mather measures and their analogues for the system, which we call respectively Mather and Green-Poisson measures, and prove a convergence theorem for the vanishing discount problem. Moreover, we establish an existence result for the ergodic problem.
This paper is a review of results on Optimisation which are perhaps not so standard in the PDE realm. To this end, we consider the problem of deriving the PDEs associated to the optimal control of a system of either ODEs or SDEs with respect to a vector-valued cost functional. Optimisation is considered with respect to a partial ordering generated by a given cone. Since in the vector case minima may not exist, we define vectorial value functions as (Pareto) minimals of the ordering. Our main objective is the derivation of the model PDEs which turn out to be parametric families of HJB single eq…
We investigate the large-time behavior of viscosity solutions of quasi-monotone weakly coupled systems of Hamilton--Jacobi equations on the $n$-dimensional torus. We establish a convergence result to asymptotic solutions as time goes to infinity under rather restricted assumptions.
A new stochastic control model for the long-run environmental management of rivers is mathematically and numerically analyzed, focusing on a modern sediment replenishment problem with unique nonsmooth and nonlinear properties. Rational inattention as a novel adaptive strategy to collect information and intervene against the target system is modeled using Erlangization. The system dynamics containing the river discharge following a continuous-state branching with an immigration-type process and the controlled sediment storage dynamics lead to a nonsmooth and nonlocal infinitesimal generator. Mo…
