变分法中值函数的长时间平均行为已知与相应 Abel 均值的极限存在性相关。在 Tonelli 情形下，该极限又与临界 Hamilton-Jacobi 方程解的存在性相关。本文旨在探讨当设定在整个欧几里得空间且 Hamilton 量不满足 Tonelli 条件时的类似问题。我们首先研究时间平均值函数在时间跨度趋于无穷时的收敛性，证明了一般控制系统的临界常数存在性。接着，我们证明遍历方程对于与满足 Lie 代数秩条件的一族向量场相关的系统存在解。最后，我们在整个空间上构造了 HJB 方程的临界解，该解与其 Lax-Oleinik 演化一致。
英文摘要：
The long-time average behavior of the value function in the calculus of variations is known to be connected to the existence of the limit of the corresponding Abel means. Still in the Tonelli case, such a limit is in turn related to the existence of solutions of the critical Hamilton-Jacobi equation. The goal of this paper is to address similar issues when set on the whole Euclidean space and the Hamiltonian fails to be Tonelli. We first study the convergence of the time-averaged value function as the time horizon goes to infinity, proving the existence of the critical constant for a general control system. Then, we show that the ergodic equation admits solutions for systems associated with a family of vector fields which satisfies the Lie Algebra rank condition. Finally, we construct a critical solution of the HJB equation on the whole space which coincides with its Lax-Oleinik evolution.
In this paper we show a uniqueness result for weak epigraphical solutions of Hamilton-Jacobi-Bellman (HJB) equations on infinite horizon for a class of lower semicontinuous functions vanishing at infinity. Weak epigraphical solutions of HJB equations, with time-measurable data and fiber-convex, turn out to be viscosity solutions - in the classical sense - whenever they are locally Lipschitz continuous. Here we extend the notion of locally absolutely continuous tubes to set-valued maps with continuous epigraph of locally bounded variations. This new notion fits with the lack of uniform lower bo…
This paper derives the Hamilton-Jacobi-Bellman equation of nonlinear optimal control problems for cost functions with fractional discount rate from the Bellman's principle of optimality. The fractional discount rate is described by Mittag-Leffler function that can be considered as a generalized exponential function.
We study the well-posedness of Hamilton-Jacobi-Bellman equations on subsets of $\mathbb{R}^d$ in a context without boundary conditions. The Hamiltonian is given as the supremum over two parts: an internal Hamiltonian depending on an external control variable and a cost functional penalizing the control. The key feature in this paper is that the control function can be unbounded and discontinuous. This way we can treat functionals that appear e.g. in the Donsker-Varadhan theory of large deviations for occupation time measures. To allow for this flexibility, we assume that the internal Hamiltoni…
In this paper we study the fully nonlinear stochastic Hamilton-Jacobi-Bellman (HJB) equation for the optimal stochastic control problem of stochastic differential equations with random coefficients. The notion of viscosity solution is introduced, and we prove that the value function of the optimal stochastic control problem is the maximal viscosity solution of the associated stochastic HJB equation. For the superparabolic cases when the diffusion coefficients are deterministic functions of time, states and controls, the uniqueness is addressed as well.
In this paper we study controlled continuous time random walks (CTRWs) and heuristically derive pay-off function dynamic programming (DP) equations which turn in the limit of standard scaling to fractional Hamilton Jacobi Bellman type equations. This paper aims to extend results from [1] in a controlled setting.
For a non-cooperative m-persons differential game, the value functions ofthe various players satisfy a system of Hamilton-Jacobi-Bellman equations.Nashequilibrium solutions in feedback form can be obtained by studying a related system of P.D.E's.A new approach, which is proposed in this paper allows one to construct the feedback optimal control and cost functions J_i(t,x),i=1,...,m directly,i.e.,without any reference to the corresponding Hamilton-Jacobi-Bellman equations.
We analyze the consequences that the so-called turnpike property has on the long-time behavior of the value function corresponding to a finite-dimensional linear-quadratic optimal control problem with general terminal cost and constrained controls. We prove that, when the time horizon $T$ tends to infinity, the value function asymptotically behaves as $W(x) + c\, T + λ$, and we provide a control interpretation of each of these three terms, making clear the link with the turnpike property. As a by-product, we obtain the long-time behavior of the solution to the associated Hamilton-Jacobi-Bellma…
We extend the theory of Barles Jakobsen to develop numerical schemes for Hamilton Jacobi Bellman equations. We show that the monotonicity of the schemes can be relaxed still leading to the convergence to the viscosity solution of the equation. We give some examples of such numerical schemes and show that the bounds obtained by the framework developed are not tight. At last we test some numerical schemes.
We introduce some approximation schemes for linear and fully non-linear diffusion equations of Bellman-Isaacs type. Although they are not monotone one can prove their convergence to the viscosity solution of the problem. Effective implementation of these scheme is discussed and they are extensively tested.
We study an optimal control problem in Bolza form and we consider the value function associated to this problem. We prove two verification theorems which ensure that, if a function $W$ satisfies some suitable weak continuity assumptions and a Hamilton-Jacobi-Bellman inequality outside a countably $\mathcal H^n$-rectifiable set, then it is lower or equal to the value function. These results can be used for optimal synthesis approach.
We introduce some sparse grids interpolations used in Semi-Lagrangian schemes for linear and fully non-linear diffusion Hamilton Jacobi Bellman equations arising in stochastic control. We prove that the method introduced converges toward the viscosity solution of the problem and we show that some potentially high order schemes can be efficiently implemented. Numerical test in dimension $2$ to $5$ are achieved and show that deterministic methods can be used efficiently in stochastic control in moderate dimension.
In this paper, we study the optimal singular controls for stochastic recursive systems, in which the control has two components: the regular control, and the singular control. Under certain assumptions, we establish the dynamic programming principle for this kind of optimal singular controls problem, and prove that the value function is a unique viscosity solution of the corresponding Hamilton-Jacobi-Bellman inequality, in a given class of bounded and continuous functions. At last, an example is given for illustration.
We consider a Bolza-type optimal control problem for a dynamical system described by a fractional differential equation with the Caputo derivative of an order $α\in (0, 1)$. The value of this problem is introduced as a functional in a suitable space of histories of motions. We prove that this functional satisfies the dynamic programming principle. Based on a new notion of coinvariant derivatives of the order $α$, we associate the considered optimal control problem with a Hamilton-Jacobi-Bellman equation. Under certain smoothness assumptions, we establish a connection between the value function…
In this article, a class of optimal control problems of differential equations with delays are investigated for which the associated Hamilton-Jacobi-Bellman (HJB) equations are nonlinear partial differential equations with delays. This type of HJB equation has not been previously studied and is difficult to solve because the state equations do not possess smoothing properties. We introduce a slightly different notion of viscosity solutions and identify the value function of the optimal control problems as a unique viscosity solution to the associated HJB equations.
The paper deals with a Bolza optimal control problem for a dynamical system which motion is described by a delay differential equation under an initial condition defined by a piecewise continuous function. For the value functional in this problem, the Cauchy problem for the Hamilton-Jacobi-Bellman equation with coinvariant derivatives is considered. Minimax and viscosity solutions of this problem are studied. It is proved that both of these solutions exist, are unique and coincide with the value functional.
This paper is concerned with the stochastic Hamilton-Jacobi-Bellman equation with controlled leading coefficients, which is a type of fully nonlinear backward stochastic partial differential equation (BSPDE for short). In order to formulate the weak solution for such kind of BSPDEs, the classical potential theory is generalized in the backward stochastic framework. The existence and uniqueness of the weak solution is proved, and for the partially non-Markovian case, we obtain the associated gradient estimate. As a byproduct, the existence and uniqueness of solution for a class of degenerate re…
In this paper, we propose and study the stochastic path-dependent Hamilton-Jacobi-Bellman (SPHJB) equation that arises naturally from the optimal stochastic control problem of stochastic differential equations with path-dependence and measurable randomness. Both the notions of viscosity solution and classical solution are proposed, and the value function of the optimal stochastic control problem is proved to be the viscosity solution to the associated SPHJB equation. A uniqueness result about viscosity solutions is also given for certain superparabolic cases, while the uniqueness of classical …
We study the regularity properties of integro-partial differential equations of Hamilton-Jocobi-Bellman type with terminal condition, which can be interpreted through a stochastic control system, composed of a forward and a backward stochastic differential equation, both driven by a Brownian motion and a compensated Poisson random measure. More precisely, we prove that, under appropriate assumptions, the viscosity solution of such equations is jointly Lipschitz and jointly semiconcave in $(t,x)\inΔ\times\R^d$, for all compact time intervals $Δ$ excluding the terminal time. Our approach is base…
In this article, a notion of viscosity solutions is introduced for second order path-dependent Hamilton-Jacobi-Bellman (PHJB) equations associated with optimal control problems for path-dependent stochastic differential equations. We identify the value functional of optimal control problems as unique viscosity solution to the associated PHJB equations. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property. Applications to backward stochastic Hamilton-Jacobi-Bellman equations are also given.
We show strong uniform convergence of monotone P1 finite element methods to the viscosity solution of isotropic parabolic Hamilton-Jacobi-Bellman equations with mixed boundary conditions on unstructured meshes and for possibly degenerate diffusions. Boundary operators can generally be discontinuous across face-boundaries and type changes. Robin-type boundary conditions are discretised via a lower Dini derivative. In time the Bellman equation is approximated through IMEX schemes. Existence and uniqueness of numerical solutions follows through Howard's algorithm. Keywords: Finite element method,…
The paper deals with a zero-sum differential game for a dynamical system which motion is described by a nonlinear delay differential equation under an initial condition defined by a piecewise continuous function. The corresponding Cauchy problem for Hamilton-Jacobi-Bellman-Isaacs equation with coinvariant derivatives is derived and the definition of a viscosity solution of this problem is considered. It is proved that the differential game has the value that is the unique viscosity solution. Moreover, based on notions of sub- and superdifferentials corresponding to coinvariant derivatives, the…
In this paper, a stochastic optimal control problem is investigated in which the system is governed by a stochastic functional differential equation. In the framework of functional Itô calculus, we build the dynamic programming principle and the related Path-dependent Hamilton-Jacobi-Bellman (HJB) equation. We prove that the value function is the viscosity solution of the Path-dependent HJB equation.
A novel method for computing reachable sets is proposed in this paper. In the proposed method, a Hamilton-Jacobi-Bellman equation with running cost functionis numerically solved and the reachable sets of different time horizons are characterized by a family of non-zero level sets of the solution of the Hamilton-Jacobi-Bellman equation. In addition to the classical reachable set, by setting different running cost functions and terminal conditionsof the Hamilton-Jacobi-Bellman equation, the proposed method allows to compute more generalized reachable sets, which are referred to as cost-limited r…
In the present paper, it is provided a representation result for the weak solutions of a class of evolutionary Hamilton-Jacobi-Bellman equations on infinite horizon, with Hamiltonians measurable in time and fiber convex. Such Hamiltonians are associated with a - faithful - representation, namely involving two functions measurable in time and locally Lipschitz in the state and control. Our results concern the recovering of a representation of convex Hamiltonians under a relaxed assumption on the Fenchel transform of the Hamiltonian with respect to the fiber. We apply them to investigate the uni…
This paper proposes a new framework to model control systems in which a dynamic friction occurs. The model consists in a controlled differential inclusion with a discontinuous right hand side, which still preserves existence and uniqueness of the solution for each given input function $u(t)$. Under general hypotheses, we are able to derive the Hamilton-Jacobi-Bellman equation for the related free time optimal control problem and to characterise the value function as the unique, locally Lipschitz continuous viscosity solution.
An example of a nonunique solution of the Cauchy problem of Hamilton-Jacobi-Bellman (HJB) equation with surprisingly regular Hamiltonian is presented. The Hamiltonian H(t,x,p) is locally Lipschitz continuous with respect to all variables, convex in p and with linear growth with respect to p and x. The HJB equation possesses two distinct lower semicontinuous solutions with the same final conditions; moreover, one of them is the value function of the corresponding Bolza problem. The definition of lower semicontinuous solution was proposed by Barron-Jensen (1990) and Frankowska (1993). Using the …
In this paper, we study one kind of stochastic recursive optimal control problem with the obstacle constraints for the cost function where the cost function is described by the solution of one reflected backward stochastic differential equations. We will give the dynamic programming principle for this kind of optimal control problem and show that the value function is the unique viscosity solution of the obstacle problem for the corresponding Hamilton-Jacobi-Bellman equations.
We present a method for determining optimal walking paths in steep terrain using the level set method and an optimal control formulation. By viewing the walking direction as a control variable, we can determine the optimal control by solving a Hamilton-Jacobi-Bellman equation. We then calculate the optimal walking path by solving an ordinary differential equation. We demonstrate the effectiveness of our method by computing optimal paths which travel throughout mountainous regions of Yosemite National Park. We include details regarding the numerical implementation of our model and address a spe…
This article is devoted to the study of lower semicontinuous solutions of Hamilton-Jacobi equations with convex Hamiltonians in a gradient variable. Such Hamiltonians appear in the optimal control theory. We present a necessary and sufficient condition for a reduction of a Hamiltonian satisfying optimality conditions to the case when the Hamiltonian is positively homogeneous and also satisfies optimality conditions. It allows us to reduce some uniqueness problems of lower semicontinuous solutions to Barron-Jensen and Frankowska theorems. For Hamiltonians, which cannot be reduced in that way, w…
We present a novel method to approximate optimal feedback laws for nonlinear optimal control based on low-rank tensor train (TT) decompositions. The approach is based on the Dirac-Frenkel variational principle with the modification that the optimisation uses an empirical risk. Compared to current state-of-the-art TT methods, our approach exhibits a greatly reduced computational burden while achieving comparable results. A rigorous description of the numerical scheme and demonstrations of its performance are provided.
In this paper, we study a stochastic recursive optimal control problem in which the value functional is defined by the solution of a backward stochastic differential equation (BSDE) under $\tilde{G}$-expectation. Under standard assumptions, we establish the comparison theorem for this kind of BSDE and give a novel and simple method to obtain the dynamic programming principle. Finally, we prove that the value function is the unique viscosity solution of a type of fully nonlinear HJB equation.
In this article, a notion of viscosity solutions is introduced for first order path-dependent Hamilton-Jacobi-Bellman (PHJB) equations associated with optimal control problems for path-dependent evolution equations in Hilbert space. We identify the value functional of optimal control problems as unique viscosity solution to the associated PHJB equations. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property.
We investigate the long time behavior of weakly dissipative semilinear Hamilton-Jacobi-Bellman (HJB) equations and the turnpike property for the corresponding stochastic control problems. To this aim, we develop a probabilistic approach based on a variant of coupling by reflection adapted to the study of controlled diffusion processes. We prove existence and uniqueness of solutions for the ergodic Hamilton-Jacobi-Bellman equation and different kind of quantitative exponential convergence results at the level of the value function, of the optimal controls and of the optimal processes. Moreover,…
This paper introduces a notion of viscosity solutions for second order elliptic Hamilton-Jacobi-Bellman (HJB) equations with infinite delay associated with infinite-horizon optimal control problems for stochastic differential equations with infinite delay. We identify the value functional of optimal control problems as unique viscosity solution to associated second order elliptic HJB equation with infinite delay. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property.
In this work we study the stochastic recursive control problem, in which the aggregator (or called generator) of the backward stochastic differential equation describing the running cost is continuous but not necessarily Lipschitz with respect to the first unknown variable and the control, and monotonic with respect to the first unknown variable. The dynamic programming principle and the connection between the value function and the viscosity solution of the associated Hamilton-Jacobi-Bellman equation are established in this setting by the generalized comparison theorem of backward stochastic …
In this article, a notion of viscosity solutions is introduced for second order path-dependent Hamilton-Jacobi-Bellman (PHJB) equations associated with optimal control problems for path-dependent stochastic evolution equations in Hilbert spaces. We identify the value functional of optimal control problems as unique viscosity solution to the associated PHJB equations. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property.
This paper introduces the Hamilton-Jacobi-Bellman Proximal Policy Optimization (HJBPPO) algorithm into reinforcement learning. The Hamilton-Jacobi-Bellman (HJB) equation is used in control theory to evaluate the optimality of the value function. Our work combines the HJB equation with reinforcement learning in continuous state and action spaces to improve the training of the value network. We treat the value network as a Physics-Informed Neural Network (PINN) to solve for the HJB equation by computing its derivatives with respect to its inputs exactly. The Proximal Policy Optimization (PPO)-Cl…
A deep learning approach for the approximation of the Hamilton-Jacobi-Bellman partial differential equation (HJB PDE) associated to the Nonlinear Quadratic Regulator (NLQR) problem. A state-dependent Riccati equation control law is first used to generate a gradient-augmented synthetic dataset for supervised learning. The resulting model becomes a warm start for the minimization of a loss function based on the residual of the HJB PDE. The combination of supervised learning and residual minimization avoids spurious solutions and mitigate the data inefficiency of a supervised learning-only approa…
Policy iteration for Hamilton-Jacobi-Isaacs equations with control constraints and comparison with Hamilton-Jacobi-Bellman equations
Rajiv Gandhi Institute of Petroleum Technology（拉吉夫·甘地石油技术学院）；University of Graz（格拉茨大学）；Radon Institute for Computational and Applied Mathematics (RICAM)（雷登计算与应用数学研究所）
