This paper, which is the natural continuation of a previous paper by the same authors, studies a class of optimal control problems with state constraints where the state equation is a differential equation with delays. This class includes some problems arising in economics, in particular the so-called models with time to build. The problem is embedded in a suitable Hilbert space H and the regularity of the associated Hamilton-Jacobi-Bellman (HJB) equation is studied. Therein the main result is that the value function V solves the HJB equation and has continuous classical derivative in the direction of the present. The goal of the present paper is to exploit such result to find optimal feedback strategies for the problem. While it is easy to define formally a feedback strategy in classical sense the proof of its existence and of its optimality is hard due to lack of full regularity of V and to the infinite dimension. Finally, we show some approximation results that allow us to apply our main theorem to obtain epsilon-optimal strategies for a wider class of problems.
In this paper, a robust optimal reinsurance-investment problem with delay is studied under the $α$-maxmin mean-variance criterion. The surplus process of an insurance company approximates Brownian motion with drift. The financial market consists of a risk-free asset and a risky asset that obeys geometric Brownian motion. Using the principle of dynamic programming and Hamilton-Jacobin-Bellman (HJB) equation, the specific expression of optimal strategy and the explicit solution of the corresponding HJB equation are obtained. In addition, a verification theorem is provided to ensure that the valu…
A learning technique for finite horizon optimal control problems and its approximation based on polynomials is analyzed. It allows to circumvent, in part, the curse dimensionality which is involved when the feedback law is constructed by using the Hamilton-Jacobi-Bellman (HJB) equation. The convergence of the method is analyzed, while paying special attention to avoid the use of a global Lipschitz condition on the nonlinearity which describes the control system. The practicality and efficiency of the method is illustrated by several examples. For two of them a direct approach based on the HJB …
This paper introduces a notion of viscosity solutions for second order elliptic Hamilton-Jacobi-Bellman (HJB) equations with infinite delay associated with infinite-horizon optimal control problems for stochastic differential equations with infinite delay. We identify the value functional of optimal control problems as unique viscosity solution to associated second order elliptic HJB equation with infinite delay. We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property.
In this paper we study a first extension of the theory of mild solutions for HJB equations in Hilbert spaces to the case when the domain is not the whole space. More precisely, we consider a half-space as domain, and a semilinear Hamilton-Jacobi-Bellman (HJB) equation. Our main goal is to establish the existence and the uniqueness of solutions to such HJB equations, that are continuously differentiable in the space variable. We also provide an application of our results to an exit time optimal control problem and we show that the corresponding value function is the unique solution to a semilin…
University of California, Santa Cruz（加州大学圣克鲁兹分校）；Naval Postgraduate School（海军研究生院）
Computing optimal feedback controls for nonlinear systems generally requires solving Hamilton-Jacobi-Bellman (HJB) equations, which are notoriously difficult when the state dimension is large. Existing strategies for high-dimensional problems often rely on specific, restrictive problem structures, or are valid only locally around some nominal trajectory. In this paper, we propose a data-driven method to approximate semi-global solutions to HJB equations for general high-dimensional nonlinear systems and compute candidate optimal feedback controls in real-time. To accomplish this, we model solu…
This paper presents the optimal control and synchronization problem of a multilevel network of Rössler chaotic oscillators. Using the Hamilton-Jacobi-Bellman (HJB) technique, the optimal control law with three-state variables feedback is designed such that the trajectories of all the Rössler oscillators in the network are optimally synchronized in each level. Furthermore, we provide numerical simulations to demonstrate the effectiveness of the proposed approach for the cases of one and three networks. A perfect correlation between the MATLAB and the PSPICE results was obtained, thus allowing t…
Policy iteration is a widely used technique to solve the Hamilton Jacobi Bellman (HJB) equation, which arises from nonlinear optimal feedback control theory. Its convergence analysis has attracted much attention in the unconstrained case. Here we analyze the case with control constraints both for the HJB equations which arise in deterministic and in stochastic control cases. The linear equations in each iteration step are solved by an implicit upwind scheme. Numerical examples are conducted to solve the HJB equation with control constraints and comparisons are shown with the unconstrained case…
Lagrangian Relaxation for Continuous-Time Optimal Control of Coupled Hydrothermal Power Systems Including Storage Capacity and a Cascade of Hydropower Systems with Time Delays
We treat infinite horizon optimal control problems by solving the associated stationary Hamilton-Jacobi-Bellman (HJB) equation numerically to compute the value function and an optimal feedback law. The dynamical systems under consideration are spatial discretizations of non linear parabolic partial differential equations (PDE), which means that the HJB is non linear and suffers from the curse of dimensionality. Its non linearity is handled by the Policy Iteration algorithm, where the problem is reduced to a sequence of linear, hyperbolic PDEs. These equations remain the computational bottlenec…
The Dynamic Programming approach allows to compute a feedback control for nonlinear problems, but suffers from the curse of dimensionality. The computation of the control relies on the resolution of a nonlinear PDE, the Hamilton-Jacobi-Bellman equation, with the same dimension of the original problem. Recently, a new numerical method to compute the value function on a tree structure has been introduced. The method allows to work without a structured grid and avoids any interpolation. Here, we aim to test the algorithm for nonlinear two dimensional PDEs. We apply model order reduction to decrea…
This paper deals with a family of stochastic control problems in Hilbert spaces which arises in typical applications (such as boundary control and control of delay equations with delay in the control) and for which is difficult to apply the dynamic programming approach due to the unboudedness of the control operator and to the lack of regularity of the underlying transition semigroup. We introduce a specific concept of partial derivative, designed for this situation, and we develop a method to prove that the associated HJB equation has a solution with enough regularity to find optimal controls…
This paper presents a new Python software for the parametric design of stabilizing feedback laws with time delays, called Partial Pole Placement via Delay Action (P3$δ$). After an introduction recalling recent theoretical results on the multiplicity-induced-dominancy (MID) and coexisting real roots-induced-dominancy (CRRID) properties and their use for the feedback stabilization of control systems operating under time delays, the paper presents the current version of P3$δ$, which relies on the MID property to compute delayed stabilizing feedback laws for scalar differential equations with a si…
In this work, we study the optimal control of stochastic Burgers equation perturbed by Gaussian and Levy type noises with distributed control process acting on the state equation. We use the dynamic programming approach for the second order Hamilton-Jacobi- Bellman (HJB) equation consisting of an integro-differential operator with Levy measure associated with the stochastic control problem. Using the regularizing properties of the transition semigroup corresponding to the stochastic Burgers equation and compactness arguments, we solve the HJB equation and the resultant feedback control problem…
In this paper, we investigate the estimator-based output feedback control problem of multi-delay systems. This work is an extension of recently developed operator-value LMI framework for infinite-dimensional time-delay systems. Based on the optimal convex state feedback controller and generalized Luenberger observer synthesis conditions we already have, the estimator-based output feedback controller is designed to contain the estimates of both the present state and history of the state. An output feedback controller synthesis condition is proposed using SOS method, which is expressed in a set …
An optimized variant of the State Dependent Riccati Equations (SDREs) approach for nonlinear optimal feedback stabilization is presented. The proposed method is based on the construction of equivalent semilinear representations associated to the dynamics and their affine combination. The optimal combination is chosen to minimize the discrepancy between the SDRE control and the optimal feedback law stemming from the solution of the corresponding Hamilton Jacobi Bellman (HJB) equation. Numerical experiments assess effectiveness of the method in terms of stability of the closed-loop with near-to-…
For any suitable Optimal Control Problem (OCP) there exists a value function, defined as the unique viscosity solution to the Hamilton-Jacobi-Bellman (HJB) Partial-Differential-Equation (PDE), and which can be used to design an optimal feedback controller for the given OCP. In this paper, we approximately solve the HJB-PDE by proposing a sequence of Sum-Of-Squares (SOS) problems, each of which yields a polynomial subsolution to the HJB-PDE. We show that the resulting sequence of polynomial sub-solutions converges to the value function of the OCP in the L1 norm. Furthermore, for each polynomial…
A novel approach to design the feedback control based on past states is proposed for hybrid stochastic differential equations (HSDEs). This new theorem builds up the connection between the delay feedback control and the control function without delay terms, which enables one to construct the delay feedback control using the existing results on stabilities of HSDEs. Methods to find the upper bound of the length of the time delay are also investigated. Numerical simulations are presented to demonstrate the new theorem.
Motivated by improving performance of a bi-stable vibration energy harvester (VEH) from the viewpoint of vibration control, the time-delayed feedback control of displacement and velocity are constructively proposed into an electromechanical coupled VEH mounted on a rotational automobile tire, which is subject to colored noise and the periodic excitation. Using the improved stochastic averaging procedure based on energy-dependent frequency, the expressions of stationary probability density (SPD) and signal-to-noise ratio (SNR) are obtained analytically. Then, the efficiency of time-delayed feed…
A gradient-enhanced functional tensor train cross approximation method for the resolution of the Hamilton-Jacobi-Bellman (HJB) equations associated to optimal feedback control of nonlinear dynamics is presented. The procedure uses samples of both the solution of the HJB equation and its gradient to obtain a tensor train approximation of the value function. The collection of the data for the algorithm is based on two possible techniques: Pontryagin Maximum Principle and State Dependent Riccati Equations. Several numerical tests are presented in low and high dimension showing the effectiveness o…
Solving a class of stochastic optimal control problems by physics-informed neural networks
School of Mathematics and Statistics（数学与统计学学院）；MOE Key Laboratory for Complexity Science in Aerospace（复杂科学在航空航天中的关键实验室）；State Key Laboratory of Fluid Power and Mechatronic Systems, Department of Mechanics（流体动力与机械系统国家重点实验室，力学系）
In this paper, we study accelerating a Laplacian-based dynamic average consensus algorithm by splitting the conventional delay-free disagreement feedback into weighted summation of a current and an outdated term. We determine for what weighted sum there exists a range of time delay that results in the higher rate of convergence for the algorithm. For such weights, using the Lambert W function, we obtain the rate increasing range of the time delay, the maximum reachable rate and comment on the value of the corresponding maximizer delay. We also study the effect of use of outdated feedback on th…
This paper studies the boundary output feedback stabilization of general 1-D reaction-diffusion PDEs in the presence of a state delay in the reaction term. The control input applies through a Robin boundary condition while the system output is selected as a either Dirichlet or Neumann boundary trace. The control strategy takes the form of a finite-dimensional observer-based controller with feedback and observer gains that are computed in order to dominate the state delayed term. For any arbitrarily given value of the state delay, we show the exponential stability of the resulting closed-loop s…
In black-box optimization problems, we aim to maximize an unknown objective function, where the function is only accessible through feedbacks of an evaluation or simulation oracle. In real-life, the feedbacks of such oracles are often noisy and available after some unknown delay that may depend on the computation time of the oracle. Additionally, if the exact evaluations are expensive but coarse approximations are available at a lower cost, the feedbacks can have multi-fidelity. In order to address this problem, we propose a generic extension of hierarchical optimistic tree search (HOO), calle…
This paper discusses the in-domain feedback stabilization of reaction-diffusion PDEs with Robin boundary conditions in the presence of an uncertain time- and spatially-varying delay in the distributed actuation. The proposed control design strategy consists of a constant-delay predictor feedback designed based on the known nominal value of the control input delay and is synthesized on a finite-dimensional truncated model capturing the unstable modes of the original infinite-dimensional system. By using a small-gain argument, we show that the resulting closed-loop system is exponentially stable…
This paper studies the infinite-horizon optimal consumption with a path-dependent reference under exponential utility. The performance is measured by the difference between the nonnegative consumption rate and a fraction of the historical consumption maximum. The consumption running maximum process is chosen as an auxiliary state process, and hence the value function depends on two state variables. The Hamilton-Jacobi-Bellman (HJB) equation can be heuristically expressed in a piecewise manner across different regions to take into account all constraints. By employing the dual transform and smo…
