Using the tools of the Markov Decision Processes, we justify the dynamic programming approach to the optimal impulse control of deterministic dynamical systems. We prove the equivalence of the integral and differential forms of the optimality equation. The theory is illustrated by an example from mathematical epidemiology. The developed methods can be also useful for the study of piecewise deterministic Markov processes.
We consider the optimal control problem of minimizing some quadratic functional over all possible solutions of an internally controlled multi-dimensional heat equation with a periodic terminal state constraint. This problem has a unique optimal solution, which can be characterized by an optimality system derived from the Pontryagin maximum principle. We define two approximations of this optimal control problem. The first one is an impulse approximation, and consists of considering a system of linear heat equations with impulse control. The second one is obtained by the sample-and-hold procedur…
We consider an optimal control problem for a system governed by a Volterra integral equation with impulsive terms. The impulses act on both the state and the control; the control consists of switchings at discrete times. The cost functional includes both, an integrated cost rate (continuous part) and switching costs at the discrete impulse times (discrete part). We prove necessary optimality conditions of a form analogous to a discrete maximum principle. For the particular case of a system governed by impulsive ordinary differential equations, we obtain an impulsive maximum principle as a spec…
We discuss the problem of optimal impulse control representing the preventive maintenance of a simple reparable system. The system model is governed by coupled transport and integro-differential equations in a nonreflexive Banach space. The objective of this paper is to construct nonnegative impulse control inputs at given system running times that minimize the probability of the system in failure mode. To guarantee the nonnegativity of the controlled system, we consider the control inputs to depend on the system state. This essentially leads to a bilinear control problem. We first present a r…
This paper considers an optimal impulse control problem of dynamical systems generated by a flow. The performance criteria are total costs over the infinite time horizon. Apart from the main performance to be minimized, there are multiple constraints on performance functionals of a similar type. Under a natural set of compactness-continuity conditions on the system primitives, we establish a linear programming approach, and prove the existence of a stationary optimal control strategy out of a more general class of randomized strategies. This is done by making use of the tools from Markov decis…
Deterministic optimal impulse control problem with terminal state constraint is considered. Due to the appearance of the terminal state constraint, the value function might be discontinuous in general. The main contribution of this paper is the introduction of an intrinsic condition under which the value function is continuous. Then by a Bellman dynamic programming method, the corresponding Hamilton-Jacobi-Bellman type quasi-variational inequality (QVI, for short) is derived for which the value function is a viscosity solution. The issue of whether the value function is characterized as the un…
We obtain necessary conditions of optimality for impulsive Volterra integral equations with switching and impulsive controls, with variable impulse time-instants. The present work continues and complements our previous work on impulsive Volterra control with fixed impulse times.
The paper is concerned with a kind of minimal time control problem for the heat equation with impulse controls. The purpose of such a problem is to find an optimal impulse control (among certain control constraint set) steering the solution of the heat equation from a given initial state to a given target set as soon as possible. We will first study the existence and uniqueness of optimal solution for this problem. In the formulation of this problem, there are two parameters: one is the upper bound of the control constraint and the other one is the moment of impulse time. Then, we will establi…
For a constrained optimal impulse control problem of an abstract dynamical system, we introduce the occupation measures along with aggregated occupation measures and present two associated linear programs. We prove that the two linear programs are equivalent under appropriate conditions, and each linear program gives rise to an optimal strategy in the original impulse control problem.
This paper establishes a verification theorem for impulse control problems involving conditional McKean-Vlasov jump diffusions. We obtain a Markovian system by combining the state equation of the problem with the stochastic Fokker-Planck equation for the conditional probability law of the state. We derive sufficient variational inequalities for a function to be the value function of the impulse control problem, and for an impulse control to be the optimal control. We illustrate our results by applying them to the study of an optimal stream of dividends under transaction costs. We obtain the so…
Families of Two-Impulse Optimal Rendezvous Transfers Between Elliptic Orbits
School of Aeronautics and Astronautics, Purdue University（普渡大学航空航天学院）；Department of Aerospace Engineering, Korea Advanced Institute of Science and Technology（韩国科学技术院航空航天工程系）
This paper studies an optimal stochastic impulse control problem in a finite horizon with a decision lag, by which we mean that after an impulse is made, a fixed number units of time has to be elapsed before the next impulse is allowed to be made. The continuity of the value function is proved. A suitable version of dynamic programming principle is established, which takes into account the dependence of state process on the elapsed time. The corresponding Hamilton-Jacobi-Bellman (HJB) equation is derived, which exhibit some special feature of the problem. The value function of this optimal imp…
In this paper we consider impulse control of continuous time Markov processes with average cost per unit time functional. This problem is approximated using impulse control problems stopped at the first exit time from increasing sequence of open sets. We find solution to Bellman equation corresponding to the original problem and show that stopped impulse control problems approximate optimal value of the cost functional.
In this paper, we consider joint drift rate control and impulse control for a stochastic inventory system under long-run average cost criterion. Assuming the inventory level must be nonnegative, we prove that a $\{(0,q^{\star},Q^{\star},S^{\star}),\{μ^{\star}(x): x\in[0, S^{\star}]\}\}$ policy is an optimal joint control policy, where the impulse control follows the control band policy $(0,q^{\star},Q^{\star},S^{\star})$, that brings the inventory level up to $q^{\star}$ once it drops to $0$ and brings it down to $Q^{\star}$ once it rises to $S^{\star}$, and the drift rate only depends on the …
A method to compute optimal collision avoidance maneuvers for short-term encounters is presented. The maneuvers are modeled as multiple-impulses to handle impulsive cases and to approximate finite burn arcs associated either with short alert times or the use of low-thrust propulsion. The maneuver design is formulated as a sequence of convex optimization problems solved in polynomial time by state-of-the-art primal-dual interior-point algorithms. The proposed approach calculates optimal solutions without assumptions about the thrust arc structure and thrust direction. The execution time is frac…
The purpose of this paper is to investigate the role that the continuous-time generalised Riccati equation plays within the context of singular linear-quadratic optimal control. This equation has been defined following the analogy with the discrete-time generalised Riccati equation, but, differently from the discrete case, to date the importance of this equation in the context of optimal control is yet to be understood. This note addresses this point. We show in particular that when the continuous-time generalised Riccati equation admits a symmetric solution, the corresponding linear-quadratic…
We consider a class of stochastic impulse control problems of general stochastic processes i.e. not necessarily Markovian. Under fairly general conditions we establish existence of an optimal impulse control. We also prove existence of combined optimal stochastic and impulse control of a fairly general class of diffusions with random coefficients. Unlike, in the Markovian framework, we cannot apply quasi-variational inequalities techniques. We rather derive the main results using techniques involving reflected BSDEs and the Snell envelope.
We consider an investment problem in which an investor performs capital injections to increase the liquidity of a firm for it to maximise profit from market operations. Each time the investor performs an injection, the investor incurs a fixed transaction cost. In addition to maximising their terminal reward, the investor seeks to minimise the risk of loss of their investment (from a possible firm ruin) by exiting the market at some point in time. We show that the problem can be reformulated in terms of a new stochastic differential game of control and stopping in which one of the players modif…
We consider impulse control of stochastic functional differential equations (SFDEs) driven by Lévy processes under an additional $L^p$-Lipschitz condition on the coefficients. Our results, which are first derived for a general stochastic optimization problem over infinite horizon impulse controls and then applied to the case of a controlled SFDE, apply to the infinite horizon as well as the random horizon settings. The methodology employed to show existence of optimal controls is a probabilistic one based on the concept of Snell envelopes.
We consider stochastic impulse control problems where the process is driven by a general one-dimensional diffusion. We shall show a new mathematical characterization of the value function as a linear function in a certain transformed space. Our approach can relieve us from the burden of guessing and proving the optimal strategy, (2) present a simple method to find the value function and the corresponding control policies, and (3) handle systematically a broader class of reward and cost functions since the existence of the value function can be shown in much a simpler way.
Optimal Impulse Control of SIR Epidemics over Scale-Free Networks
St. Petersburg State University（圣彼得堡国立大学）；Department of Electrical and Computer Engineering, Tandon School of Engineering, New York University（纽约大学坦登工程学院电气与计算机工程系）
This paper solves a Bayes sequential impulse control problem for a diffusion, whose drift has an unobservable parameter with a change point. The partially-observed problem is reformulated into one with full observations, via a change of probability measure which removes the drift. The optimal impulse controls can be expressed in terms of the solutions and the current values of a Markov process adapted to the observation filtration. We shall illustrate the application of our results using the Longstaff-Schwartz algorithm for multiple optimal stopping times in a geometric Brownian motion stock p…
This paper studies a continuous-review backlogged inventory model considered by Helmes et al. (2015) but with discontinuous quantity-dependent setup cost for each order. In particular, the setup cost is characterized by a two-step function and a higher cost would be charged once the order quantity exceeds a threshold $Q$. Unlike the optimality of $(s,S)$-type policy obtained by Helmes et al. (2015) for continuous setup cost with the discounted cost criterion, we find that, in our model, although some $(s,S)$-type policy is indeed optimal in some cases, the $(s,S)$-type policy can not always be…
In this paper we consider long-run risk sensitive average cost impulse control applied to a continuous-time Feller-Markov process. Using the probabilistic approach, we show how to get a solution to a suitable continuous-time Bellman equation and link it with the impulse control problem. The optimal strategy for the underlying problem is constructed as a limit of dyadic impulse strategies by exploiting regularity properties of the linked risk sensitive optimal stopping value functions. In particular, this shows that the discretized setting could be used to approximate near optimal strategies fo…
We consider an optimal stochastic impulse control problem over an infinite time horizon motivated by a model of irreversible investment choices with fixed adjustment costs. By employing techniques of viscosity solutions and relying on semiconvexity arguments, we prove that the value function is a classical solution to the associated quasi-variational inequality. This enables us to characterize the structure of the continuation and action regions and construct an optimal control. Finally, we focus on the linear case, discussing, by a numerical analysis, the sensitivity of the solution with resp…
ENSAE IP Paris, Center for Research in Economics and Statistics（巴黎高等统计与行政学院，经济与统计研究中心）；UC Berkeley, Department of Industrial engineering and Operations Research（加州大学伯克利分校，工业工程与运筹学系）；Laboratoire Manceau de Mathématiques, Institut du Risque et de l’Assurance, Le Mans Université（勒芒大学，风险与保险研究所，马恩河数学实验室）
We pose the deterministic, nonparametric, approximation problem for scalar nonnegative input/output systems via finite impulse response convolutions, based on repeated observations of input/output signal pairs. The problem is converted into a nonnegative matrix factorization with special structure for which we use Csiszár's I-divergence as the criterion of optimality. Conditions are given, on the input/output data, that guarantee the existence and uniqueness of the minimum. We propose a standard algorithm of the alternating minimization type for I-divergence minimization, and study its asympto…
Long-Term Average Impulse and Singular Control of a Growth Model with Two Revenue Sources
Institute for Operations Research, Humboldt University of Berlin（柏林洪堡大学运筹学研究所）；Department of Mathematical Sciences, University of Wisconsin-Milwaukee（威斯康星大学密尔沃基分校数学科学系）
For an insurance company with reserve modeled by the spectrally negative Lévy process, we study the optimal impulse dividend maximizing the expected accumulated net dividend payment subtracted by the accumulated cost of injecting capital. In this setting, the beneficiary of the dividends injects capital to ensure a non-negative risk process so that the insurer never goes bankrupt. The optimal impulse dividend and capital injection strategy together with its value function are obtained.
Optimal Reinsurance-Dividend Strategy with Fixed Transaction Costs in a Regime-Switching Brownian Risk Model: A Viscosity Solution to the Impulse Control Problem
Fujian Normal University（福建师范大学）；The Hong Kong Polytechnic University（香港理工大学）；Xi’an Jiaotong University（西安交通大学）
The concept of turnpike connects the solution of long but finite time horizon optimal control problems with steady state optimal controls. A key ingredient of the analysis of the turnpike is the linear quadratic regulator problem and the convergence of the solution of the associated differential Riccati equation as the terminal time approaches infinity. This convergence has been investigated in linear systems theory in the 1980s. We extend classical system theoretic results for the investigation of turnpike properties of standard state space systems and descriptor systems. We present condition…
In this paper, we investigate the optimization of mutual proportional reinsurance --- a mutual reserve system that is intended for the collective reinsurance needs of homogeneous mutual members, such as P&I Clubs in marine mutual insurance and reserve banks in the U.S. Federal Reserve. Compared to general (non-mutual) insurance models, which involve one-sided impulse control (i.e., either downside or upside impulse) of the underlying insurance reserve process that is required to be positive, a mutual insurance differs in allowing two-sided impulse control (i.e., both downside and upside impuls…
Trading in CEXs and DEXs with Priority Fees and Stochastic Delays
CEREMADE, Université Paris Dauphine-PSL（巴黎第九大学-CEREMADE）；CMAP, École Polytechnique（巴黎综合理工学院-CMAP）；Mathematical Institute, University of Oxford（牛津大学数学研究所）；Oxford-Man Institute of Quantitative Finance, University of Oxford（牛津大学牛津曼量化金融研究所）
This paper proposes penalty schemes for a class of weakly coupled systems of Hamilton-Jacobi-Bellman quasi-variational inequalities (HJBQVIs) arising from stochastic hybrid control problems of regime-switching models with both continuous and impulse controls. We show that the solutions of the penalized equations converge monotonically to those of the HJBQVIs. We further establish that the schemes are half-order accurate for HJBQVIs with Lipschitz coefficients, and first-order accurate for equations with more regular coefficients. Moreover, we construct the action regions and optimal impulse co…
We discuss a class of explicitly solvable mean field type control problems/mean field games with a clear economic interpretation. More precisely, we consider long term average impulse control problems with underlying general one-dimensional diffusion processes motivated by optimal harvesting problems in natural resource management. We extend the classical stochastic Faustmann models by allowing the prices to depend on the state of the market using a mean field structure. In a competitive market model, we prove that, under natural conditions, there exists an equilibrium strategy of threshold-ty…
Non-smooth dynamics driven by stochastic disturbance arise in a wide variety of engineering problems. Impulsive interventions are often employed to control stochastic systems; however, the modeling and analysis subject to execution delay have been less explored. In addition, continuously receiving information of the dynamics is not always possible. In this paper, with an application to an environmental restoration problem, a continuous-time stochastic impulse control problem subject to execution delay under discrete and random observations is newly formulated and analyzed. The dynamics have a …
Environments with fixed adjustment costs such as transaction costs or \lq menu costs\rq$ $ are widespread within economic systems. The presence of fixed minimal adjustment costs produces adjustment stickiness so that agents must choose a sequence of points at which time to perform their actions. This paper performs an analysis of the effect of transaction costs on agent behaviour within a dynamic optimisation problem by way of introducing the theory of incentive design to optimal stochastic impulse control. The setup consists of an agent that maximises their utility by performing a sequence of…
We consider a general class of nonzero-sum $N$-player stochastic games with impulse controls, where players control the underlying dynamics with discrete interventions. We adopt a verification approach and provide sufficient conditions for the Nash equilibria (NEs) of the game. We then consider the limit situation of $N \to \infty$, that is, a suitable mean-field game (MFG) with impulse controls. We show that under appropriate technical conditions, the existence of unique NE solution to the MFG, which is an $ε$-NE approximation to the $N$-player game, with $ε=O\left(\frac{1}{\sqrt{N}}\right)$.…
We consider an optimal switching problem where the terminal reward depends on the entire control trajectory. We show existence of an optimal control by applying a probabilistic technique based on the concept of Snell envelopes. We then apply this result to solve an impulse control problem for stochastic delay differential equations driven by a Brownian motion and an independent compound Poisson process. Furthermore, we show that the studied problem arises naturally when maximizing the revenue from operation of a group of hydro-power plants with hydrological coupling.
