In this paper, we study a Markov decision process with a non-linear discount function and with a Borel state space. We define a recursive discounted utility, which resembles non-additive utility functions considered in a number of models in economics. Non-additivity here follows from non-linearity of the discount function. Our study is complementary to the work of Jaśkiewicz, Matkowski and Nowak (Math. Oper. Res. 38 (2013), 108-121), where also non-linear discounting is used in the stochastic setting, but the expectation of utilities aggregated on the space of all histories of the process is applied leading to a non-stationary dynamic programming model. Our aim is to prove that in the recursive discounted utility case the Bellman equation has a solution and there exists an optimal stationary policy for the problem in the infinite time horizon. Our approach includes two cases: $(a)$ when the one-stage utility is bounded on both sides by a weight function multiplied by some positive and negative constants, and $(b)$ when the one-stage utility is unbounded from below.
This paper shows the usefulness of Perov's contraction principle, which generalizes Banach's contraction principle to a vector-valued metric, for studying dynamic programming problems in which the discount factor can be stochastic. The discounting condition $β<1$ is replaced by $ρ(B)<1$, where $B$ is an appropriate nonnegative matrix and $ρ$ denotes the spectral radius. Blackwell's sufficient condition is also generalized in this setting. Applications to asset pricing and optimal savings are discussed.
This paper is devoted to a study of infinite horizon optimal control problems with time discounting and time averaging criteria in discrete time. We establish that these problems are related to certain infinite-dimensional linear programming (IDLP) problems. We also establish asymptotic relationships between the optimal values of problems with time discounting and long-run average criteria.
Adopting a probabilistic approach we determine the optimal dividend payout policy of a firm whose surplus process follows a controlled arithmetic Brownian motion and whose cash-flows are discounted at a stochastic dynamic rate. Dividends can be paid to shareholders at unrestricted rates so that the problem is cast as one of singular stochastic control. The stochastic interest rate is modelled by a Cox-Ingersoll-Ross (CIR) process and the firm's objective is to maximize the total expected flow of discounted dividends until a possible insolvency time. We find an optimal dividend payout policy wh…
Stochastic games with discounted payoff, introduced by Shapley, model adversarial interactions in stochastic environments where two players try to optimize a discounted sum of rewards. In this model, long-term weights are geometrically attenuated based on the delay in their occurrence. We propose a temporally dual notion -- called past-discounting -- where agents have geometrically decaying memory of the rewards encountered during a play of the game. We study objective functions based on past-discounted weight sequences and examine the corresponding stochastic games with liminf, discounted, an…
In this paper, we consider a general time-inconsistent optimal control problem for a non homogeneous linear system, in which its state evolves according to a stochastic differential equation with deterministic coefficients, when the noise is driven by a Brownian motion and an independent Poisson point process. The running and the terminal costs in the objective functional, are explicitly dependent on some general discounting coefficients which cover the non-exponential and the hyperbolic discounting situations. Furthermore, the presence of some quadratic terms of the conditional expectation of…
We consider infinite horizon optimal control problems with time averaging and time discounting criteria and give estimates for the Cesaro and Abel limits of their optimal values in the case when they depend on the initial conditions. We establish that these limits are bounded from above by the optimal value of a certain infinite dimensional (ID) linear programming (LP) problem and that they are bounded from below by the optimal value of the corresponding dual problem. (These estimates imply, in particular, that the Cesaro and Abel limits exist and are equal to each other if there is no duality…
In this paper, we study backward doubly stochastic recursive optimal control problem where the cost function is described by the solution of a backward doubly stochastic differential equation. We give the dynamical programming principle for this kind of optimal control problem and show that the value function is the unique Sobolev weak solution for the corresponding stochastic Hamilton-Jacobi-Bellman equation.
This paper considers the Merton portfolio management problem. We are concerned with non-exponential discounting of time and this leads to time inconsistencies of the decision maker. Following Ekeland and Pirvu 2006, we introduce the notion of equilibrium policies and we characterize them by an integral equation. The main idea is to come up with the value function in this context. If risk preferences are of CRRA type, the integral equation which characterizes the value function is shown to have a solution which leads to an equilibrium policy. This work is an extension of Ekeland and Pirvu 2006.
Abstract Dynamic Programming on Partially Ordered Spaces
Research School of Economics, Australian National University（澳大利亚国立大学经济研究学院）；National Graduate Institute for Policy Studies (GRIPS)（政策研究大学院大学）；School of Economics, University of Sydney（悉尼大学经济系）
In this paper, we study one kind of stochastic recursive optimal control problem with the obstacle constraints for the cost function where the cost function is described by the solution of one reflected backward stochastic differential equations. We will give the dynamic programming principle for this kind of optimal control problem and show that the value function is the unique viscosity solution of the obstacle problem for the corresponding Hamilton-Jacobi-Bellman equations.
The paper aims at the development of an apparatus for analysis and construction of near optimal solutions of singularly perturbed (SP) optimal controls problems (that is, problems of optimal control of SP systems) considered on the infinite time horizon. We mostly focus on problems with time discounting criteria but a possibility of the extension of results to periodic optimization problems is discussed as well. Our consideration is based on earlier results on averaging of SP control systems and on linear programming formulations of optimal control problems. The idea that we exploit is to firs…
We study the optimal control of general stochastic McKean-Vlasov equation. Such problem is motivated originally from the asymptotic formulation of cooperative equilibrium for a large population of particles (players) in mean-field interaction under common noise. Our first main result is to state a dynamic programming principle for the value function in the Wasserstein space of probability measures, which is proved from a flow property of the conditional law of the controlled state process. Next, by relying on the notion of differentiability with respect to probability measures due to P.L. Lion…
This paper build on our recent work where we presented a dual stochastic optimal control formulation of the nonlinear filtering problem [1]. The constraint for the dual problem is a backward stochastic differential equations (BSDE). The solution is obtained via an application of the maximum principle (MP). In the present paper, a dynamic programming (DP) principle is presented for a special class of BSDE-constrained stochastic optimal control problems. The principle is applied to derive the solution of the nonlinear filtering problem.
We analyze a zero-sum stochastic differential game between two competing players who can choose unbounded controls. The payoffs of the game are defined through backward stochastic differential equations. We prove that each player's priority value satisfies a weak dynamic programming principle and thus solves the associated fully non-linear partial differential equation in the viscosity sense.
This paper studies the dynamic programming principle for general convex stochastic optimization problems introduced by Rockafellar and Wets in [30]. We extend the applicability of the theory by relaxing compactness and boundedness assumptions. In the context of financial mathematics, the relaxed assumption are satisfied under the well-known no-arbitrage condition and the reasonable asymptotic elasticity condition of the utility function. Besides financial mathematics, we obtain several new results in linear and nonlinear stochastic programming and stochastic optimal control.
We extend the proof of the dynamic programming principle (DPP) for standard stochastic optimal control problems driven by general Lévy noises. Under appropriate assumptions, it is shown that the DPP still holds when the state process fails to have any moments at all.
Multiobjective stochastic programming is a field well located to tackle problems arising in emergencies, given that uncertainty and multiple objectives are usually present in such problems. A new concept of solution is proposed in this work, especially designed for risk-aversion solutions. A linear programming model is presented to obtain such solution.
A linear quadratic (LQ) stochastic optimization problem with delay involving weakly-coupled large population is investigated in this paper. Different to classic mean field (MF) game, here agents cooperate with each other to minimize the so-called \emph{social} objective. With the aid of \emph{delayed person-by-person optimality} principle, one arrives at an auxiliary LQ delayed control problem by decentralized information. A decentralized strategy is obtained by feat of an MF type anticipated forward-backward stochastic differential delay equation (AFBSDDE) consistency condition. The discounti…
In this paper, we propose a class of penalty methods with stochastic approximation for solving stochastic nonlinear programming problems. We assume that only noisy gradients or function values of the objective function are available via calls to a stochastic first-order or zeroth-order oracle. In each iteration of the proposed methods, we minimize an exact penalty function which is nonsmooth and nonconvex with only stochastic first-order or zeroth-order information available. Stochastic approximation algorithms are presented for solving this particular subproblem. The worst-case complexity of …
Multistage stochastic programming deals with operational and planning problems that involve a sequence of decisions over time while responding to realizations that are uncertain. Algorithms designed to address multistage stochastic linear programming (MSLP) problems often rely upon scenario trees to represent the underlying stochastic process. When this process exhibits stagewise independence, sampling-based techniques, particularly the stochastic dual dynamic programming (SDDP) algorithm, have received wide acceptance. However, these sampling-based methods still operate with a deterministic r…
We present an algorithm called Tropical Dynamic Programming (TDP) which builds upper and lower approximations of the Bellman value functions in risk-neutral Multistage Stochastic Programming (MSP), with independent noises of finite supports. To tackle the curse of dimensionality, popular parametric variants of Approximate Dynamic Programming approximate the Bellman value function as linear combinations of basis functions. Here, Tropical Dynamic Programming builds upper (resp. lower) approximations of a given value function as min-plus linear (resp. max-plus linear) combinations of "basic funct…
This paper develops new extremal principles of variational analysis that are motivated by applications to constrained problems of stochastic programming and semi-infinite programming without smoothness and/or convexity assumptions. These extremal principles concern measurable set-valued mappings/multifunctions with values in finite-dimensional spaces and are established in both approximate and exact forms. The obtained principles are instrumental to derive via variational approaches integral representations and upper estimates of regular and limiting normals cones to essential intersections of…
The paper [12] examines a concept of equilibrium policies instead of optimal controls in stochastic optimization to analyze a mean-variance portfolio selection problem. We follow the same approach in order to investigate the Merton portfolio management problem in the context of non-exponential discounting, a context that give rise to time-inconsistency of the decision maker. Equilibrium policies are characterized in this context by means of a variational method which leads to a stochastic system that consists of a flow of forward-backward stochastic differential equations and an equilibrium co…
Measuring and managing risk has become crucial in modern decision making under stochastic uncertainty. In two-stage stochastic programming, mean risk models are essentially defined by a parametric recourse problem and a quantification of risk. From the perspective of qualitative robustness theory, we discuss sufficient conditions for continuity of the resulting objective functions with respect to perturbation of the underlying probability measure. Our approach covers a fairly comprehensive class of both stochastic-programming related risk measures and relevant recourse models. Not only this un…
We propose primal-dual stochastic mirror descent for the convex optimization problems with functional constraints. We obtain the rate of convergence in terms of probability of large deviations.
The aim of this paper is to show that in some cases risk averse multistage stochastic programming problems can be reformulated in a form of risk neutral setting. This is achieved by a change of the reference probability measure making ``bad" (extreme) scenarios more frequent. As a numerical example we demonstrate advantages of such change-of-measure approach applied to the Brazilian Interconnected Power System operation planning problem.
We introduce an extension of Dual Dynamic Programming (DDP) to solve convex nonlinear dynamic programming equations. We call Inexact DDP (IDDP) this extension which applies to situations where some or all primal and dual subproblems to be solved along the iterations of the method are solved with a bounded error. We show that any accumulation point of the sequence of decisions is an approximate solution to the dynamic programming equations. When these errors tend to zero as the number of iterations goes to infinity, we show that IDDP solves the dynamic programming equations. We extend the analy…
In this paper, we study a stochastic optimal control problem under degenerate G-expectation. By using implied partition method, we show that the approximation result for admissible controls still hold. Based on this result, we prove that the value function is deterministic, and obtain the dynamic programming principle. Furthermore, we prove that the value function is the unique viscosity solution to the related HJB equation under degenerate case.
In this paper, we study a stochastic recursive optimal control problem in which the cost functional is described by the solution of a backward stochastic differential equation driven by G-Brownian motion. Under standard assumptions, we establish the dynamic programming principle and the related fully nonlinear HJB equation in the framework of G-expectation. Finally, we show that the value function is the viscosity solution of the obtained HJB equation.
We introduce the class of multistage stochastic optimization problems with a random number of stages. For such problems, we show how to write dynamic programming equations and detail the Stochastic Dual Dynamic Programming algorithm to solve these equations. Finally, we consider a portfolio selection problem over an optimization period of random duration. For several instances of this problem, we show the gain obtained using a policy that takes the random duration of the number of stages into account over a policy built taking a fixed number of stages (namely the maximal possible number of sta…
This paper investigates an infinite-horizon linear quadratic stochastic (LQS) optimal control problem for a class of continuous-time stochastic systems. By employing the technique of adaptive dynamic programming (ADP), we propose a novel model-free policy iteration (PI) algorithm. Without needing all information of the system coefficient matrices, the proposed PI algorithm iterates by using the data of the input and system state collected on a fixed time interval. Finally, a numerical example is presented to demonstrate the feasibility of the obtained algorithm.
This paper deals with the stochastic control of nonlinear systems in the presence of state and control constraints, for uncertain discrete-time dynamics in finite dimensional spaces. In the deterministic case, the viability kernel is known to play a basic role for the analysis of such problems and the design of viable control feedbacks. In the present paper, we show how a stochastic viability kernel and viable feedbacks relying on probability (or chance) constraints can be defined and computed by a dynamic programming equation. An example illustrates most of the assertions.
This paper aims to explore the relationship between maximum principle and dynamic programming principle for stochastic recursive control problem with random coefficients. Under certain regular conditions for the coefficients, the relationship between the Hamilton system with random coefficients and stochastic Hamilton-Jacobi-Bellman equation is obtained. It is very different from the deterministic coefficients case since stochastic Hamilton-Jacobi-Bellman equation is a backward stochastic partial differential equation with solution being a pair of random fields rather than a deterministic func…
In this paper we assume the insurance wealth process is driven by the compound Poisson process. The discounting factor is modelled as a geometric Brownian motion at first and then as an exponential function of an integrated Ornstein-Uhlenbeck process. The objective is to maximize the cumulated value of expected discounted dividends up to the time of ruin. We give an explicit expression of the value function and the optimal strategy in the case of interest rate following a geometric Brownian motion. For the case of the Vasicek model, we explore some properties of the value function. Since we ca…
We define a regularized variant of the Dual Dynamic Programming algorithm called REDDP (REgularized Dual Dynamic Programming) to solve nonlinear dynamic programming equations. We extend the algorithm to solve nonlinear stochastic dynamic programming equations. The corresponding algorithm, called SDDP-REG, can be seen as an extension of a regularization of the Stochastic Dual Dynamic Programming (SDDP) algorithm recently introduced which was studied for linear problems only and with less general prox-centers. We show the convergence of REDDP and SDDP-REG. We assess the performance of REDDP and …
We consider an individual or household endowed with an initial capital and an income, modeled as a deterministic process with a continuous drift rate. At first, we model the discounting rate as the price of a zero-coupon bond at zero under the assumption of a short rate evolving as an Ornstein-Uhlenbeck process. Then, a geometric Brownian motion as the preference function and an Ornstein-Uhlenbeck process as the short rate are taken into consideration. It is assumed that the primal interest of the economic agent is to maximise the cumulated value of (expected) discounted consumption from a giv…
In this paper, we extend the adaptive partition-based approach for solving two-stage stochastic programs with fixed recourse to the multistage stochastic programming setting. The proposed algorithms integrate the adaptive partition-based strategy with a popular approach for solving multistage stochastic programs, the stochastic dual dynamic programming, via different tree-traversal strategies in order to enhance its computational efficiency. Our numerical experiments on a hydro-thermal power generation planning problem show the effectiveness of the proposed algorithms.
We introduce an extension of Dual Dynamic Programming (DDP) to solve linear dynamic programming equations. We call this extension IDDP-LP which applies to situations where some or all primal and dual subproblems to be solved along the iterations of the method are solved with a bounded error (inexactly). We provide convergence theorems both in the case when errors are bounded and for asymptotically vanishing errors. We extend the analysis to stochastic linear dynamic programming equations, introducing Inexact Stochastic Dual Dynamic Programming for linear programs (ISDDP-LP), an inexact variant…
A Sequential Quadratic Programming Method for Optimization with Stochastic Objective Functions, Deterministic Inequality Constraints and Robust Subproblems
We use a rank one Gaussian perturbation to derive a smooth stochastic approximation of the maximum eigenvalue function. We then combine this smoothing result with an optimal smooth stochastic optimization algorithm to produce an efficient method for solving maximum eigenvalue minimization problems. We show that the complexity of this new method is lower than that of deterministic smoothing algorithms in certain precision/dimension regimes.
We introduce an extension of Stochastic Dual Dynamic Programming (SDDP) to solve stochastic convex dynamic programming equations. This extension applies when some or all primal and dual subproblems to be solved along the forward and backward passes of the method are solved with bounded errors (inexactly). This inexact variant of SDDP is described both for linear problems (the corresponding variant being denoted by ISDDP-LP) and nonlinear problems (the corresponding variant being denoted by ISDDP-NLP). We prove convergence theorems for ISDDP-LP and ISDDP-NLP both for bounded and asymptotically …
The main goal of this paper is to apply the machinery of variational analysis and generalized differentiation to study infinite horizon stochastic dynamic programming (DP) with discrete time in the Banach space setting without convexity assumptions. Unlike to standard stochastic DP with stationary Markov processes, we investigate here stochastic DP in $L^p$ spaces to deal with nonstationary stochastic processes, which describe a more flexible learning procedure for the decision-maker. Our main concern is to calculate generalized subgradients of the corresponding value function and to derive ne…
A stagewise decomposition algorithm called value function gradient learning (VFGL) is proposed for large-scale multistage stochastic convex programs. VFGL finds the parameter values that best fit the gradient of the value function within a given parametric family. Widely used decomposition algorithms for multistage stochastic programming, such as stochastic dual dynamic programming (SDDP), approximate the value function by adding linear subgradient cuts at each iteration. Although this approach has been successful for linear problems, nonlinear problems may suffer from the increasing size of e…
We consider the infinite dimensional linear programming (inf-LP) approach for solving stochastic control problems. The inf-LP corresponding to problems with uncountable state and input spaces is in general computationally intractable. By focusing on linear systems with quadratic cost (LQG), we establish a connection between this approach and the well-known Riccati LMIs. In particular, we show that the semidefinite programs known for the LQG problem can be derived from the pair of primal and dual inf-LPs. Furthermore, we establish a connection between multi-objective and chance constraint crite…
