In this paper we consider discrete and continuous time risk sensitive optimal stopping problem. Using suitable properties of the underlying Feller-Markov process we prove continuity of the optimal stopping value function and provide formula for the optimal stopping policy. Next, we show how to link continuous time framework with its discrete time analogue. By considering suitable approximations, we obtain uniform convergence of the corresponding value functions.
In this paper we consider stopping problems with partial observation under a general risk-sensitive optimization criterion for problems with finite and infinite time horizon. Our aim is to maximize the certainty equivalent of the stopping reward. We develop a general theory and discuss the Bayesian risk-sensitive house selling problem as a special example. In particular we are able to study the influence of the attitude towards risk of the decision maker on the optimal stopping rule.
In this paper we study optimal stopping problems with respect to distorted expectations of the form \begin{eqnarray*} \mathcal{E}(X)=\int_{-\infty}^{\infty} x\,dG(F_X(x)), \end{eqnarray*} where $F_X$ is the distribution function of $X$ and $G$ is a convex distribution function on $[0,1].$ As a matter of fact, except for $G$ being the identity on $[0,1],$ dynamic versions of $\mathcal{E}(X)$ do not have the so-called time-consistency property necessary for the dynamic programming approach. So the standard approaches are not applicable to optimal stopping under $\mathcal{E}(X).$ In this paper, w…
In this paper we consider an infinite time horizon risk-sensitive optimal stopping problem for a Feller--Markov process with an unbounded terminal cost function. We show that in the unbounded case an associated Bellman equation may have multiple solutions and we give a probabilistic interpretation for the minimal and the maximal one. Also, we show how to approximate them using finite time horizon problems. The analysis, covering both discrete and continuous time case, is supported with illustrative examples.
The optimal stopping problem for the risk process with interests rates and when claims are covered immediately is considered. An insurance company receives premiums and pays out claims which have occured according to a renewal process and which have been recognized by them. The capital of the company is invested at interest rate $α\in\Re^{+}$, the size of claims increase at rate $β\in\Re^{+}$ according to inflation process. The immediate payment of claims decreases the company investment by rate $α_1$. The aim is to find the stopping time which maximizes the capital of the company. The improve…
We use martingale and stochastic analysis techniques to study a continuous-time optimal stopping problem, in which the decision maker uses a dynamic convex risk measure to evaluate future rewards. We also find a saddle point for an equivalent zero-sum game of control and stopping, between an agent (the "stopper") who chooses the termination time of the game, and an agent (the "controller", or "nature") who selects the probability measure.
We study the optimal stopping problem for a monotonous dynamic risk measure induced by a BSDE with jumps in the Markovian case. We show that the value function is a viscosity solution of an obstacle problem for a partial integro-differential variational inequality, and we provide an uniqueness result for this obstacle problem.
Bellman formulated a vague principle for optimization over time, which characterizes optimal policies by stating that a decision maker should not regret previous decisions retrospectively. This paper addresses time consistency in stochastic optimization. The problem is stated in generality first. The paper discusses time consistent decision-making by addressing risk measures which are recursive, nested, dynamically or time consistent and introduces stopping time risk measures. It turns out that the paradigm of time consistency is in conflict with various desirable, classical properties of gene…
We consider an investment problem in which an investor performs capital injections to increase the liquidity of a firm for it to maximise profit from market operations. Each time the investor performs an injection, the investor incurs a fixed transaction cost. In addition to maximising their terminal reward, the investor seeks to minimise the risk of loss of their investment (from a possible firm ruin) by exiting the market at some point in time. We show that the problem can be reformulated in terms of a new stochastic differential game of control and stopping in which one of the players modif…
We develop a theory for solving continuous time optimal stopping problems for non-linear expectations. Our motivation is to consider problems in which the stopper uses risk measures to evaluate future rewards.
Risk-sensitive control has received considerable interest since the seminal work of Howard and Matheson [120] because of its ability to account for fluctuations about the mean, its connection with $H_\infty$ control, and its application to financial mathematics. In this article, we attempt to put together a comprehensive survey on the research done on ergodic risk-sensitive control over the last four decades.
The paper solves constrained Dynkin games with risk-sensitive criteria, where two players are allowed to stop at two independent Poisson random intervention times, via the theory of backward stochastic differential equations. This generalizes the previous work of [Liang and Sun, Dynkin games with Poisson random intervention times, SIAM Journal on Control and Optimization, 2019] from the risk-neutral criteria and common signal times for both players to the risk-sensitive criteria and two heterogenous signal times. Furthermore, the paper establishes a connection of such constrained risk-sensitiv…
A multiplicative relative value iteration algorithm for solving the dynamic programming equation for the risk-sensitive control problem is studied for discrete time controlled Markov chains with a compact Polish state space, and controlled diffusions in on the whole Euclidean space. The main result is a proof of convergence to the desired limit in each case.
Zero sum games with risk-sensitive cost criterion are considered with underlying dynamics being given by controlled stochastic differential equations. Under the assumption of geometric stability on the dynamics , we completely characterize all possible saddle point strategies in the class of stationary Markov controls. In addition, we also establish existence-uniqueness result for the value function of the Hamilton-Jacobi-Isaacs equation.
A G/M/N queue is considered in the moderate deviation heavy traffic regime. The rate function for the customers-in-system process is obtained for the single class model. A risk-sensitive type control problem is considered for multi-class G/M/N model under the moderate deviation scaling and shown that the optimal control problem is related to a differential game problem.
We establish existence of controlled Markov chain of mean-field type with unbounded jump intensities by means of a fixed point argument using the Wasserstein distance. Using a Markov chain entropic backward SDE approach, we further suggest conditions for existence of an optimal control and a saddle-point for respectively a control problem and a zero-sum differential game associated with risk sensitive payoff functionals of mean-field type.
This paper is devoted to solving a time-inconsistent risk-sensitive control problem with parameter $\e$ and its limit case ($\e\rightarrow0^+$) for countable-stated Markov decision processes (MDPs for short). Since the cost functional is time-inconsistent, it is impossible to find a global optimal strategy for both cases. Instead, for each case, we will prove the existence of time-inconstant equilibrium strategies which verify the so-called step-optimality. Moreover, we prove the convergence of $\e$-equilibriums and the corresponding value functions as $\e\rightarrow0^+$.
We study sequences, parametrized by the number of agents, of many agent exit time stochastic control problems with risk-sensitive cost structure. We identify a fully characterizing assumption, under which each of such control problem corresponds to a risk-neutral stochastic control problem with additive cost, and sequentially to a risk-neutral stochastic control problem on the simplex, where the specific information about the state of each agent can be discarded. We also prove that, under some additional assumptions, the sequence of value functions converges to the value function of a determin…
We formulate a probabilistic Markov property in discrete time under a dynamic risk framework with minimal assumptions. This is useful for recursive solutions to risk-sensitive versions of dynamic optimisation problems such as optimal prediction, where at each stage the recursion depends on the whole future. The property holds for standard measures of risk used in practice, and is formulated in several equivalent versions including a representation via acceptance sets, a strong version, and a dual representation.
In this article, we prove the existence of optimal risk-sensitive control with state constraints. We use near monotone assumption on the running cost to prove the existence of optimal risk-sensitive control.
In this paper, we investigate dynamic optimization problems featuring both stochastic control and optimal stopping in a finite time horizon. The paper aims to develop new methodologies, which are significantly different from those of mixed dynamic optimal control and stopping problems in the existing literature, to study a manager's decision. We formulate our model to a free boundary problem of a fully nonlinear equation. Furthermore, by means of a dual transformation for the above problem, we convert the above problem to a new free boundary problem of a linear equation. Finally, we apply the …
We study a risk sensitive control version of the lifetime ruin probability problem. We consider a sequence of investments problems in Black-Scholes market that includes a risky asset and a riskless asset. We present a differential game that governs the limit behavior. We solve it explicitly and use it in order to find an asymptotically optimal policy.
We derive equivalent linear and dynamic programs for infinite horizon risk-sensitive control for minimization of the asymptotic growth rate of the cumulative cost.
We address the variational formulation of the risk-sensitive reward problem for non-degenerate diffusions on $\mathbb{R}^d$ controlled through the drift. We establish a variational formula on the whole space and also show that the risk-sensitive value equals the generalized principal eigenvalue of the semilinear operator. This can be viewed as a controlled version of the variational formulas for principal eigenvalues of diffusion operators arising in large deviations. We also revisit the average risk-sensitive minimization problem and by employing a gradient estimate developed in this paper, w…
In this paper we study a class of risk-sensitive Markovian control problems in discrete time subject to model uncertainty. We consider a risk-sensitive discounted cost criterion with finite time horizon. The used methodology is the one of adaptive robust control combined with machine learning.
In this paper, we analyze the convergence of a risk sensitive like filter where the risk sensitivity parameter is time varying. Such filter has a Kalman like structure and its gain matrix is updated according to a Riccati like iteration. We show that the iteration converges to a fixed point by using the contraction analysis.
Throughout this paper, we focused our aim on the problem of optimal control under a risk-sensitive performance functional, where the system is given by a fully coupled forward-backward stochastic differential equation with jump. The risk neutral control system has been used as preliminary step, where the admissible controls are convex, and the optimal solution exists. The necessary as well as sufficient optimality conditions for risk-sensitive performance are proved. At the end of this work, we illustrate our main result by giving an example of mean-variance for risk sensitive control problem …
This paper studies continuous-time Markov decision processes under the risk-sensitive average cost criterion. The state space is a finite set, the action space is a Borel space, the cost and transition rates are bounded, and the risk-sensitivity coefficient can take arbitrary positive real numbers. Under the mild conditions, we develop a new approach to establish the existence of a solution to the risk-sensitive average cost optimality equation and obtain the existence of an optimal deterministic stationary policy.
We consider a class of diffusions controlled through the drift and jump size, and driven by a jump Lévy process and a nondegenerate Wiener process, and we study infinite horizon (ergodic) risk-sensitive control problem for this model. We start with the controlled Dirichlet eigenvalue problem in smooth bounded domains, which also allows us to generalize current results in the literature on exit rate control problems. Then we consider the infinite horizon average risk-sensitive minimization problem and maximization problems on the whole domain. Under suitable hypotheses, we establish existence a…
This paper is concerned with an optimal investment problem under correlated noises in the financial market, and the expected utility functional is hyperbolic absolute risk aversion (HARA) with the exponent $γ\neq0$. The problem can be reformulated as a risk-sensitive stochastic control problem. A new stochastic maximum principle is obtained first, where the adjoint equations and maximum condition heavily depend on the risk-sensitive parameter and the correlation coefficient. The optimal investment strategy is obtained explicitly in a state feedback form via the solution to a certain Riccati eq…
In this article we consider risk-sensitive control of semi-Markov processes with a discrete state space. We consider general utility functions and discounted cost in the optimization criteria. We consider random finite horizon and infinite horizon problems. Using a state augmentation technique we characterise the value functions and also prescribe optimal controls.
In this paper, we consider risk-sensitive discounted control problem for continuous-time jump Markov processes taking values in general state space. The transition rates of underlying continuous-time jump Markov processes and the cost rates are allowed to be unbounded. Under certain Lyapunov condition, we establish the existence and uniqueness of the solution to the Hamilton-Jacobi-Bellman (HJB) equation. Also we prove the existence of optimal risk-sensitive control in the class of Markov control.
We study two person nonzero-sum stochastic differential games with risk-sensitive discounted and ergodic cost criteria. Under certain conditions we establish a Nash equilibrium in Markov strategies for the discounted cost criterion and a Nash equilibrium in stationary strategies for the ergodic cost criterion. We achieve our results by studying the relevant systems of coupled HJB equations.
The "value" of infinite horizon risk-sensitive control is the principal eigenvalue of a certain positive operator. This facilitates the use of Chang's extension of the Collatz-Wielandt formula to derive a variational characterization thereof. For the uncontrolled case, this reduces to the Donsker-Varadhan functional.
In this article, we study the ergodic risk-sensitive control problem for controlled regime-switching diffusions. Under a blanket stability hypothesis, we solve the associated nonlinear eigenvalue problem for weakly coupled systems and characterize the optimal stationary Markov controls via a suitable verification theorem. We also consider the near-monotone case and obtain the existence of principal eigenfunction and optimal stationary Markov controls.
We derive a variational formula for the optimal growth rate of reward in the infinite horizon risk-sensitive control problem for discrete time Markov decision processes with compact metric state and action spaces, extending a formula of Donsker and Varadhan for the Perron-Frobenius eigenvalue of a positive operator. This leads to a concave maximization formulation of the problem of determining this optimal growth rate.
We study data-driven decision-making problems in the Bayesian framework, where the expectation in the Bayes risk is replaced by a risk-sensitive entropic risk measure. We focus on problems where calculating the posterior distribution is intractable, a typical situation in modern applications with large datasets and complex data generating models. We leverage a dual representation of the entropic risk measure to introduce a novel risk-sensitive variational Bayesian (RSVB) framework for jointly computing a risk-sensitive posterior approximation and the corresponding decision rule. The proposed R…
In this article we consider zero and non-zero sum risk-sensitive average criterion games for semi-Markov processes with a finite state space. For the zero-sum case, under suitable assumptions we show that the game has a value. We also establish the existence of a stationary saddle point equilibrium. For the non-zero sum case, under suitable assumptions we establish the existence of a stationary Nash equilibrium.
We introduce the Lyapunov approach to optimal control problems of average risk-sensitive Markov control processes with general risk maps. Motivated by applications in particular to behavioral economics, we consider possibly non-convex risk maps, modeling behavior with mixed risk preference. We introduce classical objective functions to the risk-sensitive setting and we are in particular interested in optimizing the average risk in the infinite-time horizon for Markov Control Processes on general, possibly non-compact, state spaces allowing also unbounded cost. Existence and uniqueness of an op…
We consider a risk-sensitive continuous-time Markov decision process over a finite time duration. Under the conditions that can be satisfied by unbounded transition and cost rates, we show the existence of an optimal policy, and the existence and uniqueness of the solution to the optimality equation out of a class of possibly unbounded functions, to which the Feynman-Kac formula was also justified to hold.
We study nonzero-sum stochastic differential games with risk-sensitive ergodic cost criterion. Under certain conditions, using multi-parameter eigenvalue approach, we establish the existence of a Nash equilibrium in the space of stationary Markov strategies. We achieve our results by studying the relevant systems of coupled HJB equations. Exploiting the stochastic representation of the principal eigenfunctions we completely characterize Nash equilibrium points in the space of stationary Markov strategies.
There are no computationally feasible algorithms that provide solutions to the finite horizon Risk-sensitive Constrained Markov Decision Process (Risk-CMDP) problem, even for problems with moderate horizon. With an aim to design the same, we derive a fixed-point equation such that the optimal policy of Risk-CMDP is also a solution. We further provide two optimization problems equivalent to the Risk-CMDP. These formulations are instrumental in designing a global algorithm that converges to the optimal policy. The proposed algorithm is based on random restarts and a local improvement step, where…
This article is related to risk-sensitive nonzero-sum stochastic differential games in the Markovian framework. This game takes into account the attitudes of the players toward risk and the utility is of exponential form. We show the existence of a Nash equilibrium point for the game when the drift is no longer bounded and only satisfies a linear growth condition. The main tool is the notion of backward stochastic differential equation, which in our case, is multidimensional with continuous generator involving both a quadratic term of Z and a stochastic linear growth component with respect to …
In this paper we study a class of matrix-valued linear-quadratic mean-field-type games for both the risk-neutral, risk-sensitive and robust cases. Non-cooperation, full cooperation and adversarial between teams are treated. We provide a semi-explicit solution for both problems by means of a direct method. The state dynamics is described by a matrix-valued linear jump-diffusion-regime switching system of conditional mean-field type where the conditioning is with respect to common noise which is a regime switching process. The optimal strategies are in a state-and-conditional mean-field feedback…
We study infinite horizon discounted-cost and ergodic-cost risk-sensitive zero-sum stochastic games for controlled continuous time Markov chains on a countable state space. For the discounted-cost game we prove the existence of value and saddle-point equilibrium in the class of Markov strategies under nominal conditions. For the ergodic-cost game we prove the existence of values and saddle point equilibrium by studying the corresponding Hamilton-Jacobi-Isaacs equation under a certain Lyapunov condition.
We present a heuristic policy and performance bound for risk-sensitive convex stochastic control that generalizes linear-exponential-quadratic regulator (LEQR) theory. Our heuristic policy extends standard, risk-neutral model predictive control (MPC); however, instead of ignoring uncertain noise terms, our policy assumes these noise terms turns out either favorably or unfavorably, depending on a risk aversion parameter. In the risk-seeking case, this modified planning problem is convex. In the risk-averse case, it requires minimizing a difference of convex functions, which is done (approximate…
We study risk-sensitive control of continuous time Markov chains taking values in discrete state space. We study both finite and infinite horizon problems. In the finite horizon problem we characterise the value function via HJB equation and obtain an optimal Markov control. We do the same for infinite horizon discounted cost case. In the infinite horizon average cost case we establish the existence of an optimal stationary control under certain Lyapunov condition. We also develop a policy iteration algorithm for finding an optimal control.
We study zero-sum stochastic games for controlled discrete time Markov chains with risk-sensitive average cost criterion with countable state space and Borel action spaces. The payoff function is nonnegative and possibly unbounded. Under a certain Lyapunov stability assumption on the dynamics, we establish the existence of a value and saddle point equilibrium. Further we completely characterize all possible saddle point strategies in the class of stationary Markov strategies. Finally, we present and analyze an illustrative example.
Learning to Stop: Deep Learning for Mean Field Optimal Stopping
Shanghai Frontiers Science Center of Artificial Intelligence（上海人工智能前沿科学中心）；NYU-ECNU Institute of Mathematical Sciences（纽约大学-复旦大学数学科学研究院）；Georgia Institute of Technology（佐治亚理工学院）
We study risk-sensitive optimal control of a stochastic differential equation (SDE) of mean-field type, where the coefficients are allowed to depend on some functional of the law as well as the state and control processes. Moreover the risk-sensitive cost functional is also of mean-field type. We derive optimality equations in infinite dimensions connecting dual functions associated with Bellman functional to the adjoint process of the Pontryagin maximum principle. The case of linear-exponentiated quadratic cost and its connection with the risk-neutral solution is discussed.
