This paper attempts to study the optimal stopping time for semi-Markov processes (SMPs) under the discount optimization criteria with unbounded cost rates. In our work, we introduce an explicit construction of the equivalent semi-Markov decision processes (SMDPs). The equivalence is embodied in the value functions of SMPs and SMDPs, that is, every stopping time of SMPs can induce a policy of SMDPs such that the value functions are equal, and vice versa. The existence of the optimal stopping time of SMPs is proved by this equivalence relation. Next, we give the optimality equation of the value function and develop an effective iterative algorithm for computing it. Moreover, we show that the optimal and ε-optimal stopping time can be characterized by the hitting time of the special sets. Finally, to illustrate the validity of our results, an example of a maintenance system is presented in the end.
According to conventional wisdom, ambiguity accelerates optimal timing by decreasing the value of waiting in comparison with the unambiguous benchmark case. We study this mechanism in a multidimensional setting and show that in a multifactor model ambiguity does not only influence the rate at which the underlying processes are expected to grow, it also affects the rate at which the problem is discounted. This mechanism where nature also selects the rate at which the problem is discounted cannot appear in a one-dimensional setting and as such we identify an indirect way of how ambiguity affects…
We study an infinite horizon optimal stopping problem which arises naturally in the optimal timing of a firm/project sale or in the valuation of natural resources: the functional to be maximised is a sum of a discounted running reward and a discounted final reward. The running and final rewards as well as the instantaneous interest rate (used for calculating discount factors) depend on a Feller-Markov process modelling the underlying randomness in the world, such as prices of stocks, prices of natural resources (gas, oil, etc.), or factors influencing the interest rate. For years, it had seeme…
Consider a set of discounted optimal stopping problems for a one-parameter family of objective functions and a fixed diffusion process, started at a fixed point. A standard problem in stochastic control/optimal stopping is to solve for the problem value in this setting. In this article we consider an inverse problem; given the set of problem values for a family of objective functions, we aim to recover the diffusion. Under a natural assumption on the family of objective functions we can characterise existence and uniqueness of a diffusion for which the optimal stopping problems have the specif…
In this paper, we consider the optimal stopping problem on semi-Markov processes (SMPs) with finite horizon, and aim to establish the existence and computation of optimal stopping times. To achieve the goal, we first develop the main results of finite horizon semi-Markov decision processes (SMDPs) to the case with additional terminal costs, introduce an explicit construction of SMDPs, and prove the equivalence between the optimal stopping problems on SMPs and SMDPs. Then, using the equivalence and the results on SMDPs developed here, we not only show the existence of optimal stopping time of S…
We consider Backward Stochastic Differential Equations in a setting where noise is generated by a countable state, continuous time Markov chain, and the terminal value is prescribed at a stopping time. We show that, given sufficient integrability of the stopping time and a growth bound on the terminal value and BSDE driver, these equations admit unique solutions satisfying the same growth bound (up to multiplication by a constant). This holds without assuming that the driver is monotone in y, that is, our results do not require that the terminal value be discounted at some uniform rate. We sho…
The paper is concerned with a variant of the continuous-time finite state Markov game of control and stopping where both players can affect transition rates, while only one player can choose a stopping time. We use the dynamic programming principle and reduce this problem to a system of ODEs with unilateral constraints. This system plays the role of the Bellman equation. We show that its solution provides the optimal strategies of the players. Additionally, we prove the existence and uniqueness theorem for the deduced system of ODEs with unilateral constraints.
This work considers two-player zero-sum semi-Markov games with incomplete information on one side and perfect observation. At the beginning, the system selects a game type according to a given probability distribution and informs to Player 1 only. After each stage, the actions chosen are observed by both players before proceeding to the next stage. Firstly, we show the existence of the value function under the expected discount criterion and the optimality equation. Secondly, the existence and iterative algorithm of the optimal policy for Player 1 are introduced through the optimality equation…
Previous authors have considered optimal stopping problems driven by the running maximum of a spectrally negative Lévy process $X$, as well as of a one-dimensional diffusion. Many of the aforementioned results are either implicitly or explicitly dependent on Peskir's maximality principle. In this article, we are interested in understanding how some of the main ideas from these previous works can be brought into the setting of problems driven by the maximum of a class of Markov additive processes (more precisely Markov modulated Lévy processes). Similarly to previous works in the Lévy setting, …
We consider a non-Markovian optimal stopping problem on finite horizon. We prove that the value process can be represented by means of a backward stochastic differential equation (BSDE), defined on an enlarged probability space, containing a stochastic integral having a one-jump point process as integrator and an (unknown) process with a sign constraint as integrand. This provides an alternative representation with respect to the classical one given by a reflected BSDE. The connection between the two BSDEs is also clarified. Finally, we prove that the value of the optimal stopping problem is t…
Standard Markovian optimal stopping problems are consistent in the sense that the first entrance time into the stopping set is optimal for each initial state of the process. Clearly, the usual concept of optimality cannot in a straightforward way be applied to non-standard stopping problems without this time-consistent structure. This paper is devoted to the solution of time-inconsistent stopping problems with the reward depending on the initial state using an adaptation of Strotz's consistent planning. More precisely, we give a precise equilibrium definition --- of the type subgame perfect Na…
This paper is devoted to studying constrained continuous-time Markov decision processes (MDPs) in the class of randomized policies depending on state histories. The transition rates may be unbounded, the reward and costs are admitted to be unbounded from above and from below, and the state and action spaces are Polish spaces. The optimality criterion to be maximized is the expected discounted rewards, and the constraints can be imposed on the expected discounted costs. First, we give conditions for the nonexplosion of underlying processes and the finiteness of the expected discounted rewards/c…
We describe a new construction of a family of measures on a group with the same Poisson boundary. Our approach is based on applying Markov stopping times to an extension of the original random walk.
We consider a general $k$ dimensional discounted infinite server queues process (alternatively, an Incurred But Not Reported (IBNR) claim process) where the multivariate inputs (claims) are given by a $k$ dimensional finite state Markov Chain and the arrivals follow a renewal process. After deriving a multidimensional integral equation for the moment generating function jointly to the state of the input at time $t$ given the initial state of the input at time 0, asymptotic results for the first and second (matrix) moments of the process are provided. In particular, when the interarrival or ser…
This paper develops an approach for solving perpetual discounted optimal stopping problems for multidimensional diffusions, with special emphasis on the $d$-dimensional Wiener process. We first obtain some verification theorems for diffusions, based on the Green kernel representation of the value function associated with the problem. Specializing to the multidimensional Wiener process, we apply the Martin boundary theory to obtain a set of tractable integral equations involving only harmonic functions that characterize the stopping region of the problem. These equations allow to formulate a di…
Our purpose is to study a particular class of optimal stopping problems for Markov processes. We justify the value function convexity and we deduce that there exists a boundary function such that the smallest optimal stopping time is the first time when the Markov process passes over the boundary depending on time. Moreover, we propose a method to find the optimal boundary function.
We consider a pair $(X,Y)$ of stochastic processes satisfying the equation $dX=a(X)Y\,dB$ driven by a Brownian motion and study the monotonicity and continuity in $y$ of the value function $v(x,y)=\sup_τE_{x,y}[e^{-qτ}g(X_τ)]$, where the supremum is taken over stopping times with respect to the filtration generated by $(X,Y)$. Our results can successfully be applied to pricing American options where $X$ is the discounted price of an asset while $Y$ is given by a stochastic volatility model such as those proposed by Heston or Hull and White. The main method of proof is based on time-change and …
We study a model of two-player, zero-sum, stopping games with asymmetric information. We assume that the payoff depends on two continuous-time Markov chains (X, Y), where X is only observed by player 1 and Y only by player 2, implying that the players have access to stopping times with respect to different filtrations. We show the existence of a value in mixed stopping times and provide a variational characterization for the value as a function of the initial distribution of the Markov chains. We also prove a verification theorem for optimal stopping rules which allows to construct optimal sto…
In this paper, we propose an extension of the forward improvement iteration algorithm, originally introduced in Irle (2006) and recently reconsidered in Miclo and Villeneuve (2021). The main new ingredient is a flexible window parameter describing the look-ahead distance in the improvement step. We consider the framework of a Markovian optimal stopping problem in discrete time with random discounting and infinite time horizon. We prove convergence and show that the additional flexibility may significantly reduce the runtime.
In this paper we consider stopping problems for continuous-time Markov chains under a general risk-sensitive optimization criterion for problems with finite and infinite time horizon. More precisely our aim is to maximize the certainty equivalent of the stopping reward minus cost over the time horizon. We derive optimality equations for the value functions and prove the existence of optimal stopping times. The exponential utility is treated as a special case. In contrast to risk-neutral stopping problems it may be optimal to stop between jumps of the Markov chain. We briefly discuss the influe…
We characterize the value function and the optimal stopping time for a large class of optimal stopping problems where the underlying process to be stopped is a fairly general Markov process. The main result is inspired by recent findings for Lévy processes obtained essentially via the Wiener-Hopf factorization. The main ingredient in our approach is the representation of the $β$-excessive functions as expected suprema. A variety of examples is given.
This paper deals with the optimal stopping problem under partial observation for piecewise-deterministic Markov processes. We first obtain a recursive formulation of the optimal filter process and derive the dynamic programming equation of the partially observed optimal stopping problem. Then, we propose a numerical method, based on the quantization of the discrete-time filter process and the inter-jump times, to approximate the value function and to compute an actual $ε$-optimal stopping time. We prove the convergence of the algorithms and bound the rates of convergence.
Considering a real-valued diffusion, a real-valued reward function and a positive discount rate, we provide an algorithm to solve the optimal stopping problem consisting in finding the optimal expected discounted reward and the optimal stopping time at which it is attained. Our approach is based on Dynkin's characterization of the value function. The combination of Riesz's representation of $α$-excessive functions and the inversion formula gives the density of the representing measure, being only necessary to determine its support. This last task is accomplished through an algorithm. The propo…
In the literature on optimal stopping, the problem of maximizing the expected discounted reward over all stopping times has been explicitly solved for some special reward functions (including $(x^+)^ν$, $(e^x-K)^+$, $(K-e^{-x})^+$, $x\in\mathbb{R}$, $ν\in(0,\infty)$ and $K>0$) under general random walks in discrete time and Lévy processes in continuous time (subject to mild integrability conditions). All of such reward functions are continuous, increasing and logconcave while the corresponding optimal stopping times are of threshold type (i.e. the solutions are one-sided). In this paper, we sh…
Suppose that $\mathcal C$ is a finite collection of patterns. Observe a Markov chain until one of the patterns in $\mathcal C$ occurs as a run. This time is denoted by $τ$. In this paper, we aim to give an easy way to calculate the mean waiting time $E(τ)$ and the stopping probabilities $P(τ=τ_A)$ with $A\in\mathcal C$, where $τ_A$ is the waiting time until the pattern $A$ appears as a run.
In this Note we study optimal stopping problems for strong Markov processes and affine functions. We give a justification of the Snell envelope form using standard results of optimal stopping. We also justify the convexity of the value function, and without a priori restriction to a particular class of stopping times, we deduce that the smallest optimal stopping time is necessarily a hitting time.
This paper is concerned with the solution of the optimal stopping problem associated to the valuation of Perpetual American options driven by continuous time Markov chains. We introduce a new dynamic approach for the numerical pricing of this type of American options where the main idea is to build a monotone sequence of almost excessive functions that are associated to hitting times of explicit sets. Under minimal assumptions about the payoff and the Markov chain, we prove that the value function of an American option is characterized by the limit of this monotone sequence.
We propose a numerical method to approximate the value function for the optimal stopping problem of a piecewise deterministic Markov process (PDMP). Our approach is based on quantization of the post jump location---inter-arrival time Markov chain naturally embedded in the PDMP, and path-adapted time discretization grids. It allows us to derive bounds for the convergence rate of the algorithm and to provide a computable $ε$-optimal stopping time. The paper is illustrated by a numerical example.
This paper investigates the random horizon optimal stopping problem for measure-valued piecewise deterministic Markov processes (PDMPs). This is motivated by population dynamics applications, when one wants to monitor some characteristics of the individuals in a small population. The population and its individual characteristics can be represented by a point measure. We first define a PDMP on a space of locally finite measures. Then we define a sequence of random horizon optimal stopping problems for such processes. We prove that the value function of the problems can be obtained by iterating …
In a classical problem for the stopping of a diffusion process $(X_t)_{t \geq 0}$, where the goal is to maximise the expected discounted value of a function of the stopped process ${\mathbb E}^x[e^{-βτ}g(X_τ)]$, maximisation takes place over all stopping times $τ$. In a Poisson optimal stopping problem, stopping is restricted to event times of an independent Poisson process. In this article we consider whether the resulting value function $V_θ(x) = \sup_{τ\in {\mathcal T}({\mathbb T}^θ)}{\mathbb E}^x[e^{-βτ}g(X_τ)]$ (where the supremum is taken over stopping times taking values in the event ti…
In this paper, we present a discrete-type approximation scheme to solve continuous-time optimal stopping problems based on fully non-Markovian continuous processes adapted to the Brownian motion filtration. The approximations satisfy suitable variational inequalities which allow us to construct $ε$-optimal stopping times and optimal values in full generality. Explicit rates of convergence are presented for optimal values based on reward functionals of path-dependent SDEs driven by fractional Brownian motion. In particular, the methodology allows us to design concrete Monte-Carlo schemes for no…
We propose a method based on continuous time Markov chain approximation to compute the distribution of Parisian stopping times and price Parisian options under general one-dimensional Markov processes. We prove the convergence of the method under a general setting and obtain sharp estimate of the convergence rate for diffusion models. Our theoretical analysis reveals how to design the grid of the CTMC to achieve faster convergence. Numerical experiments are conducted to demonstrate the accuracy and efficiency of our method for both diffusion and jump models. To show the versality of our approa…
In this paper we consider discrete and continuous time risk sensitive optimal stopping problem. Using suitable properties of the underlying Feller-Markov process we prove continuity of the optimal stopping value function and provide formula for the optimal stopping policy. Next, we show how to link continuous time framework with its discrete time analogue. By considering suitable approximations, we obtain uniform convergence of the corresponding value functions.
Reinforcement schemes are a class of non-Markovian stochastic processes. Their non-Markovian nature allows them to model some kind of memory of the past. One subclass of such models are those in which the past is exponentially discounted or forgotten. Often, models in this subclass have the property of becoming trapped with probability~1 in some degenerate state. While previous work has concentrated on such limit results, we concentrate here on a contrary effect, namely that the time to become trapped may increase exponentially in 1/x as the discount rate, 1-x, approaches~1. As a result, the t…
We consider the optimal stopping problem consisting in, given a strong Markov process, a reward function and a discount rate, finding the stopping time such that the expected reward at the stopping time is maximum. The approach we follow, has two main components: the Dynkin's characterization of the value function as the smallest excessive function dominating the reward; and the Riesz representation of excessive functions in terms of the Green kernel, the main reference being Salminen 85. In the context of one-dimensional diffusions we give a complete characterization of the solution under som…
In this paper, we present a Longstaff-Schwartz-type algorithm for optimal stopping time problems based on the Brownian motion filtration. The algorithm is based on Leão, Ohashi and Russo and, in contrast to previous works, our methodology applies to optimal stopping problems for fully non-Markovian and non-semimartingale state processes such as functionals of path-dependent stochastic differential equations and fractional Brownian motions. Based on statistical learning theory techniques, we provide overall error estimates in terms of concrete approximation architecture spaces with finite Vapni…
We aim at characterizing the asymptotic behavior of value functions in the control of piece-wise deterministic Markov processes (PDMP) of switch type under nonexpansive assumptions. For a particular class of processes inspired by temperate viruses, we show that uniform limits of discounted problems as the discount decreases to zero and time-averaged problems as the time horizon increases to infinity exist and coincide. The arguments allow the limit value to depend on initial configuration of the system and do not require dissipative properties on the dynamics. The approach strongly relies on v…
In this paper, we solve explicitly the optimal stopping problem with random discounting and an additive functional as cost of observations for a regular linear diffusion. We also extend the results to the class of one-sided regular Feller processes. This generalizes the result of Beibel and Lerche [Statist. Sinica 7 (1997) 93-108] and [Teor. Veroyatn. Primen. 45 (2000) 657-669] and Irles and Paulsen [Sequential Anal. 23 (2004) 297-316]. Our approach relies on a combination of techniques borrowed from potential theory and stochastic calculus. We illustrate our results by detailing some new exam…
In this paper we revise the theory of turnpikes in discounted Markov decision processes, prove the turnpike theorem for the undiscounted model and apply the results to the specific random walk.
We study the stochastic control-stopping problem when the data are of polynomial growth. The approach is based on backward stochastic dierential equations (BSDEs for short). The problem turns into the study of a specic reected BSDE with a stochastic Lipschitz coecient for which we show existence and uniqueness of the solution. We then establish its relationship with the value function of the control-stopping problem. The optimal strategy is exhibited. Finally in the Markovian framework we prove that the value function is the unique viscosity solution of the associated Hamilton-Jacobi-Bellman e…
We propose an alternative approach for solving a number of well-studied optimal stopping problems for Lévy processes. Instead of the usual method of guess-and-verify based on martingale properties of the value function, we suggest a more direct method by showing that the general theory of optimal stopping for strong Markov processes together with some elementary observations imply that the stopping set must be of a certain form for the optimal stopping problems we consider. The independence of increments and the strong Markov property of Lévy processes then allow us to use straightforward opti…
In this paper we consider an infinite time horizon risk-sensitive optimal stopping problem for a Feller--Markov process with an unbounded terminal cost function. We show that in the unbounded case an associated Bellman equation may have multiple solutions and we give a probabilistic interpretation for the minimal and the maximal one. Also, we show how to approximate them using finite time horizon problems. The analysis, covering both discrete and continuous time case, is supported with illustrative examples.
Let X be a continuous-time Markov chain in a finite set I, let h be a mapping of I onto another set, and let Y be defined by Y_t=h(X_t), (for t nonnegative). We address the filtering problem for X in terms of the observation Y, which is not directly affected by noise. We write down explicit equations for the filtering process and show that this is a Markov process with the Feller property. We also prove that it is a piecewise-deterministic Markov process in the sense of Davis, and we identify its characteristics explicitly. We finally solve an optimal stopping problem for X with partial observ…
Consider a real hyperplane arrangement and let $\mathcal{C}$ denote the occurring chambers. Bidigare, Hanlon and Rockmore introduced a Markov chain on $\mathcal{C}$ which is a generalization of some card shuffling models used in computer science, biology and card games. This paper introduces strong stationary arguments for this Markov chain, which provide explicit bounds for the separation distance.
A discrete-time Markov chain can be transformed into a new Markov chain by looking at its states along iterations of an almost surely finite stopping time. By the optional stopping theorem, any bounded harmonic function with respect to the transition function of the original chain is harmonic with respect to the transition function of the transformed chain. The reverse inclusion is in general not true. Our main result provides a sufficient condition on the stopping time which guarantees that the space of bounded harmonic functions for the transformed chain embeds in the space of bounded harmon…
In this paper we compute the stopping times in the game Rock-Paper-Scissors. By exploiting the recurrence relation we compute the mean values of stopping times. On the other hand, by constructing a transition matrix for a Markov chain associated with the game, we get also the distribution of the stopping times and thereby we compute the mean stopping times again. Then we show that the mean stopping times increase exponentially fast as the number of the participants increases.
We extend the Longstaff-Schwartz algorithm for approximately solving optimal stopping problems on high-dimensional state spaces. We reformulate the optimal stopping problem for Markov processes in discrete time as a generalized statistical learning problem. Within this setup we apply deviation inequalities for suprema of empirical processes to derive consistency criteria, and to estimate the convergence rate and sample complexity. Our results strengthen and extend earlier results.
