This paper is devoted to studying the average optimality in continuous-time Markov decision processes with fairly general state and action spaces. The criterion to be maximized is expected average rewards. The transition rates of underlying continuous-time jump Markov processes are allowed to be unbounded, and the reward rates may have neither upper nor lower bounds. We first provide two optimality inequalities with opposed directions, and also give suitable conditions under which the existence of solutions to the two optimality inequalities is ensured. Then, from the two optimality inequalities we prove the existence of optimal (deterministic) stationary policies by using the Dynkin formula. Moreover, we present a ``semimartingale characterization'' of an optimal stationary policy. Finally, we use a generalized Potlach process with control to illustrate the difference between our conditions and those in the previous literature, and then further apply our results to average optimal control problems of generalized birth--death systems, upwardly skip-free processes and two queueing systems. The approach developed in this paper is slightly different from the ``optimality inequality approach'' widely used in the previous literature.
The aim of this note is to show, by providing an elementary way to construct counter-examples, that the strong Feller and the joint (space-time) continuity for a semigroup of Markov kernels on a Polish space are not enough to ensure the existence of an associated càdlàg Markov process on the same space. One such simple counter-example is the Brownian semigroup on $\mathbb{R}$ restricted to $\mathbb{R}\setminus \{0\}$, for which it is shown that there is no associated càdlàg Markov process. Using the same idea and results from potential theory we then prove that the analogous result with càdlàg…
This paper is devoted to studying constrained continuous-time Markov decision processes (MDPs) in the class of randomized policies depending on state histories. The transition rates may be unbounded, the reward and costs are admitted to be unbounded from above and from below, and the state and action spaces are Polish spaces. The optimality criterion to be maximized is the expected discounted rewards, and the constraints can be imposed on the expected discounted costs. First, we give conditions for the nonexplosion of underlying processes and the finiteness of the expected discounted rewards/c…
The dissertation describes ergodic properties of some stochastic dynamical systems generated by Markov chains with values in the state space which is a Polish space. The mathematical model describing the process of cell division is analyzed. The assumptions are satisfied, among others, by the model introduced by J.J. Tyson and K.B. Hannsgen, J. Math. Biol. (1986). Within the thesis the existence of a unique invariant measure is established and asymptotic stability of the model is verified. In addition, the rate of convergence of the sequence of measures, transformed by the subsequent iteration…
We consider a time inhomogeneous strong Markov process $(ξ_t)_{t\ge 0}$ taking values in a Polish state space whose semigroup has a $T$-periodic structure. We give simple conditions which imply ergodicity of the grid chain $(ξ_{kT})_{k\in \mathbb{N}_0}$ In case of $d$-dimensional possibly degenerate diffusions, the conditions are stated in terms of drift and diffusion coefficient of the process.
We provide sufficient conditions for uniqueness of an invariant probability measure of a Markov kernel in terms of (generalized) couplings. Our main theorem generalizes previous results which require the state space to be Polish. We provide an example showing that uniqueness can fail if the state space is separable and metric (but not Polish) even though a coupling defined via a continuous and positive definite function exists.
Suppose that $\{X_t,\,t\ge0\}$ is a non-stationary Markov process, taking values in a Polish metric space $E$. We prove the law of large numbers and central limit theorem for an additive functional of the form $\int_0^Tψ(X_s)ds$, provided that the dual transition probability semigroup, defined on measures, is strongly contractive in an appropriate Wasserstein metric. Function $ψ$ is assumed to be Lipschitz on $E$.
We present a dynamic programming-based solution to a stochastic optimal control problem up to a hitting time for a discrete-time Markov control process. Firstly, we determine an optimal control policy to steer the process toward a compact target set while simultaneously minimizing an expected discounted cost. We then provide a rolling-horizon strategy for approximating the optimal policy, together with quantitative characterization of its sub-optimality with respect to the optimal policy. Finally, we address related issues of asymptotic discount-optimality of the value-iteration policy. Both t…
The correspondence between the cross-entropy method and the zero-variance approximation to simulate a rare event problem in Markov chains is shown. This leads to a sufficient condition that the cross-entropy estimator is asymptotically optimal.
Max-stable processes are central models for spatial extremes. In this paper, we focus on some space-time max-stable models introduced in Embrechts et al. (2016). The processes considered induce discrete-time Markov chains taking values in the space of continuous functions from the unit sphere of $\mathbb{R}^3$ to $(0, \infty)$. We show that these Markov chains are geometrically ergodic. An interesting feature lies in the fact that the state space is not locally compact, making the classical methodology inapplicable. Instead, we use the fact that the state space is Polish and apply results pres…
This paper deals with the long run average continuous control problem of piecewise deterministic Markov processes (PDMP's) taking values in a general Borel space and with compact action space depending on the state variable. The control variable acts on the jump rate and transition measure of the PDMP, and the running and boundary costs are assumed to be positive but not necessarily bounded. Our first main result is to obtain an optimality equation for the long run average cost in terms of a discrete-time optimality equation related to the embedded Markov chain given by the post-jump location …
This paper studies transition probabilities from a Borel subset of a Polish space to a product of two Borel subsets of Polish spaces. For such transition probabilities it introduces and studies the property of semi-uniform Feller continuity. This paper provides several equivalent definitions of semi-uniform Feller continuity and establishes its preservation under integration. The motivation for this study came from the theory of Markov decision processes with incomplete information, and this paper provides fundamental results useful for this theory.
For a stochastic process with state space some Polish space, this paper gives sufficient conditions on the initial and conditional distributions for the joint law to satisfy Gaussian concentration inequalities, transportation inequalities and also logarithmic Sobolev inequalities in the case of the Euclidean space. In several cases, the obtained constants are of optimal order of growth with respect to the number of variables, or are independent of this number. These results extend results known for mutually independent variables to weakly dependent variables under Dobrushin-Shlosman type condi…
Various Poincare-Sobolev type inequalities are studied for a reaction-diffusion model of particle systems on Polish spaces. The systems we consider consist of finite particles which are killed or produced at certain rates, while particles in the system move on the Polish space interacting with one another (i.e. diffusion). Thus, the corresponding Dirichlet form, which we call reaction-diffusion Dirichlet form, consists of two parts: the diffusion part induced by certain Markov processes on the product spaces $E^n (n \geq 1)$ which determine the motion of particles, and the reaction part induce…
Let $(X_n)_{n=0}^\infty$ denote a Markov chain on a Polish space that has a stationary distribution $\varpi$. This article concerns upper bounds on the Wasserstein distance between the distribution of $X_n$ and $\varpi$. In particular, an explicit geometric bound on the distance to stationarity is derived using generalized drift and contraction conditions whose parameters vary across the state space. These new types of drift and contraction allow for sharper convergence bounds than the standard versions, whose parameters are constant. Application of the result is illustrated in the context of …
In this paper we prove a version of the Berman-Konsowa principle for reversible Markov jump processes on Polish spaces. The Berman-Konsowa principle provides a variational formula for the capacity of a pair of disjoint measurable sets. There are two versions, one involving a class of probability measures for random finite paths from one set to the other, the other involving a class of finite unit flows from one set to the other. The Berman-Konsowa principle complements the Dirichlet principle and the Thomson principle, and turns out to be especially useful for obtaining sharp estimates on cros…
In this paper we investigate the existence of $L^{2}(π)$-spectral gaps for $π$-irreducible, positive recurrent Markov chains on general state space. We obtain necessary and sufficient conditions for the existence of $L^{2}(π)$-spectral gaps in terms of a sequence of isoperimetric constants and establish their asymptotic behavior. It turns out that in some cases the spectral gap can be understood in terms of convergence of an induced probability flow to the uniform flow. The obtained theorems can be interpreted as mixing results and yield sharp estimates for the spectral gap of some Markov chai…
We prove the existence of the persistence exponent $$\logλ:=\lim_{n\to\infty}\frac{1}{n}\log \mathbb{P}_μ(X_0\in S,\ldots,X_n\in S)$$ for a class of time homogeneous Markov chains $\{X_i\}_{i\geq 0}$ taking values in a Polish space, where $S$ is a Borel measurable set and $μ$ is an initial distribution. Focusing on the case of AR($p$) and MA($q$) processes with $p,q\in \mathbb{N}$ and continuous innovation distribution, we study the existence of $λ$ and its continuity in the parameters of the AR and MA processes, respectively, for $S=\mathbb{R}_{\geq 0}$. For AR processes with log-concave inno…
Given two n-dimensional measures $μ$ and $ν$ on Polish spaces, we propose an optimal transportation's formulation, inspired by classical Kan-torovitch's formulation in the scalar case. In particular, we established a strong duality result and as a consequence, optimality conditions are investigated. Wasserstein's metrics induced by our formulation are also investigated.
The Metric Coalescent (MC) is a measure-valued Markov Process generalizing the classical Kingman Coalescent. We show how the MC arises naturally from a discrete agent based model (Compulsive Gambler) of social dynamics and prove an existence and uniqueness theorem extending the MC to the space of all Borel probability measures on any locally compact Polish space.
We formulate a criterion for the existence of an invariant measure for a Feller semigroup defined on a metric space with the e-property for bounded continuous functions and use it to prove the asymptotic stability of a semigroup satisfying a lower bound condition. Our results complement those of A. Lasota and J. A. Yorke in proper metric spaces and of T. Szarek in Polish spaces.
We consider mean field games with ergodic cost in the framework of a general discrete time controlled Markov processes. The state space of the processes is given by a general $σ$-compact Polish space. Under certain conditions, we show the existence of a mean field game equilibrium. We also study the $N$-person game where the players interacts with each other via their empirical measure. We show that the $N$-person game has Nash equilibrium and as $N$ tends to infinity the equilibria converge to a mean field game solution.
We investigate a piecewise-deterministic Markov process, evolving on a Polish metric space, whose deterministic behaviour between random jumps is governed by some semi-flow, and any state right after the jump is attained by a randomly selected continuous transformation. It is assumed that the jumps appear at random moments, which coincide with the jump times of a Poisson process with intensity $λ$. The model of this type, although in a more general version, was examined in our previous papers, where we have shown, among others, that the Markov process under consideration possesses a unique inv…
We study the existence, optimality, and construction of non-randomised stopping times that solve the Skorokhod embedding problem (SEP) for Markov processes which satisfy a duality assumption. These stopping times are hitting times of space-time subsets, so-called Root barriers. Our main result is, besides the existence and optimality, a potential-theoretic characterisation of this Root barrier as a free boundary. If the generator of the Markov process is sufficiently regular, this reduces to an obstacle PDE that has the Root barrier as free boundary and thereby generalises previous results fro…
We construct two types of equilibrium dynamics of infinite particle systems in a locally compact Polish space $X$, for which certain fermion point processes are invariant. The Glauber dynamics is a birth-and-death process in $X$, while in the case of the Kawasaki dynamics interacting particles randomly hop over $X$. We establish conditions on generators of both dynamics under which corresponding conservative Markov processes exist.
We formulate a criterion for the existence and uniqueness of an invariant measure for a Markov process taking values in a Polish phase space. In addition, weak-$^*$ ergodicity, that is, the weak convergence of the ergodic averages of the laws of the process starting from any initial distribution, is established. The principal assumptions are the existence of a lower bound for the ergodic averages of the transition probability function and its local uniform continuity. The latter is called the e-property. The general result is applied to solutions of some stochastic evolution equations in Hilbe…
A marked metric measure space (mmm-space) is a triple (X,r,mu), where (X,r) is a complete and separable metric space and mu is a probability measure on XxI for some Polish space I of possible marks. We study the space of all (equivalence classes of) marked metric measure spaces for some fixed I. It arises as state space in the construction of Markov processes which take values in random graphs, e.g. tree-valued dynamics describing randomly evolving genealogical structures in population models. We derive here the topological properties of the space of mmm-spaces needed to study convergence in d…
This paper studies the bailout optimal dividend problem with regime switching under the constraint that dividend payments can be made only at the arrival times of an independent Poisson process while capital can be injected continuously in time. We show the optimality of the regime-modulated Parisian-classical reflection strategy when the underlying risk model follows a general spectrally negative Markov additive process. In order to verify the optimality, first we study an auxiliary problem driven by a single spectrally negative \lev process with a final payoff at an exponential terminal time…
This work considers two-player zero-sum semi-Markov games with incomplete information on one side and perfect observation. At the beginning, the system selects a game type according to a given probability distribution and informs to Player 1 only. After each stage, the actions chosen are observed by both players before proceeding to the next stage. Firstly, we show the existence of the value function under the expected discount criterion and the optimality equation. Secondly, the existence and iterative algorithm of the optimal policy for Player 1 are introduced through the optimality equation…
In this paper we study the bicausal optimal transport problem for Markov chains, an optimal transport formulation suitable for stochastic processes which takes into consideration the accumulation of information as time evolves. Our analysis is based on a relation between the transport problem and the theory of Markov decision processes. This way we are able to derive necessary and sufficient conditions for optimality in the transport problem, as well as an iterative algorithm, namely the value iteration, for the calculation of the transportation cost. Additionally, we draw the connection with …
By using the fact that the space of all probability measures with finite support can be somehow completed in two different fashions, one generating the Arens-Eells space and another generating the Kantorovich-Wasserstein (Wasserstein-1) space, and by exploiting the duality relationship between the Arens-Eells space with the space of Lipschitz functions, we provide a dual representation of Fenchel-Moreau-Rockafellar type for proper convex functionals on Wasserstein-1. We retrieve dual transportation inequalities as a Corollary and we provide examples where the theorem can be used to easily prov…
In this paper we consider stopping problems for continuous-time Markov chains under a general risk-sensitive optimization criterion for problems with finite and infinite time horizon. More precisely our aim is to maximize the certainty equivalent of the stopping reward minus cost over the time horizon. We derive optimality equations for the value functions and prove the existence of optimal stopping times. The exponential utility is treated as a special case. In contrast to risk-neutral stopping problems it may be optimal to stop between jumps of the Markov chain. We briefly discuss the influe…
We present some new results on sample path optimality for the ergodic control problem of a class of non-degenerate diffusions controlled through the drift. The hypothesis most often used in the literature to ensure the existence of an a.s. sample path optimal stationary Markov control requires finite second moments of the first hitting times $τ$ of bounded domains over all admissible controls. We show that this can be considerably weakened: ${\mathrm E}[τ^2]$ may be replaced with ${\mathrm E}[τ\ln^+(τ)]$, thus reducing the required rate of convergence of averages from polynomial to logarithmic…
We study optimal Markovian couplings of Markov processes, where the optimality is understood in terms of minimization of concave transport costs between the time-marginal distributions of the coupled processes. We provide explicit constructions of such optimal couplings for one-dimensional finite-activity Lévy processes (continuous-time random walks) whose jump distributions are unimodal but not necessarily symmetric. Remarkably, the optimal Markovian coupling does not depend on the specific concave transport cost. To this end, we combine McCann's results on optimal transport and Rogers' resul…
The main goal of this paper is to apply the so-called policy iteration algorithm (PIA) for the long run average continuous control problem of piecewise deterministic Markov processes (PDMP's) taking values in a general Borel space and with compact action space depending on the state variable. In order to do that we first derive some important properties for a pseudo-Poisson equation associated to the problem. In the sequence it is shown that the convergence of the PIA to a solution satisfying the optimality equation holds under some classical hypotheses and that this optimal solution yields to…
The main goal of this paper is to derive sufficient conditions for the existence of an optimal control strategy for the long run average continuous control problem of piecewise deterministic Markov processes (PDMP's) taking values in a general Borel space and with compact action space depending on the state variable. In order to do that we apply the so-called vanishing discount approach to obtain a solution to an average cost optimality inequality associated to the long run average cost problem. Our main assumptions are written in terms of some integro-differential inequalities related to the …
Consider a finite irreducible Markov chain with invariant probability $π$. Define its inverse communication speed as the expectation to go from x to y, when x, y are sampled independently according to $π$. In the discrete time setting and when $π$ is the uniform distribution $\upsilon$, Litvak and Ejov have shown that the permutation matrices associated to Hamiltonian cycles are the fastest Markov chains. Here we prove (A) that the above optimality is with respect to all processes compatible with a fixed graph of permitted transitions (assuming that it does contain a Hamiltonian cycle), not on…
Exponential ergodicity in the bounded-Lipschitz distance for a subclass of piecewise-deterministic Markov processes with random switching between flows
This paper deals with discrete-time Markov control processes on a general state space. A long-run risk-sensitive average cost criterion is used as a performance measure. The one-step cost function is nonnegative and possibly unbounded. Using the vanishing discount factor approach, the optimality inequality and an optimal stationary strategy for the decision maker are established.
Let $(\mathbb{P}^{s,x})_{(s,x)\in[0,T]\times E}$ be a family of probability measures, where $E$ is a Polish space,defined on the canonical probability space ${\mathbb D}([0,T],E)$ of $E$-valued cadlag functions. We suppose that a martingale problem with respect to a time-inhomogeneous generator $a$ is well-posed. We consider also an associated semilinear {\it Pseudo-PDE} with generator $a$ for which we introduce a notion of so called {\it decoupled mild} solution and study the equivalence with the notion of martingale solution introduced in a companion paper. We also investigate well-posedness…
For an infinite system of particles arriving in and departing from a habitat $X$ -- a locally compact Polish space with a positive Radon measure $χ$ -- a Markov process is constructed in an explicit way. Along with its location $x\in X$, each particle is characterized by age $α\geq 0$ -- time since arriving. As the state space one takes the set of marked configurations $\widehatΓ$, equipped with a metric that makes it a complete and separable metric space. The stochastic evolution of the system is described by a Kolmogorov operator $L$, expressed through the measure $χ$ and a departure rate $m…
This paper is devoted to solving a time-inconsistent risk-sensitive control problem with parameter $\e$ and its limit case ($\e\rightarrow0^+$) for countable-stated Markov decision processes (MDPs for short). Since the cost functional is time-inconsistent, it is impossible to find a global optimal strategy for both cases. Instead, for each case, we will prove the existence of time-inconstant equilibrium strategies which verify the so-called step-optimality. Moreover, we prove the convergence of $\e$-equilibriums and the corresponding value functions as $\e\rightarrow0^+$.
We consider killed Markov decision processes for countable models on a finite time-interval. Existence of a uniform $\varepsilon$-optimal policy is proven. We show the correctness of the fundamental equation. The optimal control problem is reduced to a similar problem for the derived model. We receive an optimality equation and a method for the construction of simple optimal policies. The sufficiency of simple policies for countable models is proven. We show the correctness of the Markovian property. Additionally, a dynamic programming principle is considered.
We prove a large-deviation principle (LDP) for the sample paths of jump Markov processes in the small noise limit when, possibly, all the jump rates vanish uniformly, but slowly enough, in a region of the state space. We further discuss the optimality of our assumptions on the decay of the jump rates. As a direct application of this work we relax the assumptions needed for the application of LDPs to, e.g., Chemical Reaction Network dynamics, where vanishing reaction rates arise naturally particularly the context of mass action kinetics.
