This paper uses the generator approach of Stein's method to analyze the gap between steady-state distributions of Markov chains and diffusion processes. Until now, the standard way to invoke Stein's method for this problem was to use the Poisson equation for the diffusion as a starting point. The main technical difficulty with this approach is obtaining bounds on the derivatives of the solution to the Poisson equation, also known as Stein factor bounds. In this paper we propose starting with the discrete Poisson equation of the Markov chain. An important step in our approach is extending the discrete Poisson equation to be defined on the continuum where the diffusion is defined, and we achieve this by using interpolation. Although there are still Stein factor bounds to prove, these now correspond to the finite differences of the discrete Poisson equation solution, as opposed to the derivatives of the solution to the continuous one. Discrete Stein factor bounds can be easier to obtain, for instance when the drift is not everywhere differentiable, when the diffusion has a state-dependent diffusion coefficient, or in the presence of a reflecting boundary condition. We use the join the shortest queue model in the Halfin-Whitt regime as a working example to illustrate the methodology. We show that the steady-state approximation error of the diffusion limit converges to zero at a rate of $1/\sqrt{n}$, where $n$ is the number of servers in the system.
The mathematics of the finite single server queue with Poisson input and semi-Markov service times($M/SM/1/b$) is similar to that used for $BMAP/G/1/b$ systems. This observation results in new analytical formulas for a queue size in the $M/SM/1/b$ system. Both stationary and the transient solutions are considered
Stochastically monotone Markov chains arise in many applied domains, especially in the setting of queues and storage systems. Poisson's equation is a key tool for analyzing additive functionals of such models, such as cumulative sums of waiting times or sums of rewards. In this paper, we show that when the reward function for such a Markov chain is monotone, the solution of Poisson's equation is monotone. This implies that the value function associated with infinite horizon average reward is monotone in the state when the reward is monotone.
In this paper, the weak convergence of additive functionals of processes with locally independent increments and with Markov switching in the scheme of Poisson approximation is proved. For the relative compactness, a method proposed by R. Liptser for semimartingales is used with a modification, where we apply a solution of a singular perturbation problem instead of an ergodic theorem.
In this paper, we are devoted to developing matrix-analytic methods for solving Poisson's equation for irreducible and positive recurrent discrete-time Markov chains (DTMCs). Two special solutions, including the deviation matrix D and the expected additive-type functional matrix K, will be considered. The results are applied to Markov chains of GI/G/1-type and MAP/G/1 queues with negative customers. Further extensions to continuous-time Markov chains (CTMCs) are also investigated.
Generalization of the Lorden's inequality is an excellent tool for obtaining strong upper bounds for the convergence rate for various complicated stochastic models. This paper demonstrates a method for obtaining such bounds for some generalization of the Markov modulated Poisson process (MMPP). The proposed method can be applied to reliability and queuing theory.
This paper studies new Lancaster characterizations of bivariate multivariate Poisson, negative binomial and normal distributions which have diagonal expansions in multivariate orthogonal polynomials. The characterizations extend classical Lancaster characterizations of bivariate 1-dimensional distributions. Multivariate Poisson-Charlier, Meixner and Hermite-Chebycheff orthogonal polynomials, used in the characterizations, are constructed from classical 1-dimensional orthogonal polynomials and multivariate Krawtchouk polynomials. New classes of transition functions of discrete and continuous ti…
We consider the almost semi-continuous processes defined on a finite Markov chain. The representation of the moment generating functions for the absolute maximum after achievement positive level and for the recovery time are obtained. Modified processes with two-step rate of negative jumps are investigated.
This paper considers the Poisson equation for general state-space Markov chains in continuous time. The main purpose of this paper is to present specific bounds for the solutions of the Poisson equation for general state-space Markov chains. The solutions of the Poisson equation are unique in the sense that they are expressed in terms of a certain probabilistically interpretable solution (called the {\it standard solution}). Thus, we establish some specific bounds for the standard solution under the $f$-modulated drift condition (which is a kind of Foster-Lyapunov-type condition) and some mode…
Poisson's equation has a lot of applications in various areas. Usually it is hard to derive the explicit expression of the solution of Poisson's equation for a Markov chain on an infinitely many state space. We will present a computational framework for the solution for both discrete-time Markov chains (DTMCs) and continuous-time Markov chains (CTMCs), by developing the technique of augmented truncation approximations. The convergence to the solution is investigated in terms of the assumption about the monotonicity of the first return times, and is further established for two types of truncati…
Let P := {X_i,i >= 1} be a stationary Poisson point process in R^d, {C_i,i >= 1} be a sequence of i.i.d. random sets in R^d, and {Y_i^t; t \geq 0, i >= 1} be i.i.d. {0,1}-valued continuous time stationary Markov chains. We define the Markov-Poisson-Boolean model C_t := {Y_i^t(X_i + C_i), i >= 1}. C_t represents the coverage process at time t. We first obtain limit laws for k-coverage of an area at an arbitrary instant. We then obtain the limit laws for the k-coverage seen by a particle as it moves along a one-dimensional path.
The paper is concerned with the equilibrium distributions of continuous-time density dependent Markov processes on the integers. These distributions are known typically to be approximately normal, and the approximation error, as measured in Kolmogorov distance, is of the smallest order that is compatible with their having integer support. Here, an approximation in the much stronger total variation norm is established, without any loss in the asymptotic order of accuracy; the approximating distribution is a translated Poisson distribution having the same variance and (almost) the same mean. Our…
We study ergodic properties of a class of Markov-modulated general birth-death processes under fast regime switching. The first set of results concerns the ergodic properties of the properly scaled joint Markov process with a parameter that is taken large. Under very weak hypotheses, we show that if the averaged process is exponentially ergodic for large values of the parameter, then the same applies to the original joint Markov process. The second set of results concerns steady-state diffusion approximations, under the assumption that the 'averaged' fluid limit exists. Here, we establish conv…
In this paper we study Monte Carlo estimators based on the likelihood ratio approach for steady-state sensitivity. We first extend the result of Glynn and Olvera-Cravioto [doi:doi: 10.1287/stsy.2018.002] to the setting of continuous time Markov chains with a countable state space which include models such as stochastic reaction kinetics and kinetic Monte Carlo lattice system. We show that the variance of the centered likelihood ratio estimators does not grow in time. This result suggests that the centered likelihood ratio estimators should be favored for sensitivity analysis when the mixing ti…
For a class of excitatory spiking neuron models with delayed feedback fed with a Poisson stochastic process, it is proven that the stream of output interspike intervals cannot be presented as a Markov process of any order. Keywords: spiking neuron; Poisson stochastic process; probability density function; delayed feedback; non-Markov stochastic process
Changing time of simple continuous-time Markov counting processes by independent unit-rate Poisson processes results in Markov counting processes for which we provide closed-form transition rates via composition of trajectories and with which we construct novel, simpler infinitesimally over-dispersed processes.
The paper is concerned with approximating the distribution of a sum W of n integer valued random variables Y_i, whose distributions depend on the state of an underlying Markov chain X. The approximation is in terms of a translated Poisson distribution, with mean and variance chosen to be close to those of W, and the error is measured with respect to the total variation norm. Error bounds comparable to those found for normal approximation with respect to the weaker Kolmogorov distance are established, provided that the distribution of the sum of the Y_i's between the successive visits of X to a…
Motivated by queues with many servers, we study Brownian steady-state approximations for continuous time Markov chains (CTMCs). Our approximations are based on diffusion models (rather than a diffusion limit) whose steady-state, we prove, approximates that of the Markov chain with notable precision. Strong approximations provide such "limitless" approximations for process dynamics. Our focus here is on steady-state distributions, and the diffusion model that we propose is tractable relative to strong approximations. Within an asymptotic framework, in which a scale parameter $n$ is taken large,…
A discrete-time Markov chain can be transformed into a new Markov chain by looking at its states along iterations of an almost surely finite stopping time. By the optional stopping theorem, any bounded harmonic function with respect to the transition function of the original chain is harmonic with respect to the transition function of the transformed chain. The reverse inclusion is in general not true. Our main result provides a sufficient condition on the stopping time which guarantees that the space of bounded harmonic functions for the transformed chain embeds in the space of bounded harmon…
This paper studies the subgeometric convergence of the stationary distribution in taking the infinite-level limit of a finite-level M/G/1-type Markov chain, that is, in letting the upper boundary level go to infinity. This study is performed through the fundamental deviation matrix, which is a block-decomposition-friendly solution for the Poisson equation of the deviation matrix. The fundamental deviation matrix yields a difference formula for the respective stationary distributions of the finite-level chain and the corresponding infinite-level chain. The difference formula plays a crucial rol…
These are lecture notes on the subject defined in the title. As such, they do not pretend to be really new, probably except for the only section about Poisson equations with potentials. Yet, the hope of the author is that they may serve as a bridge to the important area of Poisson equations "in the whole space" and with a parameter, the latter theme not being presented here. Why this area is so important was explained in many papers and books (see the references [12, 34, 35]): it provides one of the main tools in diffusion approximation in the area stochastic averaging. Hence, the aim of these…
In this paper, we present new Poisson-type deviation inequalities for continuous-time Markov chains whose Wasserstein curvature or $Γ$-curvature is bounded below. Although these two curvatures are equivalent for Brownian motion on Riemannian manifolds, they are not comparable in discrete settings and yield different deviation bounds. In the case of birth--death processes, we provide some conditions on the transition rates of the associated generator for such curvatures to be bounded below and we extend the deviation inequalities established [Ané, C. and Ledoux, M. On logarithmic Sobolev inequa…
In the paper we consider a generalizations of the notion of Poisson process to the case when classical convolution is replaced by generalized convolution in the sense of K. Urbanik [16] following two classical definitions of Poisson process. First, for every generalized convolution $\diamond$ we define $\diamond$-generalized Poisson process type I as a Markov process with the $\diamond$-generalized Poisson distribution. Such processes have stationary independent increments in the sense of generalized convolution, but usually they do not live on $\mathbb{N}_0$. The $\diamond$-generalized Poisso…
In this paper, we employ Markov process theory to prove asymptotic results for a class of stochastic processes which arise as solutions of a stochastic evolution inclusion and are given by the representation formula \begin{align*} \mathbb{X}_{x}(t)=\sum \limits_{m=0}\limits^{\infty}T((t-α_{m})_{+})(x_{x,m})1\hspace{-0,9ex}1_{[α_{m},α_{m+1})}(t), \end{align*} where $(T(t))_{t \geq 0}$ is a (nonlinear) time-continuous, contractive semigroup acting on a separable Banach space $(V,||\cdot||_{V})$, $(α_{m})_{m \in \mathbb{N}}$ is the sequence of arrival times of a homogeneous Poisson process, $x$ i…
Representations of branching Markov processes and their measure-valued limits in terms of countable systems of particles are constructed for models with spatially varying birth and death rates. Each particle has a location and a "level," but unlike earlier constructions, the levels change with time. In fact, death of a particle occurs only when the level of the particle crosses a specified level $r$, or for the limiting models, hits infinity. For branching Markov processes, at each time $t$, conditioned on the state of the process, the levels are independent and uniformly distributed on $[0,r]…
This paper uses the generator comparison approach of Stein's method to analyze the gap between steady-state distributions of Markov chains and diffusion processes. The "standard" generator comparison approach starts with the Poisson equation for the diffusion, and the main technical difficulty is to obtain bounds on the derivatives of the solution to the Poisson equation, also known as Stein factor bounds. In this paper we propose starting with the Poisson equation of the Markov chain; we term this the prelimit approach. Although one still needs Stein factor bounds, they now correspond to fini…
The solution of Poisson's equation plays a key role in constructing the martingale through which sums of Markov correlated random variables can be analyzed. In this paper, we study two different representations for the solution in countable state space, one based on regenerative structure and the other based on an infinite sum of expectations. We also consider integrability and related uniqueness issues associated with solutions to Poisson's equation, and provide verifiable Lyapunov conditions to support our theory. Our key results include a central limit theorem and law of the iterated logari…
This paper presents a new view of household epidemic models, where the interaction between the households is of mean field type. We thus obtain in the limit of infinitely many households a nonlinear Markov process solution of a McKean - Vlasov type Poisson driven SDE, and a propagation of chaos result. We also define a basic reproduction number R0, and show that if R0>1, then the nonlinear Markov process has a unique non trivial ergodic invariant probability measure, whereas if R0<=1, it converges to 0 as t tends to infinity.
In this paper we prove the Poisson Hypothesis for the limiting behavior of the large queueing systems in some simple ("mean-field") cases. We show in particular that the corresponding dynamical systems, defined by the non-linear Markov processes, have a line of fixed points which are global attractors. To do this we derive the corresponding non-linear integral equation and we explore its self-averaging properties. Our derivation relies on a solution of a combinatorial problem of rode placements.
We study the analogue of Poisson ensembles of Markov loops ('loop soups') in the setting of one-dimensional diffusions. We give a detailed description of the corresponding intensity measure. The properties of this measure on loops lead us to an extension of Vervaat's bridge-to-excursion transformation that relates the bridges conditioned by their minimum and the excursions of all the diffusion we consider and not just the Brownian motion. Further we describe the Poisson point process of loops, their occupation fields and explain how to sample these Poisson ensembles of loops using two-dimensio…
Stochastic models for performance analysis, optimization and control of queues hinge on a multitude of alternatives for input point processes. In case of bursty traffic, one very popular model is the \textit{Markov Modulated Poisson Process} (MMPP), however it is not the only option. Here, we introduce an alternative that we call \textit{Markovian transition counting process} (MTCP). The latter is a point process counting the number of transitions of a finite continuous-time Markov chain. For a given MTCP one can establish an MMPP with the same first and second moments of counts. In this paper…
We derive and analyze new diffusion approximations of stationary distributions of Markov chains that are based on second- and higher-order terms in the expansion of the Markov chain generator. Our approximations achieve a higher degree of accuracy compared to diffusion approximations widely used for the past fifty years, while retaining a similar computational complexity. To support our approximations, we present a combination of theoretical and numerical results across three different models. Our approximations are derived recursively through Stein/Poisson equations, and the theoretical resul…
We describe a new construction of a family of measures on a group with the same Poisson boundary. Our approach is based on applying Markov stopping times to an extension of the original random walk.
In this paper we study the Poisson Hypothesis, which is a device to analyze approximately the behavior of large queueing networks. We prove it in some simple limiting cases. We show in particular that the corresponding dynamical system, defined by the non-linear Markov process, has a line of fixed points which are global attractors. To do this we derive the corresponding non-linear equation and we explore its self-averaging properties. We also argue that in cases of havy-tail service times the PH can be violated.
In this paper we prove two results. First we show that dynamical systems with a $ϕ$-mixing measure have in the limit Poisson distributed return times almost everywhere. We use the Chen-Stein method to also obtain rates of convergence. Our theorem improves on previous results by allowing for infinite partitions and dropping the requirement that the invariant measure have finite entropy with respect to the given partition. As has been shown elsewhere, the limiting distribution at periodic points is not Poissonian (but compound Poissonian). Here we show that for all non-periodic points the return…
In this paper we study the existence of Green measures for Markov processes with a nonlocal jump generator. The jump generator has no second moment and satisfies a suitable condition on its Fourier transform. We also study the same problem for certain classes of random time changes Markov processes with jump generator.
An optimal control for a dynamical system optimizes a certain objective function. Here we consider the construction of an optimal control for a stochastic dynamical system with a random structure, Poisson perturbations and random jumps, which makes the system stable in probability. Sufficient conditions of the stability in probability are obtained, using the second Lyapunov method, in which the construction of the corresponding functions plays an important role. Here we provide a solution to the problem of optimal stabilization in a general case. For a linear system with a quadratic quality fu…
We develop a new algorithm for the estimation of rare event probabilities associated with the steady-state of a Markov stochastic process with continuous state space $\mathbb R^d$ and discrete time steps (i.e. a discrete-time $\mathbb R^d$-valued Markov chain). The algorithm, which we coin Recurrent Multilevel Splitting (RMS), relies on the Markov chain's underlying recurrent structure, in combination with the Multilevel Splitting method. Extensive simulation experiments are performed, including experiments with a nonlinear stochastic model that has some characteristics of complex climate mode…
Lecture notes for a master-level applied mathematics course on stochastic processes and applications, held at the University of Orléans, France. Contents: Markov chains, Poisson point processes, Markovian jump processes, queueing theory. ----- Notes d'un cours de Master 2 professionnel en mathématiques appliquées, donné à l'Université d'Orléans. Contenu: Chaînes de Markov, processus ponctuel de Poisson, files d'attente.
We study finite horizon optimal switching problems for hidden Markov chain models under partially observable Poisson processes. The controller possesses a finite range of strategies and attempts to track the state of the unobserved state variable using Bayesian updates over the discrete observations. Such a model has applications in economic policy making, staffing under variable demand levels and generalized Poisson disorder problems. We show regularity of the value function and explicitly characterize an optimal strategy. We also provide an efficient numerical scheme and illustrate our resul…
We seek to infer the parameters of an ergodic Markov process from samples taken independently from the steady state. Our focus is on non-equilibrium processes, where the steady state is not described by the Boltzmann measure, but is generally unknown and hard to compute, which prevents the application of established equilibrium inference methods. We propose a quantity we call propagator likelihood, which takes on the role of the likelihood in equilibrium processes. This propagator likelihood is based on fictitious transitions between those configurations of the system which occur in the sample…
The conditional intensity (CI) of a counting process $Y_t$ is based on the minimal knowledge $\mathcal{F}_t^Y$, i.e., on the observation of $Y_t$ alone. Prominently, the mutual information rate of a signal and its Poisson channel output is a difference functional between the CI and the intensity that has full knowledge about the input. While the CI of Markov-modulated Poisson processes evolves according to Snyder's filter, self-exciting processes, e.g., Hawkes processes, specify the CI via the history of $Y_t$. The emergence of the CI as a self-contained stochastic process prompts us to bring …
