In a previous paper we determined one dimensional distributions of a stationary field with linear regressions and quadratic conditional variances under a linear constraint on the coefficients of the quadratic expression. In this paper we show that for stationary Markov chains with linear regressions and quadratic conditional variances the coefficients of the quadratic expression are indeed tied by a linear constraint which can take only one of the two alternative forms.
We consider the higher-order Markov Chain, and characterize the second order Markov chains admitting every probability distribution vector as a stationary vector. The result is used to construct Markov chains of higher-order with the same property. We also study conditions under which the set of stationary vectors of the Markov chain has a certain affine dimension.
We prove that the class of discrete time stationary max-stable process satisfying the Markov property is equal, up to time reversal, to the class of stationary max-autoregressive processes of order $1$. A similar statement is also proved for continuous time processes.
We study a class of Markov processes with finite state space and continuous time that have product form stationary distributions. We obtain a number of examples that can generate conjectures for diffusions with inert drift.
In this paper we consider stationary Markov chains with trivial two-sided tail sigma field, and prove that additive functionals satisfy the central limit theorem provided the variance of partial sums divided by n is bounded.
Let $X$ be an irreducible symmetric Markov process with the strong Feller property. We assume, in addition, that $X$ is explosive and has a tightness property. We then prove the existence and uniqueness of quasi-stationary distributions of $X$.
New algorithms for computing of asymptotic expansions for stationary distributions of nonlinearly perturbed semi-Markov processes are presented. The algorithms are based on special techniques of sequential phase space reduction, which can be applied to processes with asymptotically coupled and uncoupled finite phase spaces.
We study the fractal properties of the stationary distrubtion π for a simple Markov process on R. We will give bounds for the Hausdorff dimension of π, and lower bounds for the multifractal spectrum of π. Additionally, we will provide a method for numerically estimating these bounds.
We consider a simple regression model where a regressor is composed of order statistics and a noise is Markov-modulated. We introduce an empirical bridge of regression residuals and prove its weak convergence to a centered Gaussian process.
Dealing with unichain MDPs, we consider stationary distributions of policies that coincide in all but $n$ states. In these states each policy chooses one of two possible actions. We show that the stationary distributions of n+1 such policies uniquely determine the stationary distributions of all other such policies. An explicit formula for calculation is given.
We obtain necessary and sufficient conditions for the regular variation of the variance of partial sums of functionals of discrete and continuous-time stationary Markov processes with normal transition operators. We also construct a class of Metropolis-Hastings algorithms which satisfy a central limit theorem and invariance principle when the variance is not linear in $n$.
Let $(ξ_j)_{j\ge1} $, be a non-stationary Markov chain with phase space $X$ and let $\mathfrak{g}_j:\,X\mapsto\mathrm{SL}(m,\mathbb{R})$ be a sequence of functions on $X$ with values in the unimodular group. Set $g_j=\mathfrak{g}_j(ξ_j)$ and denote by $S_n=g_n\ldots g_1$, the product of the matrices $g_j$. We provide sufficient conditions for exponential growth of the norm $\|S_n\|$ when the Markov chain is not supposed to be stationary. This generalizes the classical theorem of Furstenberg on the exponential growth of products of independent identically distributed matrices as well as its ext…
In this paper, we study quasi-stationary distributions of nonlinearly perturbed semi-Markov processes in discrete time. This type of distributions is of interest for the analysis of stochastic systems which have finite lifetimes, but are expected to persist for a long time. We obtain asymptotic power series expansions for quasi-stationary distributions and it is shown how the coefficients in these expansions can be computed from a recursive algorithm. As an illustration of this algorithm, we present a numerical example for a discrete time Markov chain.
An algorithm for estimating quasi-stationary distribution of finite state space Markov chains has been proven in a previous paper. Now this paper proves a similar algorithm that works for general state space Markov chains under very general assumptions.
We present a class of Gauss-Markov processes which can be represented as space-time scaled stationary Ornstein-Uhlenbeck processes defined on the real line. We give several explicit examples of the representation for certain Gauss bridge processes. As an application, we derive a formula for the density function of the supremum location of certain standardized Gauss-Markov processes on compact time intervals. We also present some sufficient conditions under which mean centered Gauss-Markov processes take zero at a fixed time with probability one.
We find the class, ${\cal{C}}_k, k \ge 0$, of all zero mean stationary Gaussian processes, $Y(t), ~t \in \reals$ with $k$ derivatives, for which \begin{equation} Z(t) \equiv (Y^{(0)}(t), Y^{(1)}(t), \ldots, Y^{(k)}(t) ), ~ t \ge 0 \end{equation} \noindent is a $(k+1)$-vector Markov process. (here, $Y^{(0)}(t) = Y(t)$).
This article describes an accurate procedure for computing the mean first passage times of a finite irreducible Markov chain and a Markov renewal process. The method is a refinement to the Kohlas, Zeit fur Oper Res, 30,197-207, (1986) procedure. The technique is numerically stable in that it doesn't involve subtractions. Algebraic expressions for the special cases of one, two, three and four states are derived. A consequence of the procedure is that the stationary distribution of the embedded Markov chain does not need to be derived in advance but can be found accurately from the derived mean …
It has been well known for some time that for strictly stationary Markov chains that are ``reversible'', that special symmetry provides special extra features in the mathematical theory. This paper here is primarily a purely expository review of some of the basic features of that special theory. The mathematical techniques employed in this review are relatively gentle, involving only some basic measure-theoretic probability theory.
We prove a complete class theorem that characterizes \emph{all} stationary time reversible Markov processes whose finite dimensional marginal distributions (of all orders) are infinitely divisible. Aside from two degenerate cases (iid and constant), in both discrete and continuous time every such process with full support is a branching process with Poisson or Negative Binomial marginal univariate distributions and a specific bivariate distribution at pairs of times. As a corollary, we prove that every nondegenerate stationary integer valued processes constructed by the Markov thinning process…
For downward skip-free continuous-time Markov chains on non-negative integers stopped at zero, existence of a quasi-stationary distribution is studied. The scale function for these processes is introduced and the boundary is classified by a certain integrability condition on the scale function, which gives an extension of Feller's classification of the boundary for birth-and-death processes. The existence and the set of quasi-stationary distributions are characterized by the scale function and the new classification of the boundary.
A class of examples is constructed to show that for strictly stationary Markov chains that are reversible, the simultaneous mixing rates for the $ρ$-mixing and strong mixing ($α$-mixing) conditions can be fairly arbitrary, within certain unavoidable tight restrictions. The examples constructed here have the added property that the mixing rate for the absolute regularity ($β$-mixing) condition is within a constant factor of that for strong mixing.
For Markov chains with a partially ordered finite state space we show strong stationary duality under the condition of Möbius monotonicity of the chain. We show relations of Möbius monotonicity to other definitions of monotone chains. We give examples of dual chains in this context which have transitions only upwards. We illustrate general theory by an analysis of nonsymmetric random walks on the cube with an application to networks of queues.
We construct strong stationary dual chains for Ising model on a circle, non-symmetric random walk on square lattice and a random walk on hypercube. The strong stationary dual chains are all sharp and have the same state space as original chains. We use Möbius monotonicity of these chains with respect to natural orderings of the corresponding state spaces. This method provides an easy way to find eigenvalues in the Ising model and for a random walk on hypercube. Keywords: Markov chains; stochastic monotonicity; eigenvalues; Möbius monotonicity; strong stationary duality; strong stationary times…
Asymptotic expansions with explicit upper bounds for remainders are given for stationary distributions of nonlinearly perturbed semi-Markov processes with finite phase spaces. The corresponding algorithms are based on a special technique of sequential phase space reduction, which can be applied to processes with an arbitrary asymptotic communicative structure of phase spaces.
We consider a Markov modulated fluid network with a finite number of stations. We are interested in the tail asymptotics behavior of the stationary distribution of its buffer content process. Using two different approaches, we derive upper and lower bounds for the stationary tail decay rate in various directions. Both approaches are based on a well-known time-evolution formula of a Markov process, so-called Dynkin's formula, where a key ingredient is a suitable choice of test functions. Those results show how multidimensional tail asymptotics can be studied for the more than two-dimensional ca…
By introducing a key combinatorial structure for words produced by a Variable Length Markov Chain (VLMC), the longest internal suffix, precise characterizations of existence and uniqueness of a stationary probability measure for a VLMC chain are given. These characterizations turn into necessary and sufficient conditions for VLMC associated to a subclass of probabilised context trees: the shift-stable context trees. As a by-product, we prove that a VLMC chain whose stabilized context tree is again a context tree has at most one stationary probability measure.
In this paper we give sufficient conditions for the almost sure central limit theorem started at a point, known under the name of quenched central limit theorem. This is achieved by using a new idea of conditioning with respect to both the past and the future of the Markov chain. As applications we provide a new sufficient projective conditions for the quenched CLT.
We consider a random walk on a discrete connected graph having some infinite branches plus finitely many vertices with finite degrees. We find the generator of a strong stationary dual in the sense of Fill, and use it to find some equivalent condition to the existence of a strong stationary time. This strong stationary dual process lies in the set of connected compact sets of the compactification of the graph. When this graph is Z, this is simply the set of (possibly infinite) segments of Z.
New algorithms for construction of asymptotic expansions for stationary distributions of nonlinearly perturbed semi-Markov processes with finite phase spaces are presented. These algorithms are based on a special technique of sequential phase space reduction, which can be applied to processes with an arbitrary asymptotic communicative structure of phase spaces. Asymptotic expansions are given in two forms, without and with explicit bounds for remainders.
Consider a discrete time, ergodic Markov chain with finite state space which is started from stationarity. Fill and Lyzinski (2014) showed that, in some cases, the hitting time for a given state may be represented as a sum of a geometric number of IID random variables. We extend this result by giving explicit bounds on the distance between any such hitting time and an appropriately chosen geometric sum, along with other related approximations. The compounding random variable in our approximating geometric sum is a strong stationary time for the underlying Markov chain; we also discuss the appr…
We consider a Markov chain on $R^+$ with asymptotically zero drift and finite second moments of jumps which is positive recurrent. A power-like asymptotic behaviour of the invariant tail distribution is proven; such a heavy-tailed invariant measure happens even if the jumps of the chain are bounded. Our analysis is based on test functions technique and on construction of a harmonic function.
In this work, we focus on the stationary analysis of a specific class of continuous time Markov-modulated reflected random walks in the quarter plane with applications in the modelling of two-node Markov-modulated queueing networks with coupled queues. The transition rates of the two-dimensional process depend on the state of a finite state Markovian background process. Such a modulation is space homogeneous in the set of inner states of the two-dimensional lattice but may be different in the set of states at its boundaries. To obtain the stationary distribution, we apply the power series appr…
This paper is concerned with the development of rigorous approximations to various expectations associated with Markov chains and processes having non-stationary transition probabilities. Such non-stationary models arise naturally in contexts in which time-of-day effects or seasonality effects need to be incorporated. Our approximations are valid asymptotically in regimes in which the transition probabilities change slowly over time. Specifically, we develop approximations for the expected infinite horizon discounted reward, the expected reward to the hitting time of a set, the expected reward…
In this paper, we are interested in investigating the perturbation bounds for the stationary distributions for discrete-time or continuous-time Markov chains on a countable state space. For discrete-time Markov chains, two new norm-wise bounds are obtained. The first bound is rather easy to be obtained since the needed condition, equivalent to uniform ergodicity, is imposed on the transition matrix directly. The second bound, which holds for a general (possibly periodic) Markov chain, involves finding a drift function. This drift function is closely related with the mean first hitting times. S…
In an influential paper, Courtois and Semal (1984) establish that when $G$ is an irreducible substochastic matrix for which $\sum_{n=0}^{\infty}G^n <\infty$, then the stationary distribution of any stochastic matrix $P\ge G$ can be expressed as a convex combination of the normalized rows of $(I-G)^{-1} = \sum_{n=0}^{\infty} G^n$. In this note, we give a short proof of this result that extends the theory to the countably infinite and continuous state space settings. This result plays an important role in obtaining error bounds in algorithms involving nearly decomposable Markov chains, and also …
We present a general functional central limit theorem started at a point also known under the name of quenched. As a consequence, we point out several new classes of stationary processes, defined via projection conditions, which satisfy this type of asymptotic result. One of the theorems shows that if a Markov chain is stationary ergodic and reversible, this result holds for bounded additive functionals of the chain which have a martingale coboundary in L_1 representation. Our results are also well adapted for strongly mixing sequences providing for this case an alternative, shorter approach t…
We consider a discrete time semi-Markov process where the characteristics defining the process depend on a small perturbation parameter. It is assumed that the state space consists of one finite communicating class of states and, in addition, one absorbing state. Our main object of interest is the asymptotic behaviour of the joint probabilities of the position of the semi-Markov process and the event of non-absorption as time tends to infinity and the perturbation parameter tends to zero. The main result gives exponential expansions of these probabilities together with an recursive algorithm f…
We consider a Markov process $ X(t) $ on the nonnegative integers $E= S \cup \{0\}$, where $S=\{1,2,...\}$ is an irreducible class and 0 is an absorbing state. In this paper, we investigate conditions under which the quasi-stationary distribution for $X(t)$ exists and is unique, and any initial distribution supported in $S$ is in the domain of attraction of this quasi-stationary distribution. We further find five conditions which are equivalent to that the extinction time is uniformly bounded. As a consequence, we prove the van Doorn's conjecture in \cite{VD2012}. And we can greatly improve th…
We study the long time behaviour of a Markov process evolving in $\mathbb{N}$ and conditioned not to hit 0. Assuming that the process comes back quickly from infinity, we prove that the process admits a unique quasi-stationary distribution (in particular, the distribution of the conditioned process admits a limit when time goes to infinity). Moreover, we prove that the distribution of the process converges exponentially fast in total variation norm to its quasi-stationary distribution and we provide an explicit rate of convergence. As a first application of our result, we bring a new insight o…
We study a class of stationary Markov processes with marginal distributions identifiable by moments such that every conditional moment of degree say $m$ is a polynomial of degree at most $m\;\text{.}\;$ We show that then under some additional, natural technical assumption there exists a family of orthogonal polynomial martingales. More precisely we show that such a family of processes is completely characterized by the sequence $\{(α_{n} ,p_{n})\}_{n \geq 0}$ where $α_{n}^{ \prime } s$ are some positive reals while $p_{n}^{ \prime } s$ are some monic orthogonal polynomials. Paper of Bakry\&Maz…
We consider a class of discrete time Markov chains with state space [0,1] and the following dynamics. At each time step, first the direction of the next transition is chosen at random with probability depending on the current location. Then the length of the jump is chosen independently as a random proportion of the distance to the respective end point of the unit interval, the distributions of the proportions being fixed for each of the two directions. Chains of that kind were subjects of a number of studies and are of interest for some applications. Under simple broad conditions, we establis…
This paper considers the subexponential asymptotics of the stationary distributions of GI/G/1-type Markov chains in two cases: (i) the phase transition matrix in non-boundary levels is stochastic; and (ii) it is strictly substochastic. For the case (i), we present a weaker sufficient condition for the sub exponential asymptotics than those given in the literature. As for the case (ii), the subexponential asymptotics has not been studied, as far as we know. We show that the subexponential asymptotics in the case (ii) is different from that in the case (i). We also study the locally subexponenti…
We compute the stationary distribution of a continuous-time Markov chain which is constructed by gluing together two finite, irreducible Markov chains by identifying a pair of states of one chain with a pair of states of the other and keeping all transition rates from either chain (the rates between the two shared states are summed). The result expresses the stationary distribution of the glued chain in terms of quantities of the two original chains. Some of the required terms are nonstandard but can be computed by solving systems of linear equations using the transition rate matrices of the t…
This paper proposes a new algorithm for computing the stationary distribution vector in continuous-time upper block-Hessenberg Markov chains. To this end, we consider the last-block-column-linearly-augmented (LBCL-augmented) truncation of the (infinitesimal) generator of the upper block-Hessenberg Markov chain. The LBCL-augmented truncation is a linearly-augmented truncation such that the augmentation distribution has its probability mass only on the last block column. We first derive an upper bound for the total variation distance between the respective stationary distribution vectors of the …
We consider a simple but important class of metastable discrete time Markov chains, which we call perturbed Markov chains. Basically, we assume that the transition matrices depend on a parameter $\varepsilon$, and converge as $\varepsilon$. We further assume that the chain is irreducible for $\varepsilon$ but may have several essential communicating classes when $\varepsilon$. This leads to metastable behavior, possibly on multiple time scales. For each of the relevant time scales, we derive two effective chains. The first one describes the (possibly irreversible) metastable dynamics, while th…
This paper studies the light-tailed asymptotics of the stationary tail probability vectors of a Markov chain of M/G/1 type. Almost all related studies have focused on the typical case, where the transition block matrices in the non-boundary levels have a dominant impact on the decay rate of the stationary tail probability vectors and their decay is aperiodic. In this paper, we study not only the typical case but also atypical cases such that the stationary tail probability vectors decay periodically and/or their decay rate is determined by the tail distribution of jump sizes from the boundary …
We continue the study of Markov systems started in \cite{Wer1}. In this paper, we prove a generalization of Breiman's strong low of large numbers \cite{Br} which implies a necessary condition for the uniqueness of the stationary state of a Markov system.
We consider a strictly substochastic matrix or an stochastic matrix with absorbing states. By using quasi-stationary distributions one shows there is a canonical associated stationary Markov chain. Based upon $2-$stringing representation of the resurrected chain we supply a stationary representation of the killed and the absorbed chains. The entropies of these representations has a clear meaning when one identifies the probability measure of natural factors. The balance between the entropies of these representations and the canonical chain, serves to check the suitability of the whole construc…
Let $(X_n)_{n \in\mathbb{N}}$ be a $V$-geometrically ergodic Markov chain on a measurable space $\mathbb{X}$ with invariant probability distribution $π$. In this paper, we propose a discretization scheme providing a computable sequence $(\widehatπ_k)_{k\ge 1}$ of probability measures which approximates $π$ as $k$ growths to infinity. The probability measure $\widehatπ_k$ is computed from the invariant probability distribution of a finite Markov chain. The convergence rate in total variation of $(\widehatπ_k)_{k\ge 1}$ to $π$ is given. As a result, the specific case of first order autoregressiv…
We analyze the stationary distribution of regulated Markov modulated Brownian motions (MMBM) modified so that their evolution is slowed down when the process reaches level zero --- level zero is said to be {\em sticky}. To determine the stationary distribution, we extend to MMBMs a construction of Brownian motion with sticky boundary, and we follow a Markov-regenerative approach similar to the one developed in past years in the context of quasi-birth-and-death processes and fluid queues. We also rely on recent work showing that Markov-modulated Brownian motions may be analyzed as limits of a p…
