Importance sampling is a variance reduction technique for efficient estimation of rare-event probabilities by Monte Carlo. In standard importance sampling schemes, the system is simulated using an a priori fixed change of measure suggested by a large deviation lower bound analysis. Recent work, however, has suggested that such schemes do not work well in many situations. In this paper we consider dynamic importance sampling in the setting of uniformly recurrent Markov chains. By ``dynamic'' we mean that in the course of a single simulation, the change of measure can depend on the outcome of the simulation up till that time. Based on a control-theoretic approach to large deviations, the existence of asymptotically optimal dynamic schemes is demonstrated in great generality. The implementation of the dynamic schemes is carried out with the help of a limiting Bellman equation. Numerical examples are presented to contrast the dynamic and standard schemes.
We study the entropy of the distribution of the set R_n of vertices visited by a simple random walk on a graph with bounded degrees in its first n steps. It is shown that this quantity grows linearly in the expected size of R_n if the graph is uniformly transient, and sublinearly in the expected size if the graph is uniformly recurrent with subexponential volume growth. This in particular answers a question asked by Benjamini, Kozma, Yadin and Yehudayoff (arXiv:0903.3179v1).
In this paper, the weak convergence of impulsive recurrent process with semi-Markov switching in the scheme of Levy approximation is proved. Singular perturbation problem for the compensating operator of the extended Markov renewal process is used to prove the relative compactness.
In this paper, the weak convergence of impulsive recurrent process with Markov switching in the scheme of Levy approximation is proved. For the relative compactness, a method proposed by R. Liptser for semimartingales is used with a modification, where we apply a solution of a singular perturbation problem instead of an ergodic theorem.
Let $X=(X_t, t\geq 0)$ be a self-similar Markov process taking values in $\mathbb{R}$ such that the state 0 is a trap. In this paper, we present a necessary and sufficient condition for the existence of a self-similar recurrent extension of $X$ that leaves 0 continuously. The condition is expressed in terms of the associated Markov additive process via the Lamperti-Kiu representation. Our results extend those of Fitzsimmons (2006) and Rivero (2005, 2007) where the existence and uniqueness of a recurrent extension for positive self similar Markov processes were treated. In particular, we descri…
We prove that a positive self-similar Markov process $(X,\mathbb{P})$ that hits 0 in a finite time admits a self-similar recurrent extension that leaves 0 continuously if and only if the underlying Lévy process satisfies Cramér's condition.
In our monograph with B. Roynette and M. Yor, we construct a sigma-finite measure related to penalisations of different stochastic processes, including the Brownian motion in dimension 1 or 2, and a large class of linear diffusions. In the last chapter of the monograph, we define similar measures from recurrent Markov chains satisfying some technical conditions. In the present paper, we give a classification of these measures, in function of the minimal Martin boundary of the Markov chain considered at the beginning. We apply this classification to the examples considered at the end of our mon…
We give a bound on the mixing time of a uniformly ergodic, reversible Markov chain in terms of the spectral radius of the transition operator. This bound has been established previously in finite state spaces, and is widely believed to hold in general state spaces, but a proof has not been provided to our knowledge.
This paper generalizes the work of Kendall [Electron. Comm. Probab. 9 (2004) 140--151], which showed that perfect simulation, in the form of dominated coupling from the past, is always possible (although not necessarily practical) for geometrically ergodic Markov chains. Here, we consider the more general situation of positive recurrent chains and explore when it is possible to produce such a simulation algorithm for these chains. We introduce a class of chains which we name tame, for which we show that perfect simulation is possible.
In this paper, we investigate the properties of recurrent planar Markov random walks. More precisely, we study the set of recurrent points with the use of local limit theorems. The Nagaev-Guivarc'h spectral method provides several examples for which these local limit theorems are satisfied as soon as the (standard or non-standard) central limit theorem holds.
The paper presents new asymptotic recurrent algorithms of phase space reduction for regularly and singularly perturbed semi-Markov processes. These algorithms give effective conditions of weak convergence for distributions and convergence of expectations for hitting times as well as recurrent formulas for computing the corresponding normalisation functions, Laplace transforms for limiting distributions and limits for expectations.
The Doeblin Graph of a countable state space Markov chain describes the joint pathwise evolutions of the Markov dynamics starting from all possible initial conditions, with two paths coalescing when they reach the same point of the state space at the same time. Its Bridge Doeblin subgraph only contains the paths starting from a tagged point of the state space at all possible times. In the irreducible, aperiodic, and positive recurrent case, the following results are known: the Bridge Doeblin Graph is an infinite tree that is unimodularizable. Moreover, it contains a single bi-infinite path, wh…
We prove a Berry-Esseen theorem and Edgeworth expansions for partial sums of the form $S_N=\sum_{n=1}^{N}f_n(X_n,X_{n+1})$, where $\{X_n\}$ is a uniformly elliptic inhomogeneous Markov chain and $\{f_n\}$ is a sequence of uniformly bounded functions. The Berry-Esseen theorem holds without additional assumptions, while expansions of order $1$ hold when $\{f_n\}$ is irreducible, which is an optimal condition. For higher order expansions, we then focus on two situations. The first is when the essential supremum of $f_n$ is of order $O(n^{-\be})$ for some $\be\in(0,1/2)$. In this case it turns out…
We study positive recurrence and transience of a two-station network in which the behavior of the server in each station is governed by a Markov chain with a finite number of server states; this service process can represent various service disciplines such as a non-preemptive priority service and K-limited service. Assuming that exogenous customers arrive according to independent Markovian arrival processes (MAPs), we represent the behavior of the whole network as a continuous-time Markov chain and, by the uniformization technique, obtain the corresponding discrete-time Markov chain, which is…
We present a recurrence-transience classification for discrete-time Markov chains on manifolds with negative curvature. Our classification depends only on geometric quantities associated to the increments of the chain, defined via the Riemannian exponential map. We deduce that there exist Markov chains on a large class of such manifolds which are both recurrent and have zero average drift at every point. We give an explicit example of such a chain on hyperbolic space of arbitrary dimension, and also on a stochastically incomplete manifold. We also prove that such recurrent chains cannot be uni…
Let $\{X_n\}$ be a Markov chain with transition probability $p_{ij}=a_{j-(i-1)^+},\forall i,j\ge 0$, where $a_j=0$ provided $j<0$, $a_0>0$, $a_0+a_1<1$ and $\sum_{n=0}^\infty a_n=1$. Let $μ=\sum_{n=1}^\infty na_n$. It's known that $\{X_n\}$ is positive recurrent when $μ<1$; is null recurrent when $μ=1$; and is transient when $μ>1$. In this paper, we shall discuss the first returning speed and the last exit speed more precisely by means of $\{a_n\}$
Let $(M_{n},S_{n})_{n\ge 0}$ be a Markov random walk with positive recurrent driving chain $(M_{n})_{n\ge 0}$ having countable state space $\mathcal{S}$ and stationary distribution $π$. It is shown in this note that, if the dual sequence $({}^{\#}M_{n},{}^{\#}S_{n})_{n\ge 0}$ is positive divergent, i.e. ${}^{\#}S_{n}\to\infty$ a.s., then the strictly ascending ladder epochs $σ_{n}^{>}$ of $(M_{n},S_{n})_{n\ge 0}$ are a.s. finite and the ladder chain $(M_{σ_{n}^{>}})_{n\ge 0}$ is positive recurrent on some $\mathcal{S}^{>}\subset\mathcal{S}$. We also provide simple expressions for its stationar…
We prove a large-deviation principle (LDP) for the sample paths of jump Markov processes in the small noise limit when, possibly, all the jump rates vanish uniformly, but slowly enough, in a region of the state space. We further discuss the optimality of our assumptions on the decay of the jump rates. As a direct application of this work we relax the assumptions needed for the application of LDPs to, e.g., Chemical Reaction Network dynamics, where vanishing reaction rates arise naturally particularly the context of mass action kinetics.
We investigate recurrence and transience of Branching Markov Chains (BMC) in discrete time. Branching Markov Chains are clouds of particles which move (according to an irreducible underlying Markov Chain) and produce offspring independently. The offspring distribution can depend on the location of the particle. If the offspring distribution is constant for all locations, these are Tree-Indexed Markov chains in the sense of \cite{benjamini94}. Starting with one particle at location $x$, we denote by $α(x)$ the probability that $x$ is visited infinitely often by the cloud. Due to the irreducibil…
The complexity of the algorithm Radix Selection is considered for independent data generated from a Markov source. The complexity is measured by the number of bucket operations required and studied as a stochastic process indexed by the ranks; also the case of a uniformly chosen rank is considered. The orders of mean and variance of the complexity and limit theorems are derived. We find weak convergence of the appropriately normalized complexity towards a Gaussian process with explicit mean and covariance functions (in the space D[0,1] of cadlag functions on [0,1] with the Skorokhod metric) fo…
It is well known that for a strictly stationary, reversible, Harris recurrent Markov chain, the $ρ$-mixing condition is equivalent to geometric ergodicity and to a "spectral gap" condition. In this note, it will be shown with an example that for that class of Markov chains, the "interlaced" variant of the $ρ$-mixing condition fails to be equivalent to those conditions.
A $ϕ$-irreducible and aperiodic Markov chain with stationary probability distribution will converge to its stationary distribution from almost all starting points. The property of Harris recurrence allows us to replace ``almost all'' by ``all,'' which is potentially important when running Markov chain Monte Carlo algorithms. Full-dimensional Metropolis--Hastings algorithms are known to be Harris recurrent. In this paper, we consider conditions under which Metropolis-within-Gibbs and trans-dimensional Markov chains are or are not Harris recurrent. We present a simple but natural two-dimensional…
If $(C_n)$ is a Markov chain on a discrete state space ${\mathcal{S}}$, a Markov chain $(C_n,M_n)$ on the product space ${\mathcal{S}}\times{\mathcal{S}}$, the cat and mouse Markov chain, is constructed. The first coordinate of this Markov chain behaves like the original Markov chain and the second component changes only when both coordinates are equal. The asymptotic properties of this Markov chain are investigated. A representation of its invariant measure is, in particular, obtained. When the state space is infinite it is shown that this Markov chain is in fact null recurrent if the initial…
In this paper, we give quantitative bounds on the $f$-total variation distance from convergence of an Harris recurrent Markov chain on an arbitrary under drift and minorisation conditions implying ergodicity at a sub-geometric rate. These bounds are then specialized to the stochastically monotone case, covering the case where there is no minimal reachable element. The results are illustrated on two examples from queueing theory and Markov Chain Monte Carlo.
We are interested in the asymptotic behavior of Markov chains on the set of positive integers for which, loosely speaking, large jumps are rare and occur at a rate that behaves like a negative power of the current state, and such that small positive and negative steps of the chain roughly compensate each other. If $X_{n}$ is such a Markov chain started at $n$, we establish a limit theorem for $\frac{1}{n}X_{n}$ appropriately scaled in time, where the scaling limit is given by a nonnegative self-similar Markov process. We also study the asymptotic behavior of the time needed by $X_{n}$ to reach…
We give recurrence and transience criteria for two cases of time-homogeneous Markov chains on the real line with transition kernel $p(x,dy)=f_x(y-x)dy$, where $f_x(y)$ are probability densities of symmetric distributions and, for large $|y|$, have a power-law decay with exponent $α(x)+1$, with $α(x)\in(0,2)$. If $f_x(y)$ is the density of a symmetric $α$-stable distribution for negative $x$ and the density of a symmetric $β$-stable distribution for non-negative $x$, where $α,β\in(0,2)$, then the chain is recurrent if and only if $α+β\geq2.$ If the function $x\longmapsto f_x$ is periodic and if…
In this paper we investigate the local limit theorem for additive functionals of a nonstationary Markov chain with finite or infinite second moment. The moment conditions are imposed on the individual summands and the weak dependence structure is expressed in terms of some uniformly mixing coefficients.
In this paper, the recurrent events that can occur more than one over the follow-up time have been modeled by phase-type distributions. We use the finite-state continuous-time Markov process with multi states for patients with recurrent events. The number of recurrences until time $t$, the time stay for every state and the time till death are of importances. The time till death is assumed to have a phase-type distribution (which is defined in a Markov chain environment) with interpretable parameters. The underlying continuous-time Markov chain has one absorbing state (death) and transient stat…
We consider a Markov chain on $R^+$ with asymptotically zero drift and finite second moments of jumps which is positive recurrent. A power-like asymptotic behaviour of the invariant tail distribution is proven; such a heavy-tailed invariant measure happens even if the jumps of the chain are bounded. Our analysis is based on test functions technique and on construction of a harmonic function.
Suppose $ E$ is a space with a null-recurrent Markov kernel $ P$. Furthermore, suppose there are infinite particles with variable weights on $ E$ performing a random walk following $ P$. Let $ X_{t}$ be a weighted functional of the position of particles at time $ t$. Under some conditions on the initial distribution of the particles the process $ (X_{t})$ is stationary over time. Non-Gaussian infinitely divisible (ID) distributions turn out to be natural candidates for the initial distribution and then the process $ (X_{t})$ is ID. We prove a functional large and moderate deviation principle f…
Using ideas borrowed from topological dynamics and ergodic theory we introduce topological and metric versions of the recurrence property for general Markov chains. The main question of interest here is how large is the set of recurrent points. We show that under some mild technical assumptions the set of non recurrent points is of zero reference measure. Necessary and sufficient conditions for a reference measure $m$ (which needs not to be dynamically invariant) to satisfy this property are obtained. These results are new even in the purely deterministic setting.
In this paper, we are devoted to developing matrix-analytic methods for solving Poisson's equation for irreducible and positive recurrent discrete-time Markov chains (DTMCs). Two special solutions, including the deviation matrix D and the expected additive-type functional matrix K, will be considered. The results are applied to Markov chains of GI/G/1-type and MAP/G/1 queues with negative customers. Further extensions to continuous-time Markov chains (CTMCs) are also investigated.
In this paper we study the global boundedness for the solutions to a class of possibly degenerate parabolic equations by De-Giorgi's iteration. As applications, we show the existence of weak solutions for possibly degenerate stochastic differential equations with singular diffusion and drift coefficients. Moreover, by the Markov selection theorem of Krylov [8], we also establish the existence of the associated strong Markov family.
This paper proposes a new type of recurrence where we divide the Markov chains into intervals that start when the chain enters into a subset A, then sample another subset B far away from A and end when the chain again return to A. The length of these intervals have the same distribution and if A and B are far apart, almost independent of each other. A and B may be any subsets of the state space that are far apart of each other and such that the movement between the subsets is repeated several times in a long Markov chain. The expected length of the intervals is used in a function that describe…
We prove a new concentration inequality for U-statistics of order two for uniformly ergodic Markov chains. Working with bounded and $π$-canonical kernels, we show that we can recover the convergence rate of Arcones and Gin{é} who proved a concentration result for U-statistics of independent random variables and canonical kernels. Our result allows for a dependence of the kernels $h_{i,j}$ with the indexes in the sums, which prevents the use of standard blocking tools. Our proof relies on an inductive analysis where we use martingale techniques, uniform ergodicity, Nummelin splitting and Bernst…
It is known that random walk Metropolis algorithms with heavy-tailed target densities can model atypical (slow) growth of maxima, which in general is exhibited by processes with the extremal index zero. The asymptotics of maxima of such sequences can be analyzed in terms of continuous phantom distribution functions. We show that in a large class of positive Harris recurrent Markov chains (containing the above Metropolis chains) a phantom distribution function can be recovered by starting "at the point" rather than from the stationary distribution.
This paper addresses the question when the underlying Markov process of a multiclass queueing network is positive Harris recurrent. It is well-known that stability of the fluid limit model is a sufficient condition for this. Hence, stability of fluid (limit) models is of vital interest. Recently, it has been shown that if the fluid model satisfies certain properties, it is stable if and only if there exists a Lyapunov function. In this paper a new method is provided to conclude that the underlying Markov process is positive Harris recurrent if the fluid model is stable by using explicitly the …
Given a non-negative Jacobi matrix describing higher order recurrence relations for multiple orthogonal polynomials of type~II and corresponding linear forms of type I, a general strategy for constructing a pair of stochastic matrices, dual to each other, is provided. The corresponding Markov chains (or 1D random walks) allow, in one transition, to reach for the N-th previous states, to remain in the state or reach for the immediately next state. The dual Markov chains allow, in one transition, to reach for the N-th next states, to remain in the state or reach for immediately previous state. T…
A rescaled Markov chain converges uniformly in probability to the solution of an ordinary differential equation, under carefully specified assumptions. The presentation is much simpler than those in the outside literature. The result may be used to build parsimonious models of large random or pseudo-random systems.
This paper discusses the irreducibility and geometric ergodicity of the Hamiltonian Monte Carlo (HMC) algorithm. We consider cases where the number of steps of the symplectic integrator is either fixed or random. Under mild conditions on the potential $\F$ associated with target distribution $π$, we first show that the Markov kernel associated to the HMC algorithm is irreducible and recurrent. Under more stringent conditions, we then establish that the Markov kernel is Harris recurrent. Finally, we provide verifiable conditions on $\F$ under which the HMC sampler is geometrically ergodic.
We develop a new algorithm for the estimation of rare event probabilities associated with the steady-state of a Markov stochastic process with continuous state space $\mathbb R^d$ and discrete time steps (i.e. a discrete-time $\mathbb R^d$-valued Markov chain). The algorithm, which we coin Recurrent Multilevel Splitting (RMS), relies on the Markov chain's underlying recurrent structure, in combination with the Multilevel Splitting method. Extensive simulation experiments are performed, including experiments with a nonlinear stochastic model that has some characteristics of complex climate mode…
In the analysis of Markov chains and processes, it is sometimes convenient to replace an unbounded state space with a "truncated" bounded state space. When such a replacement is made, one often wants to know whether the equilibrium behavior of the truncated chain or process is close to that of the untruncated system. For example, such questions arise naturally when considering numerical methods for computing stationary distributions on unbounded state space. In this paper, we use the principle of "regeneration" to show that the stationary distributions of "fixed state" truncations converge in …
In this paper we find nonasymptotic exponential upper bounds for the deviation in the ergodic theorem for families of homogeneous Markov processes. We find some sufficient conditions for geometric ergodicity uniformly over a parametric family. We apply this property to the nonasymptotic nonparametric estimation problem for ergodic diffusion processes.
We propose a new statistical model for computational linguistics. Rather than trying to estimate directly the probability distribution of a random sentence of the language, we define a Markov chain on finite sets of sentences with many finite recurrent communicating classes and define our language model as the invariant probability measures of the chain on each recurrent communicating class. This Markov chain, that we call a communication model, recombines at each step randomly the set of sentences forming its current state, using some grammar rules. When the grammar rules are fixed and known …
