Consider an ergodic Markov chain on a countable state space for which the return times have exponential tails. We show that the stationary version of any such chain is a finitary factor of an i.i.d. process. A key step is to show that any stationary renewal process whose jump distribution has exponential tails and is not supported on a proper subgroup of $\mathbb{Z}$ is a finitary factor of an i.i.d. process.
A consequence of Ornstein theory is that the infinite entropy flows associated with Poisson processes and continuous-time irreducible Markov chains on a finite number of states are isomorphic as measure-preserving systems. We give an elementary construction of such an isomorphism that has an additional finitariness property, subject to the additional conditions that the Markov chain has a uniform holding rate and a mixing skeleton.
In 1977, Keane and Smorodinsky showed that there exists a finitary homomorphism from any finite-alphabet Bernoulli process to any other finite-alphabet Bernoulli process of strictly lower entropy. In 1996, Serafin proved the existence of a finitary homomorphism with finite expected coding length. In this paper, we construct such a homomorphism in which the coding length has exponential tails. Our construction is source-universal, in the sense that it does not use any information on the source distribution other than the alphabet size and a bound on the entropy gap between the source and target…
It has been shown by van den Berg and Steif that the sub-critical Ising model on $\mathbb{Z}^d$ is a finitary factor of a finite-valued i.i.d. process. We strengthen this by showing that the factor map can be made to have finite expected coding volume (in fact, stretched-exponential tails), answering a question of van den Berg and Steif. The result holds at any temperature above the critical temperature. An analogous result holds for Markov random fields satisfying a high-noise assumption and for proper colorings with a large number of colors.
It has been shown by van den Berg and Steif that the sub-critical and critical Ising model on $\mathbb{Z}^d$ is a finitary factor of an i.i.d. process (ffiid), whereas the super-critical model is not. In fact, they showed that the latter is a general phenomenon in that a phase transition presents an obstruction for being ffiid. The question remained whether this is the only such obstruction. We make progress on this, showing that certain spatial mixing conditions (notions of weak dependence on boundary conditions, not to be confused with other notions of mixing in ergodic theory) imply ffiid. …
This report presents some fundamental mathematical results towards elucidating the information-geometric underpinnings of evolutionary modelling schemes for (quasi-)stationary discrete stochastic processes. The model class under consideration is that of finite causal-state processes, known from the computational mechanics programme, along with their minimal unifilar hidden Markov generators. The respective configuration space is exhibited as a collection of combinatorially related Riemannian manifolds wherein the metric tensor field is an infinitesimal version of the relative entropy rate. Fur…
We consider a discrete-time Markov decision process with Borel state and action spaces. The performance criterion is to maximize a total expected {utility determined by unbounded return function. It is shown the existence of optimal strategies under general conditions allowing the reward function to be unbounded both from above and below and the action sets available at each step to the decision maker to be not necessarily compact. To deal with unbounded reward functions, a new characterization for the weak convergence of probability measures is derived. Our results are illustrated by examples…
Let $\{X_n\}$ be a Markov chain with transition probability $p_{ij}=a_{j-(i-1)^+},\forall i,j\ge 0$, where $a_j=0$ provided $j<0$, $a_0>0$, $a_0+a_1<1$ and $\sum_{n=0}^\infty a_n=1$. Let $μ=\sum_{n=1}^\infty na_n$. It's known that $\{X_n\}$ is positive recurrent when $μ<1$; is null recurrent when $μ=1$; and is transient when $μ>1$. In this paper, we shall discuss the first returning speed and the last exit speed more precisely by means of $\{a_n\}$
We consider perturbations of positive recurrent Markov modulated fluid models. In addition to the infinitesimal generator of the phases, we also perturb the rate matrix, and analyze the effect of those perturbations on the matrix of first return probabilities to the initial level. Our main contribution is the construction of a substitute for the matrix of first return probabilities, which enables us to analyze the effect of the perturbation under consideration.
We consider moments of the return times (or first hitting times) in a discrete time discrete space Markov chain. It is classical that the finiteness of the first moment of a return time of one state implies the finiteness of the first moment of the first return time of any other state. We extend this statement to moments with respect to a function $f$, where $f$ satisfies a certain, best possible condition. This generalizes results of K. L. Chung (1954) who considered the functions $f(n)=n^p$ and wondered "[...] what property of the power $n^p$ lies behind this theorem [...]" (see Chung (1967)…
We study the long time behaviour of a Markov process evolving in $\mathbb{N}$ and conditioned not to hit 0. Assuming that the process comes back quickly from infinity, we prove that the process admits a unique quasi-stationary distribution (in particular, the distribution of the conditioned process admits a limit when time goes to infinity). Moreover, we prove that the distribution of the process converges exponentially fast in total variation norm to its quasi-stationary distribution and we provide an explicit rate of convergence. As a first application of our result, we bring a new insight o…
In this paper we prove two results. First we show that dynamical systems with a $ϕ$-mixing measure have in the limit Poisson distributed return times almost everywhere. We use the Chen-Stein method to also obtain rates of convergence. Our theorem improves on previous results by allowing for infinite partitions and dropping the requirement that the invariant measure have finite entropy with respect to the given partition. As has been shown elsewhere, the limiting distribution at periodic points is not Poissonian (but compound Poissonian). Here we show that for all non-periodic points the return…
General characterizations of ergodic Markov chains have been developed in considerable detail. In this paper, we study the transience for discrete-time Markov chains on general state spaces, including the geometric transience and algebraic transience. Criteria are presented through establishing the drift condition and considering the first return time. As an application, we give explicit criteria for the random walk on the half line and the skip-free chain on nonnegative integers.
We consider a form of state-dependent drift condition for a general Markov chain, whereby the chain subsampled at some deterministic time satisfies a geometric Foster-Lyapunov condition. We present sufficient criteria for such a drift condition to exist, and use these to partially answer a question posed by Connor & Kendall (2007) concerning the existence of so-called 'tame' Markov chains. Furthermore, we show that our 'subsampled drift condition' implies the existence of finite moments for the return time to a small set.
Extensions of Kemeny's constant, as derived for irreducible finite Markov chains in discrete time, to Markov renewal processes and Markov chains in continuous time are discussed. Three alternative Kemeny's functions and their variants are considered. Typically, they lead to a constant if and only if the mean holding times between the states in the Markov renewal process are constant. However one particular variant leads to a constant, analogous to the discrete time Markov chain result. Specifically, if the state space is finite, the weighted sum of the mean first passage times (omitting the me…
We study perturbation theory and uniform ergodicity for discrete-time Markov chains on general state spaces in terms of the uniform moments of the first hitting times on some set. The methods we adopt are different from previous ones. For reversible and non-negative definite Markov chains, we first investigate the geometrically ergodic convergence rates. Based on the estimates, together with a first passage formula, we then get the convergence rates in uniform ergodicity. If the transition kernel $P$ is only reversible, we transfer to study the two-skeleton chain with the transition kernel $P^…
Poisson's equation has a lot of applications in various areas. Usually it is hard to derive the explicit expression of the solution of Poisson's equation for a Markov chain on an infinitely many state space. We will present a computational framework for the solution for both discrete-time Markov chains (DTMCs) and continuous-time Markov chains (CTMCs), by developing the technique of augmented truncation approximations. The convergence to the solution is investigated in terms of the assumption about the monotonicity of the first return times, and is further established for two types of truncati…
We study distributions of meeting times for finite symmetric Markov chains. For Markov kernels defined on large state spaces which satisfy certain weak inhomogeneity in return probabilities of points up to large numbers of steps, we obtain approximation, with explicit error bounds, of the Laplace transforms of some meeting times (without scaling) by ratios of Green functions closely related to hitting times of points. In studying this approximation, we identify some key matrix power series in Markov kernels weighted with solutions to a discrete transport-like equation with explicit coefficient…
We justify and discuss expressions for joint lower and upper expectations in imprecise probability trees, in terms of the sub- and supermartingales that can be associated with such trees. These imprecise probability trees can be seen as discrete-time stochastic processes with finite state sets and transition probabilities that are imprecise, in the sense that they are only known to belong to some convex closed set of probability measures. We derive various properties for their joint lower and upper expectations, and in particular a law of iterated expectations. We then focus on the special cas…
We study how memory impacts passages at the origin for a so-called elephant random walk in the diffusive regime. We observe that the number of zeros always grows asymptotically like the square root of the time, despite the fact that, depending on the memory parameter, first return times to $0$ may have a finite expectation or a fat tail with exponent less than $1/2$. We resolve this apparent paradox by recasting the questions in the framework of scaling limits for Markov chains and self-similar Markov processes.
We study the so-called pinning model, which describes the behavior of a Markov chain interacting with a distinguished state. The interaction depends on an external source of randomness, called disorder, which can attract or repel the Markov chain path. We focus on the case when the disorder is heavy-tailed, with infinite mean, while the return times of the Markov chain have a stretched-exponential distribution. We prove that the set of times at which the Markov chain visits the distinguished state, suitably rescaled, converges in distribution to a limit set, which depends only on the disorder …
We study a large class of reversible Markov chains with discrete state space and transition matrix $P_N$. We define the notion of a set of {\it metastable points} as a subset of the state space $\G_N$ such that (i) this set is reached from any point $x\in \G_N$ without return to x with probability at least $b_N$, while (ii) for any two point x,y in the metastable set, the probability $T^{-1}_{x,y}$ to reach y from x without return to x is smaller than $a_N^{-1}\ll b_N$. Under some additional non-degeneracy assumption, we show that in such a situation: \item{(i)} To each metastable point corres…
Nonlinear Markov Chains (nMC) are regarded as the original (linear) Markov Chains with nonlinear small perturbations. It fits real-world data better, but its associated properties are difficult to describe. A new approach is proposed to analyze the ergodicity and even estimate the convergence bounds of nMC, which is more precise than existing results. In the new method, Coupling Markov about homogeneous Markov chains is applied to reconstitute the relationship between distribution at any times and the limiting distribution. The convergence bounds can be provided by the transition probability m…
Catastrophe Markov chain population models have received a lot of attention in the recent past. We herewith consider two special cases of such models involving total disasters, both in discrete and in continuous-time. Depending on the parameters range, the two models can show up a recurrence/transience transition and, in the critical case, a positive/null recurrence transition. The collapse transition probabilities are chosen in such a way that the models are exactly solvable and, in case of positive recurrence, intimately related to the extended Sibuya and Pareto-Zipf distributions whose divi…
For discrete-time stochastic processes, there is a close connection between return/waiting times and entropy. Such a connection cannot be straightforwardly extended to the continuous-time setting. Contrarily to the discrete-time case one does need a reference measure and so the natural object is relative entropy rather than entropy. In this paper we elaborate on this in the case of continuous-time Markov processes with finite state space. A reference measure of special interest is the one associated to the time-reversed process. In that case relative entropy is interpreted as the entropy produ…
This paper introduces a new algorithm for numerically computing equilibrium (i.e. stationary) distributions for Markov chains and Markov jump processes with either a very large finite state space or a countably infinite state space. The algorithm is based on a ratio representation for equilibrium expectations in which the numerator and denominator correspond to expectations defined over paths that start and end within a given return set $K$. When $K$ is a singleton, this representation is a well-known consequence of regenerative process theory. For computational tractability, we ignore contrib…
