We give a development of the ODE method for the analysis of recursive algorithms described by a stochastic recursion. With variability modelled via an underlying Markov process, and under general assumptions, the following results are obtained: 1. Stability of an associated ODE implies that the stochastic recursion is stable in a strong sense when a gain parameter is small. 2. The range of gain-values is quantified through a spectral analysis of an associated linear operator, providing a non-local theory. 3. A second-order analysis shows precisely how variability leads to sensitivity of the algorithm with respect to the gain parameter.
All results are obtained within the natural operator-theoretic framework of geometrically ergodic Markov processes.
In a companion paper, the authors have characterized all deterministic semigroups, and all Markov semigroups, whose trajectories are Carathe'odory solutions to a given ODE x'=f(x), with f possibly discontinuous. The present paper establishes two approximation results. Namely, every deterministic semigroup can be obtained as the pointwise limit of the flows generated by a sequence of ODEs $x'=f_n(x) with smooth right hand sides. Moreover, every Markov semigroup can be obtained as limit of a sequence of diffusion processes with smooth drifts and with diffusion coefficients approaching zero.
Fluid limit techniques have become a central tool to analyze queueing networks over the last decade, with applications to performance analysis, simulation and optimization. In this paper, some of these techniques are extended to a general class of skip-free Markov chains. As in the case of queueing models, a fluid approximation is obtained by scaling time, space and the initial condition by a large constant. The resulting fluid limit is the solution of an ordinary differential equation (ODE) in ``most'' of the state space. Stability and finer ergodic properties for the stochastic model then fo…
The focus of this work is on local stability of a class of nonlinear ordinary differential equations (ODE) that describe limits of empirical measures associated with finite-state weakly interacting N-particle systems. Local Lyapunov functions are identified for several classes of such ODE, including those associated with systems with slow adaptation and Gibbs systems. Using results from [5] and large deviations heuristics, a partial differential equation (PDE) associated with the nonlinear ODE is introduced and it is shown that positive definite subsolutions of this PDE serve as local Lyapunov…
Mean-field models approximate large stochastic systems by simpler differential equations that are supposed to approximate the mean of the larger system. It is generally assumed that as the stochastic systems get larger (i.e., more people or particles), they converge to the mean-field models. Mean-field models are common in many fields, but their convergence is rarely proved. The existing approaches rely on operator semigroups, martingales, PDEs, or infinite systems of ODEs. We give a general proof for their convergence using only Taylor's theorem and basic ODE results. We hope this allows appl…
We study an ordinary differential equation (ODE) arising as the many-server heavy-traffic fluid limit of a sequence of overloaded Markovian queueing models with two customer classes and two service pools. The system, known as the X model in the call-center literature, operates under the fixed-queue-ratio-with-thresholds (FQR-T) control, which we proposed in a recent paper as a way for one service system to help another in face of an unanticipated overload. Each pool serves only its own class until a threshold is exceeded; then one-way sharing is activated with all customer-server assignments t…
We study (backward) stochastic differential equations with noise coming from a finite state Markov chain. We show that, for the solutions of these equations to be `Markovian', in the sense that they are deterministic functions of the state of the underlying chain, the integrand must be of a specific form. This allows us to connect these equations to coupled systems of ODEs, and hence to give fast numerical methods for the evaluation of Markov-Chain BSDEs.
Many systems in biology, physics and chemistry can be modeled through ordinary differential equations, which are piecewise smooth, but switch between different states according to a Markov jump process. In the fast switching limit, the dynamics converges to a deterministic ODE. In this paper we suppose that this limit ODE supports a stable limit cycle. We demonstrate that a set of such oscillators can synchronize when they are uncoupled, but they share the same switching Markov jump process. The latter is taken to represent the effect of a common randomly switching environment. We determine th…
In this paper we study the spectral properties of Markov-operator on $L^{2}$-spaces. Lawler and Sokal (Trans. Amer. Math. Soc., 1988, 309, pp. 557-580) used isoperimetric constants for discrete and continuous time Markov chains to obtain a spectral gap at 1. For time discrete Markov chains this does not exclude periodic behavior. We define a new constant measuring the distance from periodicity and give necessary and sufficient conditions for the existence of a global spectral gap in terms of this constant.
We investigate the spectral distribution of large sample covariance matrices with independent columns and entries in the columns that stem from Markov chains. We characterize the limiting spectral densities by their moments. Correspondingly, the proof is based on a moment method.
We describe stochastic calculus in the context of processes that are driven by an adapted point process of locally finite intensity and are differentiable between jumps. This includes Markov chains as well as non-Markov processes. By analogy with Itô processes we define the drift and diffusivity, which we then use to describe a general sample path estimate. We then give several examples, including ODE approximation, processes with linear drift, first passage times, and an application to the stochastic logistic model.
The theory of $L^2$-spectral gaps for reversible Markov chains has been studied by many authors. In this paper we consider positive recurrent general state space Markov chains with stationary transition probabilities. Replacing the assumption of reversibility by a less strong one, we still obtain a simple necessary and sufficient condition for the spectral gap property of the associated Markov operator in terms of isoperimetric constant. Moreover, we define a new sequence of isoperimetric constants which provides a necessary and sufficient condition for the existence of a spectral gap in a ver…
We give a bound on the mixing time of a uniformly ergodic, reversible Markov chain in terms of the spectral radius of the transition operator. This bound has been established previously in finite state spaces, and is widely believed to hold in general state spaces, but a proof has not been provided to our knowledge.
We study networks of biochemical reactions modelled by continuous-time Markov processes. Such networks typically contain many molecular species and reactions and are hard to study analytically as well as by simulation. Particularly, we are interested in reaction networks with intermediate species such as the substrate-enzyme complex in the Michaelis-Menten mechanism. These species are virtually in all real-world networks, they are typically short-lived, degraded at a fast rate and hard to observe experimentally. We provide conditions under which the Markov process of a multiscale reaction netw…
New relations between ergodic rate, L_p convergence rates, and asymptotic behavior of tail probabilities for hitting times of a time homogeneous Markov process are established. For L_p convergence rates and related spectral and functional properties (spectral gap and Poincare inequality) sufficient conditions are given in the terms of an exponential ϕ-coupling. This provides sufficient conditions for L_p convergence rates in the terms of appropriate combination of `local mixing' and `recurrence' conditions on the initial process, typical in the ergodic theory of Markov processes. The range of …
Suppose that $\{X_t,\,t\ge0\}$ is a non-stationary Markov process, taking values in a Polish metric space $E$. We prove the law of large numbers and central limit theorem for an additive functional of the form $\int_0^Tψ(X_s)ds$, provided that the dual transition probability semigroup, defined on measures, is strongly contractive in an appropriate Wasserstein metric. Function $ψ$ is assumed to be Lipschitz on $E$.
We start by introducing a nonlinear involution operator which maps the space of solutions of Sturm-Liouville equations into the space of solutions of the associated equations which turn out to be nonlinear ordinary differential equations. We study some algebraic and analytical properties of this involution operator as well as some properties of a two-parameter family of operators describing the set of solutions of Sturm-Liouville equations. Next, we show how a specific composition of these mappings allows to connect, by means of a simple analytical expression, the law of the first passage time…
We prove a version of McDiarmid's bounded differences inequality for Markov chains, with constants proportional to the mixing time of the chain. We also show variance bounds and Bernstein-type inequalities for empirical averages of Markov chains. In the case of non-reversible chains, we introduce a new quantity called the "pseudo spectral gap", and show that it plays a similar role for non-reversible chains as the spectral gap plays for reversible chains. Our techniques for proving these results are based on a coupling construction of Katalin Marton, and on spectral techniques due to Pascal Le…
Let $\{X_n\}_{n\in\N}$ be a Markov chain on a measurable space $\X$ with transition kernel $P$ and let $V:\X\r[1,+\infty)$. The Markov kernel $P$ is here considered as a linear bounded operator on the weighted-supremum space $\cB_V$ associated with $V$. Then the combination of quasi-compactness arguments with precise analysis of eigen-elements of $P$ allows us to estimate the geometric rate of convergence $ρ_V(P)$ of $\{X_n\}_{n\in\N}$ to its invariant probability measure in operator norm on $\cB_V$. A general procedure to compute $ρ_V(P)$ for discrete Markov random walks with identically dist…
It is well known, mainly because of the work of Kurtz, that density dependent Markov chains can be approximated by sets of ordinary differential equations (ODEs) when their indexing parameter grows very large. This approximation cannot capture the stochastic nature of the process and, consequently, it can provide an erroneous view of the behavior of the Markov chain if the indexing parameter is not sufficiently high. Important phenomena that cannot be revealed include non-negligible variance and bi-modal population distributions. A less-known approximation proposed by Kurtz applies stochastic …
We propose an exact technique to calculate lower bounds of spectral gaps of discrete time reversible Markov chains on finite state sets. Spectral gaps are a common tool for evaluating convergence rates of Markov chains. As an illustration, we successfully use this technique to evaluate the ``absorption time'' of the ``Backgammon model'', a paradigmatic model for glassy dynamics. We also discuss the application of this technique to the ``Contingency table problem'', a notoriously difficult problem from probability theory. The interest of this technique is that it connects spectral gaps, which a…
Consider a Markov chain $(X_n)_{n\geqslant 0}$ with values in the state space $\mathbb X$. Let $f$ be a real function on $\mathbb X$ and set $S_0=0,$ $S_n = f(X_1)+\cdots + f(X_n),$ $n\geqslant 1$. Let $\mathbb P_x$ be the probability measure generated by the Markov chain starting at $X_0=x$. For a starting point $y \in \mathbb R$ denote by $τ_y$ the first moment when the Markov walk $(y+S_n)_{n\geqslant 1}$ becomes non-positive. Under the condition that $S_n$ has zero drift, we find the asymptotics of the probability $\mathbb P_x ( τ_y >n )$ and of the conditional law $\mathbb P_x ( y+S_n\leq…
We consider Markov chain with spectral gap in $L^2$ space. Assume that $f$ is a bounded function. Then the probabilities of large deviations of average along trajectory satisfy Hoeffding's-type inequalities. These bounds depend only on the stationary mean, spectral gap and the end-points of support of $f$.
We compute the eigenvalues and eigenspaces of random-to-random Markov chains. We use a family of maps which reveal a remarkable recursive structure of the eigenspaces, yielding an explicit and effective construction of all eigenbases starting from bases of the kernels.
It is shown that if a Markov map $T$ on a noncommutative probability space $\mathcal{M}$ has a spectral gap on $L_2(\mathcal{M})$, then it also has one on $L_p(\mathcal{M})$ for $1<p<\infty$. For fixed $p$, the converse also holds if $T$ is factorizable. These results are also new for classical probability spaces.
We consider a collection of Markov chains that model the evolution of multitype biological populations. The state space of the chains is the positive orthant, and the boundary of the orthant is absorbing representing the extinction states of different population types. We are interested in the long-term behavior of the Markov chain away from extinction, under a small noise scaling. Under this scaling, the trajectory of the Markov process over any compact interval converges in distribution to the solution of an ordinary differential equation (ODE) evolving in the positive orthant. We study the …
The essential spectral radius of a sub-Markovian process is defined as the infimum of the spectral radiuses of all local perturbations of the process. When the family of rescaled processes satisfies sample path large deviation principle, the spectral radius and the essential spectral radius are expressed in terms of the rate function. The paper is motivated by applications to reflected diffusions and jump Markov processes describing stochastic networks for which the sample path large deviation principle has been established and the rate function has been identified while essential spectral rad…
In the present paper deals with asymptotical stability of Markov operators acting on abstract state spaces (i.e. an ordered Banach space, where the norm has an additivity property on the cone of positive elements). Basically, we are interested in the rate of convergence when a Markov operator $T$ satisfies the uniform $P$-ergodicity, i.e. $\|T^n-P\|\to 0$, here $P$ is a projection. We have showed that $T$ is uniformly $P$-ergodic if and only if $\|T^n-P\|\leq Cβ^n$, $0<β<1$. In this paper, we prove that such a $β$ is characterized by the spectral radius of $T-P$. Moreover, we give Deoblin's ki…
In this paper we extend the results of the research started by the first author, in which Karlin-McGregor diagonalization of certain reversible Markov chains over countably infinite general state spaces by orthogonal polynomials was used to estimate the rate of convergence to a stationary distribution. We use a method of Koornwinder to generate a large and interesting family of random walks which exhibits a lack of spectral gap, and a polynomial rate of convergence to the stationary distribution. For the Chebyshev type subfamily of Markov chains, we use asymptotic techniques to obtain an upper…
We argue that the spectral theory of non-reversible Markov chains may often be more effectively cast within the framework of the naturally associated weighted-$L_\infty$ space $L_\infty^V$, instead of the usual Hilbert space $L_2=L_2(π)$, where $π$ is the invariant measure of the chain. This observation is, in part, based on the following results. A discrete-time Markov chain with values in a general state space is geometrically ergodic if and only if its transition kernel admits a spectral gap in $L_\infty^V$. If the chain is reversible, the same equivalence holds with $L_2$ in place of $L_\i…
The spectral gap of a Markov chain can be bounded by the spectral gaps of constituent "restriction" chains and a "projection" chain, and the strength of such a bound is the content of various decomposition theorems. In this paper, we introduce a new parameter that allows us to improve upon these bounds. We further define a notion of orthogonality between the restriction chains and "complementary" restriction chains. This leads to a new Complementary Decomposition theorem, which does not require analyzing the projection chain. For $ε$-orthogonal chains, this theorem may be iterated $O(1/ε)$ tim…
Spectral theory for Markov chains with transition matrix admitting a stochastic bidiagonal factorization
Universidade de Coimbra（科英布拉大学）；Universidade de Aveiro（阿威罗大学）；Universidad Complutense de Madrid（马德里康普顿斯大学）；Instituto de Ciencias Matematicas (ICMAT)（数学科学研究所）
