We consider symmetric Markov chains on $\Bbb Z^d$ where we do {\bf not} assume that the conductance between two points must be zero if the points are far apart. Under a uniform second moment condition on the conductances, we obtain upper bounds on the transition probabilities, estimates for exit time probabilities, and certain lower bounds on the transition probabilities. We show that a uniform Harnack inequality holds if an additional assumption is made, but that without this assumption such an inequality need not hold. We establish a central limit theorem giving conditions for a sequence of normalized symmetric Markov chains to converge to a diffusion on $\Bbb R^d$ corresponding to an elliptic operator in divergence form.
We consider a class of continuous time Markov chains on $\Z^d$. These chains are the discrete space analogue of Markov processes with jumps. Under some conditions, we show that harmonic functions associated with these Markov chains are Hölder continuous.
We consider a discrete-time Markov decision process with Borel state and action spaces. The performance criterion is to maximize a total expected {utility determined by unbounded return function. It is shown the existence of optimal strategies under general conditions allowing the reward function to be unbounded both from above and below and the action sets available at each step to the decision maker to be not necessarily compact. To deal with unbounded reward functions, a new characterization for the weak convergence of probability measures is derived. Our results are illustrated by examples…
We present a tail inequality for suprema of empirical processes generated by variables with finite $ψ_α$ norms and apply it to some geometrically ergodic Markov chains to derive similar estimates for empirical processes of such chains, generated by bounded functions. We also obtain a bounded difference inequality for symmetric statistics of such Markov chains.
We show that an isotropic self-similar Markov process in ${\Bbb R}^d$ has a skew product structure if and only if its radial and angular parts do not jump at the same time.
As well-known, transition probabilities of jump Markov processes satisfy Kolmogorov's backward and forward equations. In the seminal 1940 paper, William Feller investigated solutions of Kolmogorov's equations for jump Markov processes. Recently the authors solved the problem studied by Feller and showed that the minimal solution of Kolmogorov's backward and forward equations is the transition probability of the corresponding jump Markov process if the transition rate at each state is bounded. This paper presents more general results. For Kolmogorov's backward equation, the sufficient condition…
In this paper, we consider the control problem with the Average-Value-at-Risk (AVaR) criteria of the possibly unbounded $L^{1}$-costs in infinite horizon on a Markov Decision Process (MDP). With a suitable state aggregation and by choosing a priori a global variable $s$ heuristically, we show that there exist optimal policies for the infinite horizon problem. To our knowledge, this is the first work of deriving dynamic programming equations with $L^1$-unbounded costs via AVaR-operator.
We consider the Markov random flight $\bold X(t)$ in the Euclidean space $\Bbb R^m, \; m\ge 2,$ starting from the origin $\bold 0\in\Bbb R^m$ that, at Poisson-paced times, changes its direction at random according to arbitrary distribution on the unit $(m-1)$-dimensional sphere $S^m(\bold 0,1)$ having absolutely continuous density. For any time instant $t>0$, the convolution-type recurrent relations for the joint and conditional densities of process $\bold X(t)$ and of the number of changes of direction, are obtained. Using these relations, we derive an integral equation for the transition den…
This paper is related to nonzero-sum stochastic differential games in the Markovian framework. We show existence of a Nash equilibrium point for the game when the drift is no longer bounded and only satisfies a linear growth condition. The main tool is the notion of backward stochastic differential equations which, in our case, are multidimensional with continuous coefficient and stochastic linear growth.
In this paper, we establish a probabilistic representation as well as some integration by parts formulae for the marginal law at a given time maturity of some stochastic volatility model with unbounded drift. Relying on a perturbation technique for Markov semigroups, our formulae are based on a simple Markov chain evolving on a random time grid for which we develop a tailor-made Malliavin calculus. Among other applications, an unbiased Monte Carlo path simulation method stems from our formulas so that it can be used in order to numerically compute with optimal complexity option prices as well …
We consider a large class of piecewise expanding maps T of [0,1] with a neutral fixed point, and their associated Markov chain Y_i whose transition kernel is the Perron-Frobenius operator of T with respect to the absolutely continuous invariant probability measure. We give a large class of unbounded functions f for which the partial sums of f\circ T^i satisfy both a central limit theorem and a bounded law of the iterated logarithm. For the same class, we prove that the partial sums of f(Y_i) satisfy a strong invariance principle. When the class is larger, so that the partial sums of f\circ T^i…
We consider a sequence of Markov chains weakly convergent to a diffusion. We suppose that a drift term contains a linearly increasing component. The usual parametrix method fails because of this unbounded drift term. We show how to modify the parametrix method to obtain local limit theorems for this case.
Using the method of transportation-information inequality introduced in \cite{GLWY}, we establish Bernstein type's concentration inequalities for empirical means $\frac 1t \int_0^t g(X_s)ds$ where $g$ is a unbounded observable of the symmetric Markov process $(X_t)$. Three approaches are proposed : functional inequalities approach ; Lyapunov function method ; and an approach through the Lipschitzian norm of the solution to the Poisson equation. Several applications and examples are studied.
Certain Markov processes, or deterministic evolution equations, have the property that they are dual to a stochastic process that exhibits extinction versus unbounded growth, i.e., the total mass in such a process either becomes zero, or grows without bounds as time tends to infinity. If this is the case, then this phenomenon can often be used to determine the invariant measures, or fixed points, of the process originally under consideration, and to study convergence to equilibrium. This principle, which has been known since early work on multitype branching processes, is here demonstrated on …
We study Talagrand concentration and Poincaré type inequalities for unbounded pure jump Markov processes. In particular we focus on processes with degenerate jumps that depend on the past of the whole system, based on the model introduced by Galves and Löcherbach in \cite{G-L}, in order to describe the activity of a biological neural network. As a result we obtain exponential rates of convergence to equilibrium.
We consider a collection of Markov chains that model the evolution of multitype biological populations. The state space of the chains is the positive orthant, and the boundary of the orthant is absorbing representing the extinction states of different population types. We are interested in the long-term behavior of the Markov chain away from extinction, under a small noise scaling. Under this scaling, the trajectory of the Markov process over any compact interval converges in distribution to the solution of an ordinary differential equation (ODE) evolving in the positive orthant. We study the …
In this paper we consider a large class of symmetric Markov processes $X=(X_t)_{t\ge0}$ on $\R^d$ generated by non-local Dirichlet forms, which include jump processes with small jumps of $α$-stable-like type and with large jumps of super-exponential decay. Let $D\subset \R^d$ be an open (not necessarily bounded and connected) set, and $X^D=(X_t^D)_{t\ge0}$ be the killed process of $X$ on exiting $D$. We obtain explicit criterion for the compactness and the intrinsic ultracontractivity of the Dirichlet Markov semigroup $(P^{D}_t)_{t\ge0}$ of $X^D$. When $D$ is a horn-shaped region, we further o…
For a Harris ergodic Markov chain $(X_n)_{n\ge 0}$, on a general state space, started from the so called small measure or from the stationary distribution we provide optimal estimates for Orlicz norms of sums $\sum_{i=0}^τf(X_i)$, where $τ$ is the first regeneration time of the chain. The estimates are expressed in terms of other Orlicz norms of the function $f$ (wrt the stationary distribution) and the regeneration time $τ$ (wrt the small measure). We provide applications to tail estimates for additive functionals of the chain $(X_n)$ generated by unbounded functions as well as to classical l…
We consider the diffusion process and its approximation by Markov chain with nonlinear increasing trends. The usual parametrix method is not appliable because these models have unbounded trends. We describe a procedure that allows to exclude nonlinear growing trend and move to stochastic differential equation with reduced drift and diffusion coefficients. A similar procedure is considered for a Markov chain
A discrete-time Markov chain can be transformed into a new Markov chain by looking at its states along iterations of an almost surely finite stopping time. By the optional stopping theorem, any bounded harmonic function with respect to the transition function of the original chain is harmonic with respect to the transition function of the transformed chain. The reverse inclusion is in general not true. Our main result provides a sufficient condition on the stopping time which guarantees that the space of bounded harmonic functions for the transformed chain embeds in the space of bounded harmon…
We study the problem of exponential mixing and large deviations for discrete-time Markov processes associated with a class of random dynamical systems. Under some dissipativity and regularisation hypotheses for the underlying deterministic dynamics and a non-degeneracy condition for the driving random force, we discuss the existence and uniqueness of a stationary measure and its exponential stability in the Kantorovich-Wasserstein metric. We next turn to the large deviation principle and establish its validity for the occupation measures of the Markov processes in question. The obtained result…
We establish existence of controlled Markov chain of mean-field type with unbounded jump intensities by means of a fixed point argument using the Wasserstein distance. Using a Markov chain entropic backward SDE approach, we further suggest conditions for existence of an optimal control and a saddle-point for respectively a control problem and a zero-sum differential game associated with risk sensitive payoff functionals of mean-field type.
We obtain moment and Gaussian bounds for general Lipschitz functions evaluated along the sample path of a Markov chain. We treat Markov chains on general (possibly unbounded) state spaces via a coupling method. If the first moment of the coupling time exists, then we obtain a variance inequality. If a moment of order 1+epsilon of the coupling time exists, then depending on the behavior of the stationary distribution, we obtain higher moment bounds. This immediately implies polynomial concentration inequalities. In the case that a moment of order 1+epsilon is finite uniformly in the starting po…
In the analysis of Markov chains and processes, it is sometimes convenient to replace an unbounded state space with a "truncated" bounded state space. When such a replacement is made, one often wants to know whether the equilibrium behavior of the truncated chain or process is close to that of the untruncated system. For example, such questions arise naturally when considering numerical methods for computing stationary distributions on unbounded state space. In this paper, we use the principle of "regeneration" to show that the stationary distributions of "fixed state" truncations converge in …
In the analysis of Markov chains and processes, it is sometimes convenient to replace an unbounded state space with a "truncated" bounded state space. When such a replacement is made, one often wants to know whether the equilibrium behavior of the truncated chain or process is close to that of the untruncated system. For example, such questions arise naturally when considering numerical methods for computing stationary distributions on unbounded state space. In this paper, we study general truncation-augmentation schemes, in which the substochastic truncated "northwest corner" of the transitio…
A classical random walk $(S_t, t\in\mathbb{N})$ is defined by $S_t:=\displaystyle\sum_{n=0}^t X_n$, where $(X_n)$ are i.i.d. When the increments $(X_n)_{n\in\mathbb{N}}$ are a one-order Markov chain, a short memory is introduced in the dynamics of $(S_t)$. This so-called "persistent" random walk is nolonger Markovian and, under suitable conditions, the rescaled process converges towards the integrated telegraph noise (ITN) as the time-scale and space-scale parameters tend to zero (see Herrmann and Vallois, 2010; Tapiero-Vallois, Tapiero-Vallois2}). The ITN process is effectively non-Markovian …
We consider the following problem in one-dimensional diffusion-limited aggregation (DLA). At time $t$, we have an "aggregate" consisting of $\Bbb{Z}\cap[0,R(t)]$ [with $R(t)$ a positive integer]. We also have $N(i,t)$ particles at $i$, $i>R(t)$. All these particles perform independent continuous-time symmetric simple random walks until the first time $t'>t$ at which some particle tries to jump from $R(t)+1$ to $R(t)$. The aggregate is then increased to the integers in $[0,R(t')]=[0,R(t)+1]$ [so that $R(t')=R(t)+1$] and all particles which were at $R(t)+1$ at time $t'{-}$ are removed from the s…
First, under a geometric ergodicity assumption, we provide some limit theorems and some probability inequalities for the bifurcating Markov chains (BMC). The BMC model was introduced by Guyon to detect cellular aging from cell lineage, and our aim is thus to complete his asymptotic results. The deviation inequalities are then applied to derive first result on the moderate deviation principle (MDP) for a functional of the BMC with a restricted range of speed, but with a function which can be unbounded. Next, under a uniform geometric ergodicity assumption, we provide deviation inequalities for …
We establish existence of Markov chains of mean-field type with unbounded jump intensities by means of a fixed point argument using the Total Variation distance. We further show existence of nearly-optimal controls and, using a Markov chain backward SDE approach, we suggest conditions for existence of an optimal control and a saddle-point for respectively a control problem and a zero-sum differential game associated with payoff functionals of mean-field type, under dynamics driven by such Markov chains of mean-field type.
Let $(\E,\F)$ be a symmetric non-local Dirichlet from with unbounded coefficient on $L^2(\R^d;\d x)$ defined by $$\E(f,g)=\iint_{\R^d\times \R^d} (f(y)-f(x))(g(x)-g(y)){W(x,y)}\, J(x,\d y)\,\d x, \quad f,g\in \F,$$ where $J(x,\d y)$ is regarded as the jumping kernel for a pure-jump symmetric Lévy-type process with bounded coefficients, and $W(x,y)$ is seen as a weighted (unbounded) function. We establish sharp criteria for compactness and non-compactness of the associated Markovian semigroup $(P_t)_{t\ge0}$ on $L^2(\R^d;\d x)$. In particular, we prove that if $J(x,\d y)=|x-y|^{-d-α}\,\d y$ wit…
We develop a new simulation method for multidimensional diffusions with sticky boundaries. The challenge comes from simulating the sticky boundary behavior, for which standard methods like the Euler scheme fail. We approximate the sticky diffusion process by a multidimensional continuous time Markov chain (CTMC), for which we can simulate easily. We develop two ways of constructing the CTMC: approximating the infinitesimal generator of the sticky diffusion by finite difference using standard coordinate directions, and matching the local moments using the drift and the eigenvectors of the covar…
An unbounded one-dimensional solid-on-solid model with integer heights is studied. Unbounded here means that there is no a priori restrictions on the discret e gradient of the interface. The interaction Hamiltonian of the interface is given by a finite range part, pr oportional to the sum of height differences, plus a part of exponentially decaying long range potentials. The evolution of the interface is a reversible Markov process. We prove that if this system is started in the center of a box of size L after a time of order L^3 it reaches, with a very large probability, the top or the bottom…
ENS Paris-Saclay（巴黎萨克雷高等师范学院）；Ecole Polytechnique（巴黎综合理工学院）；HSE University（俄罗斯国家研究型高等经济大学）；The Chinese University of Hong Kong（香港中文大学）
This paper studies the exponential stability of random matrix products driven by a general (possibly unbounded) state space Markov chain. It is a cornerstone in the analysis of stochastic algorithms in machine learning (e.g. for parameter tracking in online learning or reinforcement learning). The existing results impose strong conditions such as uniform boundedness of the matrix-valued functions and uniform ergodicity of the Markov chains. Our main contribution is an exponential stability result for the $p$-th moment of random matrix product, provided that (i) the underlying Markov chain sati…
