A note on compact Markov operators

A Short Note on Stationary Distributions of Unichain Markov Decision Processes

Dealing with unichain MDPs, we consider stationary distributions of policies that coincide in all but $n$ states. In these states each policy chooses one of two possible actions. We show that the stationary distributions of n+1 such policies uniquely determine the stationary distributions of all other such policies. An explicit formula for calculation is given.

关于马尔可夫链状态占用分布的注记

A note on the state occupancy distribution for Markov chains

本文针对可数状态马尔可夫链，利用生成函数方法推导占用时间分布，给出了比现有方法更简洁的公式。

关于两状态马尔可夫链中精确状态访问概率的注记

A Note on Exact State Visit Probabilities in Two-State Markov Chains

本文推导了两状态马尔可夫链中特定状态在N次转移后恰好被访问k次的精确闭式概率，修正并扩展了先前不完整结果，并通过数值模拟验证了其在随机建模中的适用性。

A note on the passage time of finite state Markov chains

Consider a Markov chain with finite state $\{0, 1, ..., d\}$. We give the generation functions (or Laplace transforms) of absorbing (passage) time in the following two situations : (1) the absorbing time of state $d$ when the chain starts from any state $i$ and absorbing at state $d$; (2) the passage time of any state $i$ when the chain starts from the stationary distribution supposed the chain is time reversible and ergodic. Example shows that it is more convenient compared with the existing methods, especially we can calculate the expectation of the absorbing time directly.

A note on scaling limits for truncated birth-and-death processes with interaction

In this note we consider a Markov chain formed by a finite system of interacting birth-and-death processes on a finite state space. We study an asymptotic behaviour of the Markov chain as its state space becomes large. In particular, we show that the appropriately scaled Markov chain converges to a diffusion process, and derive conditions for existence of the stationary distribution of the limit diffusion process in special cases.

A note on the Esscher transform of affine Markov processes

In affine models, both the martingale property of stochastic exponentials and non-explosion of affine processes is characterized in terms of minimality of solutions to a system of generalized Riccati differential equations. This is the final improvement of previous results by Duffie, Filipovic and Schachermayer (2003), Mayerhofer, Muhle-Karbe and Smirnov (2011) and Keller-Ressel and Mayerhofer (2015) for processes on canonical state spaces.

A Short Note on Markov Duality in Multi-species Higher Spin Stochastic Vertex Models

We show that the multi-species higher spin stochastic vertex model, also called the U_q(A_n^{(1)}) vertex model, satisfies a duality where the indicator function has the form {η^x_{[1,n]} \geq ξ^x_{[1,n]} }. In other words, for every particle in the ξconfiguration of species i at vertex x, there must be a particle of species j > i at vertex x in the ηconfiguration. The proof follows by applying charge reversal to previously discovered duality functions, which also results in open boundary conditions. As a corollary, we recover the duality for the stochastic six vertex model recently found by Y…

Compactness of semigroups of explosive symmetric Markov processes

In this paper, we investigate spectral properties of explosive symmetric Markov processes. Under a condition on its life time, we prove the $L^1$-semigroup of Markov processes become compact operators.

Notes on Markov embedding

The representation problem of finite-dimensional Markov matrices in Markov semigroups is revisited, with emphasis on concrete criteria for matrix subclasses of theoretical or practical relevance, such as equal-input, circulant, symmetric or doubly stochastic matrices. Here, we pay special attention to various algebraic properties of the embedding problem, and discuss the connection with the centraliser of a Markov matrix.

Note on local mixing techniques for stochastic differential equations

This paper discusses several techniques which may be used for applying the coupling method to solutions of stochastic differential equations (SDEs). They all work in dimension $d\ge 1$, although, in $d=1$ the most natural way is to use intersections of trajectories, which requires nothing but strong Markov property and non-degeneracy of the diffusion coefficient. In dimensions $d>1$ it is possible to use embedded Markov chains either by considering discrete times $n=0,1,\ldots$, or by arranging special stopping time sequences and to use local Markov -- Dobrushin's (MD) condition. Further appli…

A note on chaotic and predictable representations for Itô-Markov additive processes

IIn this paper we provide predictable and chaotic representations for Itô-Markov additive processes $X$. Such a process is governed by a finite-state CTMC $J$ which allows one to modify the parameters of the Itô-jump process (in so-called regime switching manner). In addition, the transition of $J$ triggers the jump of $X$ distributed depending on the states of $J$ just prior to the transition. This family of processes includes Markov modulated Itô-Lévy processes and Markov additive processes. The derived chaotic representation of a square-integrable random variable is given as a sum of stocha…

关于单调马尔可夫链稳定性的注记

A Note on the Stability of Monotone Markov Chains

本文针对单调马尔可夫链，提出简单适用的条件以确保其全局稳定性，并通过自回归过程、投资组合配置和资源分配等例子验证了条件的简便性。

关于马尔可夫奖励过程中偏差与Kemeny常数的注记及其在马尔可夫链扰动中的应用

A Note on the Bias and Kemeny's Constant in Markov Reward Processes with an Application to Markov Chain Perturbation

本文利用平均首次通过时间给出单链马尔可夫奖励过程偏差的显式表达，将平稳分布扰动界推广到不可约性缺失情形并改进1-范数界，同时为Kemeny常数的常数性质提供直观解释。

A note on time-dependent additive functionals

This note develops shortly the theory of time-inhomogeneous additive functionals and is a useful support for the analysis of time-dependent Markov processes and related topics. It is a significant tool for the analysis of BSDEs in law. In particular we extend to a non-homogeneous setup some results concerning the quadratic variation and the angular bracket of Martin-gale Additive Functionals (in short MAF) associated to a homogeneous Markov processes.

The Optimal Filtering of Markov Jump Processes in Additive White Noise

This note is based on Wonham \cite{Wonham}. The differences between this note and [Wonham] are discussed in Section VIII.

Lecture Notes on Stochastic Processes

This is lecture notes on the course "Stochastic Processes". In this format, the course was taught in the spring semesters 2017 and 2018 for third-year bachelor students of the Department of Control and Applied Mathematics, School of Applied Mathematics and Informatics at Moscow Institute of Physics and Technology. The base of this course was formed and taught for decades by professors from the Department of Mathematical Foundations of Control A.A. Natan, S.A. Guz, and O.G. Gorbachev. Besides standard chapters of stochastic processes theory (correlation theory, Markov processes) in this book (a…

Ergodic Markov processes and Poisson equations (lecture notes)

These are lecture notes on the subject defined in the title. As such, they do not pretend to be really new, probably except for the only section about Poisson equations with potentials. Yet, the hope of the author is that they may serve as a bridge to the important area of Poisson equations "in the whole space" and with a parameter, the latter theme not being presented here. Why this area is so important was explained in many papers and books (see the references [12, 34, 35]): it provides one of the main tools in diffusion approximation in the area stochastic averaging. Hence, the aim of these…

Markov loops, determinants and Gaussian fields

The purpose of this note is to explore some simple relations between loop measures, determinants, and Gaussian Markov fields.

A Note on Subexponential Rate of Convergence to Equilibrium for Processes on the Half-Line

A powerful tool for studying long-term convergence of a Markov process to its stationary distribution is a Lyapunov function. In some sense, this is a substitute for eigenfunctions. For a stochastically ordered Markov process on the half-line, Lyapunov functions can be used to easily find explicit rates of convergence. Our earlier research focused on exponential rate of convergence. This note extends these results to slower rates, including power rates, thus improving results of (Douc, Fort, Guillin, 2009).

Poincaré type inequalities for compact degenerate pure jump Markov processes

We aim in proving Poincaré inequalities for a class of pure jump Markov processes inspired by the model introduced in \cite{G-L} by Galves and Löcherbach to describe the behaviour of interacting brain neurons. In particular, we consider neurons with degenerate jumps, i.e. that lose their memory when they spike, while the probability of a spike depends on the actual position and thus the past of the whole neural system.

关于吸收马尔可夫决策过程占用测度弱紧性的注记

A note on weak compactness of occupation measures for an absorbing Markov decision process

本文研究了具有Borel状态和动作空间的吸收马尔可夫决策过程，给出了使其一致吸收且占用测度集在弱拓扑下紧致的条件，推广了已有结果并回答了相关文献中的猜想。

A Simple Proof of the Monotonicity of the Invariant Distribution for a Discrete Markov Chain

This note presents a simple proof of the monotonicity of the invariant distribution of a discrete Markov chain with a finite state space. This answers a question recently raised by David Siegmund.

关于高斯随机场与通过中心极限定理的底层马尔可夫过程的注记

Note On Gaussian Random Fields \& Underlying Markov Processes Through a Central Limit Theorem

本文通过中心极限定理引入一类与遍历马尔可夫过程相关的通用高斯随机场，并建立与Dynkin、Diaconis和Evans意义下高斯随机场的联系。

Bounds on the rate on convergence for Markovian queueing models with catastrophes

In this note, a general approach to the study of non-stationary Markov chains with catastrophes and the corresponding queuing models is considered, as well as to obtain estimates of the limiting regime itself. As an illustration, an example of a queuing model is studied.

Explicit formula for the density of local times of Markov Jump Processes

In this note we show a simple formula for the joint density of local times, last exit tree and cycling numbers of continuous-time Markov Chains on finite graphs, which involves the modified Bessel function of the first type.

A note on faithful coupling of Markov chains

One often needs to turn a coupling $(X_i, Y_i)_{i\geq 0}$ of a Markov chain into a sticky coupling where once $X_T = Y_T$ at some $T$, then from then on, at each subsequent time step $T'\geq T$, we shall have $X_{T'} = Y_{T'}$. However, not all of what are considered couplings in literature, even Markovian couplings, can be turned into sticky couplings, as proved by Rosenthal through a counter example. Rosenthal then proposed a strengthening of the Markovian coupling notion, termed as faithful coupling, from which a sticky coupling can indeed be obtained. We identify the reason why a sticky co…

Pseudo Differential Operators and Markov Semigroups on Compact Lie Groups

We extend the Ruzhansky-Turunen theory of pseudo differential operators on compact Lie groups into a tool that can be used to investigate group-valued Markov processes in the spirit of the work in Euclidean spaces of N.Jacob and collaborators. Feller semigroups, their generators and resolvents are exhibited as pseudo-differential operators and the symbols of the operators forming the semigroup are expressed in terms of the Fourier transform of the transition kernel. The symbols are explicitly computed for some examples including the Feller processes associated to stochastic flows arising from …

Markov loops in discrete spaces

The main topic of these notes are Markov loops, studied in the context of continuous time Markov chains on discrete state spaces. We refer to [1] and [2] for the short "history" of the subject. In contrast with these references, symmetry is not assumed, and more attention is given to the infinite case. All results are presented in terms of the semigroup generator. In comparison with [1], some delicate proofs are given in more details or with a better method. We focus mostly on properties of the (multi)occupation field but also included some results about loop clusters (see [3] in the symmetric…

On Mixing Properties of Reversible Markov Chains

It is well known that for a strictly stationary, reversible, Harris recurrent Markov chain, the $ρ$-mixing condition is equivalent to geometric ergodicity and to a "spectral gap" condition. In this note, it will be shown with an example that for that class of Markov chains, the "interlaced" variant of the $ρ$-mixing condition fails to be equivalent to those conditions.

Some simple but challenging Markov processes

In this note, we present few examples of Piecewise Deterministic Markov Processes and their long time behavior. They share two important features: they are related to concrete models (in biology, networks, chemistry,. . .) and they are mathematically rich. Their math-ematical study relies on coupling method, spectral decomposition, PDE technics, functional inequalities. We also relate these simple examples to recent and open problems.

关于马尔可夫链重叠批均值方差估计量的集中不等式的一个注记

A note on concentration inequalities for the overlapped batch mean variance estimators for Markov chains

本文借助鞅分解方法，为一致几何遍历马尔可夫链渐近方差的 OBM 估计量建立集中不等式，给出误差 p 阶矩随 p 与混合时间变化的显式界。

Topics in Markov chains: mixing and escape rate

These are the notes for the minicourse on Markov chains delivered at the Saint Petersburg Summer School, June 2012. The main emphasis is on methods for estimating mixing times (for finite chains) and escape rates (for infinite chains). Lamplighter groups are key examples in both topics and the Varopolous-Carne long range estimate is useful in both settings.

A note on random orthogonal polynomials on a compact interval

We consider a uniform distribution on the set $\mathcal{M}_k$ of moments of order $k \in \mathbb{N}$ corresponding to probability measures on the interval $[0,1]$. To each (random) vector of moments in $\mathcal{M}_{2n-1}$ we consider the corresponding uniquely determined monic (random) orthogonal polynomial of degree $n$ and study the asymptotic properties of its roots if $n \to \infty$.

Arrêt optimal pour les processus de Markov forts et les fonctions affines

In this Note we study optimal stopping problems for strong Markov processes and affine functions. We give a justification of the Snell envelope form using standard results of optimal stopping. We also justify the convexity of the value function, and without a priori restriction to a particular class of stopping times, we deduce that the smallest optimal stopping time is necessarily a hitting time.

A martingale minimax exponential inequality for Markov chains

We prove a new inequality controlling the large deviations of the empirical measure of a Markov chain. This inequality is based on the martingale used by Donsker and Varadhan and the minimax theorem. It holds for convex sets and it requires to take an infimum over the starting point. In the case of a compact space, this inequality is a partial improvement of the large deviations estimates of Donsker and Varadhan. In the case of a non compact space, we condition on the event that the process visits $n$ times a compact subset of the space and we still obtain a control on the exponential scale.

A Note on Feller Semigroups and Resolvents

Various equivalent conditions for a semigroup or a resolvent generated by a Markov process to be of Feller type are given.

$L^2$-Quasi-compact and hyperbounded Markov operators

A Markov operator $P$ on a probability space $(S,Σ,μ)$, with $μ$ invariant, is called {\it hyperbounded} if for some $1 \le p<q \le \infty$ it maps (continuously) $L^p$ into $L^q$. We deduce from a recent result of Glück that a hyperbounded $P$ is quasi-compact, hence uniformly ergodic, in all $L^r(S,μ)$, $1<r< \infty$. We prove, using a method similar to Foguel's, that a hyperbounded Markov operator has periodic behavior similar to that of Harris recurrent operators, and for the ergodic case obtain conditions for aperiodicity. Given a probability $ν$ on the unit circle, we prove that if the c…

Long time behavior of Markov processes and beyond

This note provides several recent progresses in the study of long time behavior of Markov processes. The examples presented below are related to other scientific fields as PDE's, physics or biology. The involved mathematical tools as propagation of chaos, coupling, functional inequalities, provide a good picture of the classical methods that furnish quantitative rates of convergence to equilibrium.

Processus aléatoires et applications

Lecture notes for a master-level applied mathematics course on stochastic processes and applications, held at the University of Orléans, France. Contents: Markov chains, Poisson point processes, Markovian jump processes, queueing theory. ----- Notes d'un cours de Master 2 professionnel en mathématiques appliquées, donné à l'Université d'Orléans. Contenu: Chaînes de Markov, processus ponctuel de Poisson, files d'attente.

关于度量空间上概率测度的紧与σ-紧子集的注记，并应用于无分布报童问题

A note on compact and σ-compact subsets of probability measures on metric spaces with an application to the distribution free newsvendor problem

本文研究度量空间上概率测度在弱收敛拓扑下的紧与σ-紧子集，给出σ-紧度量空间的概率测度空间未必σ-紧的反例，并将结果用于扩展的无分布报童问题。

A Short Review of Ergodicity and Convergence of Markov chain Monte Carlo Estimators

This short note reviews the basic theory for quantifying both the asymptotic and preasymptotic convergence of Markov chain Monte Carlo estimators.

Asymptotic Shape of Quantum Markov Semigroups for Compact Uniform Trees

We give locally finite Markov trees in $L^p$-compact$,$ separable Hilbert$,$ supersymmetric process$:$ $[0,\infty)\!\times\!\mathbb{R}^{\lvert\mathcal{A}^{\otimes m}\rvert}/\mathcal{A}^{\otimes m}$ on quantum ${\rm U}(\lvert\mathcal{A}^{\otimes m}\rvert)$ semigroups$.$ In full automorphism group ${\rm Aut}({\rm\bf T})$ of modular subgroup$,$ asymptotic-ergodicity is entropy-worthy $\mathbb{R}$ shape for uniform partition$.$

Condenser physics applied to Markov chains - A brief introduction to potential theory

These notes constitute the introduction to potential theory I exposed at the XIIth brazilian school of probability inside Elisabetta Scoppola's Introduction to Metastability.

Symmetries of Lévy processes on compact quantum groups, their Markov semigroups and potential theory

Strongly continuous semigroups of unital completely positive maps (i.e. quantum Markov semigroups or quantum dynamical semigroups) on compact quantum groups are studied. We show that quantum Markov semigroups on the universal or reduced C${}^*$-algebra of a compact quantum group which are translation invariant (w.r.t. to the coproduct) are in one-to-one correspondence with Lévy processes on its $*$-Hopf algebra. We use the theory of Lévy processes on involutive bialgebras to characterize symmetry properties of the associated quantum Markov semigroup. It turns out that the quantum Markov semigr…

Distribution of a Markov chain in reverse-time with cluster observations in the extremes of a finite time window

In this brief note, we find formulas for the distribution and the transition probability matrices of a stochastic process described as a time-reversion in a finite time window of a Markov chain, with cluster observation of the Markov state in the extremes of that window.

带洞拓扑马尔可夫位移的转移算子的拟紧性

Quasi-compactness of transfer operators for topological Markov shifts with holes

本文研究了带2-柱体洞的可数状态拓扑马尔可夫位移的转移算子，在有限不可约转移矩阵和较弱Lipschitz可求和势条件下证明了Ruelle-Perron-Frobenius定理与拟紧性，并计算了逃逸率，还给出了谱隙性质及图迭代函数系统中的应用。

A note on Gromov-Hausdorff-Prokhorov distance between (locally) compact measure spaces

We present an extension of the Gromov-Hausdorff metric on the set of compact metric spaces: the Gromov-Hausdorff-Prokhorov metric on the set of compact metric spaces endowed with a finite measure. We then extend it to the non-compact case by describing a metric on the set of rooted complete locally compact length spaces endowed with a locally finite measure. We prove that this space with the extended Gromov-Hausdorff-Prokhorov metric is a Polish space. This generalization is needed to define Lévy trees, which are (possibly unbounded) random real trees endowed with a locally finite measure.

Markov semigroups on unitary duals generated by quantized characters

In this paper, we study Markov dynamics on unitary duals of compact quantum groups. We construct such dynamics from characters of quantum groups. Then we show that the dynamics have generators, and we give an explicit formula of the generators using the representation theory. Moreover, we construct Markov dynamics on the unitary dual of an inductive limit of compact quantum groups.

Large deviation principles and fluctuation theorems for currents in semi-Markov processes

In this short note we consider semi-Markov processes satisfying the condition of direction-time independence (Markov renewal processes). We derive large deviation principles and fluctuation theorems for the empirical current and the empirical currents along cycles. Our derivation is based on the joint LDP for the empirical measure and flow recently proved in \cite{MZ}.

Markov dynamics on the dual object to the infinite-dimensional unitary group

These are notes for a mini-course of 3 lectures given at the St. Petersburg School in Probability and Statistical Physics (June 2012). My aim was to explain, on the example of a particular model, how ideas from the representation theory of big groups can be applied in probabilistic problems. The material is based on the joint paper arXiv:1009.2029 by Alexei Borodin and myself; a broader range of topics is surveyed in the lecture notes by Alexei Borodin and Vadim Gorin arXiv:1212.3351.

Tail spaces estimates on Hamming cube and Bernstein--Markov inequality

This note contains some estimates for tail spaces on Hamming cube. We use analytic paraproduct operator for that purpose. We also show several types of Bernstein--Markov inequalities for Banach space valued functions on Hamming cube. Here the novelty is in getting rid of some irritating logarithms and in proving Bernstein--Markov inequalities for $|\nabla f|_X$ rather than for $Δ^{1/2} f$ for $X$ valued polynomials on Hamming cube.

交替马尔可夫链的几何与对偶性

Geometry and Duality of Alternating Markov Chains

本文将一般马尔可夫链的半步实现为联合概率分布凸集间关于逆向Kullback-Leibler散度的交替投影，并以此几何刻画证明了奇偶半步定义的马尔可夫链间的信息论对偶性。