Ronald Ortner
Dealing with unichain MDPs, we consider stationary distributions of policies that coincide in all but $n$ states. In these states each policy chooses one of two possible actions. We show that the stationary distributions of n+1 such policies uniquely determine the stationary distributions of all other such policies. An explicit formula for calculation is given.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
We show that combinations of optimal (stationary) policies in unichain Markov decision processes are optimal. That is, let M be a unichain Markov decision process with state space S, action space A and policies π_j^*: S -> A (1\leq j\leq n) with optimal average infinite horizon reward. Then any combination πof these policies, where for each state i in S there is a j such that π(i)=π_j^*(i), is optimal as well. Furthermore, we prove that any mixture of optimal policies, where at each visit in a state i an arbitrary action π_j^*(i) of an optimal policy is chosen, yields optimal average reward, t…
A Note on the Bias and Kemeny's Constant in Markov Reward Processes with an Application to Markov Chain Perturbation
本文利用平均首次通过时间给出单链马尔可夫奖励过程偏差的显式表达，将平稳分布扰动界推广到不可约性缺失情形并改进1-范数界，同时为Kemeny常数的常数性质提供直观解释。
We consider the higher-order Markov Chain, and characterize the second order Markov chains admitting every probability distribution vector as a stationary vector. The result is used to construct Markov chains of higher-order with the same property. We also study conditions under which the set of stationary vectors of the Markov chain has a certain affine dimension.
We prove that the class of discrete time stationary max-stable process satisfying the Markov property is equal, up to time reversal, to the class of stationary max-autoregressive processes of order $1$. A similar statement is also proved for continuous time processes.
We study a class of Markov processes with finite state space and continuous time that have product form stationary distributions. We obtain a number of examples that can generate conjectures for diffusions with inert drift.
In this paper we consider stationary Markov chains with trivial two-sided tail sigma field, and prove that additive functionals satisfy the central limit theorem provided the variance of partial sums divided by n is bounded.
Let $X$ be an irreducible symmetric Markov process with the strong Feller property. We assume, in addition, that $X$ is explosive and has a tightness property. We then prove the existence and uniqueness of quasi-stationary distributions of $X$.
New algorithms for computing of asymptotic expansions for stationary distributions of nonlinearly perturbed semi-Markov processes are presented. The algorithms are based on special techniques of sequential phase space reduction, which can be applied to processes with asymptotically coupled and uncoupled finite phase spaces.
Computation and applications of limits of certain non-stationary Markov chains
本文提出并数学验证了一种计算受医疗保健逗留时间周期启发的非平稳马尔可夫链极限的方法，展示了其在预测医疗保健与超大型系统维护周期中的应用，以期推动相关研究与教学。
We study the fractal properties of the stationary distrubtion π for a simple Markov process on R. We will give bounds for the Hausdorff dimension of π, and lower bounds for the multifractal spectrum of π. Additionally, we will provide a method for numerically estimating these bounds.
We obtain necessary and sufficient conditions for the regular variation of the variance of partial sums of functionals of discrete and continuous-time stationary Markov processes with normal transition operators. We also construct a class of Metropolis-Hastings algorithms which satisfy a central limit theorem and invariance principle when the variance is not linear in $n$.
In a previous paper we determined one dimensional distributions of a stationary field with linear regressions and quadratic conditional variances under a linear constraint on the coefficients of the quadratic expression. In this paper we show that for stationary Markov chains with linear regressions and quadratic conditional variances the coefficients of the quadratic expression are indeed tied by a linear constraint which can take only one of the two alternative forms.
Let $(ξ_j)_{j\ge1} $, be a non-stationary Markov chain with phase space $X$ and let $\mathfrak{g}_j:\,X\mapsto\mathrm{SL}(m,\mathbb{R})$ be a sequence of functions on $X$ with values in the unimodular group. Set $g_j=\mathfrak{g}_j(ξ_j)$ and denote by $S_n=g_n\ldots g_1$, the product of the matrices $g_j$. We provide sufficient conditions for exponential growth of the norm $\|S_n\|$ when the Markov chain is not supposed to be stationary. This generalizes the classical theorem of Furstenberg on the exponential growth of products of independent identically distributed matrices as well as its ext…
In this paper, we study quasi-stationary distributions of nonlinearly perturbed semi-Markov processes in discrete time. This type of distributions is of interest for the analysis of stochastic systems which have finite lifetimes, but are expected to persist for a long time. We obtain asymptotic power series expansions for quasi-stationary distributions and it is shown how the coefficients in these expansions can be computed from a recursive algorithm. As an illustration of this algorithm, we present a numerical example for a discrete time Markov chain.
An algorithm for estimating quasi-stationary distribution of finite state space Markov chains has been proven in a previous paper. Now this paper proves a similar algorithm that works for general state space Markov chains under very general assumptions.
We present a class of Gauss-Markov processes which can be represented as space-time scaled stationary Ornstein-Uhlenbeck processes defined on the real line. We give several explicit examples of the representation for certain Gauss bridge processes. As an application, we derive a formula for the density function of the supremum location of certain standardized Gauss-Markov processes on compact time intervals. We also present some sufficient conditions under which mean centered Gauss-Markov processes take zero at a fixed time with probability one.
We find the class, ${\cal{C}}_k, k \ge 0$, of all zero mean stationary Gaussian processes, $Y(t), ~t \in \reals$ with $k$ derivatives, for which \begin{equation} Z(t) \equiv (Y^{(0)}(t), Y^{(1)}(t), \ldots, Y^{(k)}(t) ), ~ t \ge 0 \end{equation} \noindent is a $(k+1)$-vector Markov process. (here, $Y^{(0)}(t) = Y(t)$).
This article describes an accurate procedure for computing the mean first passage times of a finite irreducible Markov chain and a Markov renewal process. The method is a refinement to the Kohlas, Zeit fur Oper Res, 30,197-207, (1986) procedure. The technique is numerically stable in that it doesn't involve subtractions. Algebraic expressions for the special cases of one, two, three and four states are derived. A consequence of the procedure is that the stationary distribution of the embedded Markov chain does not need to be derived in advance but can be found accurately from the derived mean …
It has been well known for some time that for strictly stationary Markov chains that are ``reversible'', that special symmetry provides special extra features in the mathematical theory. This paper here is primarily a purely expository review of some of the basic features of that special theory. The mathematical techniques employed in this review are relatively gentle, involving only some basic measure-theoretic probability theory.
We prove a complete class theorem that characterizes \emph{all} stationary time reversible Markov processes whose finite dimensional marginal distributions (of all orders) are infinitely divisible. Aside from two degenerate cases (iid and constant), in both discrete and continuous time every such process with full support is a branching process with Poisson or Negative Binomial marginal univariate distributions and a specific bivariate distribution at pairs of times. As a corollary, we prove that every nondegenerate stationary integer valued processes constructed by the Markov thinning process…
For downward skip-free continuous-time Markov chains on non-negative integers stopped at zero, existence of a quasi-stationary distribution is studied. The scale function for these processes is introduced and the boundary is classified by a certain integrability condition on the scale function, which gives an extension of Feller's classification of the boundary for birth-and-death processes. The existence and the set of quasi-stationary distributions are characterized by the scale function and the new classification of the boundary.
Existence and uniqueness of quasi-stationary and quasi-ergodic measures for absorbing Markov chains: a Banach lattice approach
该研究在弱条件与自然正则性假设下，借助Banach格方法，证明了几乎必然吸收的离散时间马尔可夫链的拟平稳测度和拟遍历测度的存在性与唯一性。
A class of examples is constructed to show that for strictly stationary Markov chains that are reversible, the simultaneous mixing rates for the $ρ$-mixing and strong mixing ($α$-mixing) conditions can be fairly arbitrary, within certain unavoidable tight restrictions. The examples constructed here have the added property that the mixing rate for the absolute regularity ($β$-mixing) condition is within a constant factor of that for strong mixing.
For Markov chains with a partially ordered finite state space we show strong stationary duality under the condition of Möbius monotonicity of the chain. We show relations of Möbius monotonicity to other definitions of monotone chains. We give examples of dual chains in this context which have transitions only upwards. We illustrate general theory by an analysis of nonsymmetric random walks on the cube with an application to networks of queues.
We construct strong stationary dual chains for Ising model on a circle, non-symmetric random walk on square lattice and a random walk on hypercube. The strong stationary dual chains are all sharp and have the same state space as original chains. We use Möbius monotonicity of these chains with respect to natural orderings of the corresponding state spaces. This method provides an easy way to find eigenvalues in the Ising model and for a random walk on hypercube. Keywords: Markov chains; stochastic monotonicity; eigenvalues; Möbius monotonicity; strong stationary duality; strong stationary times…
Asymptotic expansions with explicit upper bounds for remainders are given for stationary distributions of nonlinearly perturbed semi-Markov processes with finite phase spaces. The corresponding algorithms are based on a special technique of sequential phase space reduction, which can be applied to processes with an arbitrary asymptotic communicative structure of phase spaces.
We consider a Markov modulated fluid network with a finite number of stations. We are interested in the tail asymptotics behavior of the stationary distribution of its buffer content process. Using two different approaches, we derive upper and lower bounds for the stationary tail decay rate in various directions. Both approaches are based on a well-known time-evolution formula of a Markov process, so-called Dynkin's formula, where a key ingredient is a suitable choice of test functions. Those results show how multidimensional tail asymptotics can be studied for the more than two-dimensional ca…
By introducing a key combinatorial structure for words produced by a Variable Length Markov Chain (VLMC), the longest internal suffix, precise characterizations of existence and uniqueness of a stationary probability measure for a VLMC chain are given. These characterizations turn into necessary and sufficient conditions for VLMC associated to a subclass of probabilised context trees: the shift-stable context trees. As a by-product, we prove that a VLMC chain whose stabilized context tree is again a context tree has at most one stationary probability measure.
In this paper we give sufficient conditions for the almost sure central limit theorem started at a point, known under the name of quenched central limit theorem. This is achieved by using a new idea of conditioning with respect to both the past and the future of the Markov chain. As applications we provide a new sufficient projective conditions for the quenched CLT.
Stationary Distributions in Monotone Markov Models: Theory and Applications
本文提出单调马尔可夫过程平稳分布存在性的充分必要条件，适用于非紧状态空间和连续时间，并应用于工资动态、贝叶斯学习与收入过程。
Marginal Stationary Distributions and Convergence Rates of Higher Order Markov Chains
研究正则高阶马尔可夫链，解释其极限概率分布为边际平稳分布，证明其性质并建立收敛速率结果，还引入边际混合时间，扩展了一阶马尔可夫链混合时间概念到高阶链。
We consider a random walk on a discrete connected graph having some infinite branches plus finitely many vertices with finite degrees. We find the generator of a strong stationary dual in the sense of Fill, and use it to find some equivalent condition to the existence of a strong stationary time. This strong stationary dual process lies in the set of connected compact sets of the compactification of the graph. When this graph is Z, this is simply the set of (possibly infinite) segments of Z.
New algorithms for construction of asymptotic expansions for stationary distributions of nonlinearly perturbed semi-Markov processes with finite phase spaces are presented. These algorithms are based on a special technique of sequential phase space reduction, which can be applied to processes with an arbitrary asymptotic communicative structure of phase spaces. Asymptotic expansions are given in two forms, without and with explicit bounds for remainders.
Consider a discrete time, ergodic Markov chain with finite state space which is started from stationarity. Fill and Lyzinski (2014) showed that, in some cases, the hitting time for a given state may be represented as a sum of a geometric number of IID random variables. We extend this result by giving explicit bounds on the distance between any such hitting time and an appropriately chosen geometric sum, along with other related approximations. The compounding random variable in our approximating geometric sum is a strong stationary time for the underlying Markov chain; we also discuss the appr…
We consider a Markov chain on $R^+$ with asymptotically zero drift and finite second moments of jumps which is positive recurrent. A power-like asymptotic behaviour of the invariant tail distribution is proven; such a heavy-tailed invariant measure happens even if the jumps of the chain are bounded. Our analysis is based on test functions technique and on construction of a harmonic function.
In this work, we focus on the stationary analysis of a specific class of continuous time Markov-modulated reflected random walks in the quarter plane with applications in the modelling of two-node Markov-modulated queueing networks with coupled queues. The transition rates of the two-dimensional process depend on the state of a finite state Markovian background process. Such a modulation is space homogeneous in the set of inner states of the two-dimensional lattice but may be different in the set of states at its boundaries. To obtain the stationary distribution, we apply the power series appr…
This paper is concerned with the development of rigorous approximations to various expectations associated with Markov chains and processes having non-stationary transition probabilities. Such non-stationary models arise naturally in contexts in which time-of-day effects or seasonality effects need to be incorporated. Our approximations are valid asymptotically in regimes in which the transition probabilities change slowly over time. Specifically, we develop approximations for the expected infinite horizon discounted reward, the expected reward to the hitting time of a set, the expected reward…
We consider two classes of irreducible Markovian arrival processes specified by the matrices $C$ and $D$. The Markov Modulated Poison Process (MMPP) and the Markovian Switched Poison Process (MSPP). The former exhibits a diagonal $D$ while the latter exhibits a diagonal $C$. For these two classes, we consider the following statements: (I) Overdispersion of the counts process. (II) A non-increasing hazard rate of the stationary inter-event time. (III) The squared coefficient of variation of the event stationary process is greater or equal to unity. (IV) A stochastic order showing that the time …
On the Existence and Uniqueness of Stationary Distributions for Some Piecewise Deterministic Markov Processes With State-Dependent Jump Intensity
本文研究一类分段确定性马尔可夫过程，证明其平稳分布与跳跃后位置马尔可夫链的平稳分布一一对应，并据此为随机迭代函数系统情形建立平稳分布存在唯一性判据。
In this paper, we are interested in investigating the perturbation bounds for the stationary distributions for discrete-time or continuous-time Markov chains on a countable state space. For discrete-time Markov chains, two new norm-wise bounds are obtained. The first bound is rather easy to be obtained since the needed condition, equivalent to uniform ergodicity, is imposed on the transition matrix directly. The second bound, which holds for a general (possibly periodic) Markov chain, involves finding a drift function. This drift function is closely related with the mean first hitting times. S…
In an influential paper, Courtois and Semal (1984) establish that when $G$ is an irreducible substochastic matrix for which $\sum_{n=0}^{\infty}G^n <\infty$, then the stationary distribution of any stochastic matrix $P\ge G$ can be expressed as a convex combination of the normalized rows of $(I-G)^{-1} = \sum_{n=0}^{\infty} G^n$. In this note, we give a short proof of this result that extends the theory to the countably infinite and continuous state space settings. This result plays an important role in obtaining error bounds in algorithms involving nearly decomposable Markov chains, and also …
We present a general functional central limit theorem started at a point also known under the name of quenched. As a consequence, we point out several new classes of stationary processes, defined via projection conditions, which satisfy this type of asymptotic result. One of the theorems shows that if a Markov chain is stationary ergodic and reversible, this result holds for bounded additive functionals of the chain which have a martingale coboundary in L_1 representation. Our results are also well adapted for strongly mixing sequences providing for this case an alternative, shorter approach t…
We consider a discrete time semi-Markov process where the characteristics defining the process depend on a small perturbation parameter. It is assumed that the state space consists of one finite communicating class of states and, in addition, one absorbing state. Our main object of interest is the asymptotic behaviour of the joint probabilities of the position of the semi-Markov process and the event of non-absorption as time tends to infinity and the perturbation parameter tends to zero. The main result gives exponential expansions of these probabilities together with an recursive algorithm f…
We consider a Markov process $ X(t) $ on the nonnegative integers $E= S \cup \{0\}$, where $S=\{1,2,...\}$ is an irreducible class and 0 is an absorbing state. In this paper, we investigate conditions under which the quasi-stationary distribution for $X(t)$ exists and is unique, and any initial distribution supported in $S$ is in the domain of attraction of this quasi-stationary distribution. We further find five conditions which are equivalent to that the extinction time is uniformly bounded. As a consequence, we prove the van Doorn's conjecture in \cite{VD2012}. And we can greatly improve th…
We study the long time behaviour of a Markov process evolving in $\mathbb{N}$ and conditioned not to hit 0. Assuming that the process comes back quickly from infinity, we prove that the process admits a unique quasi-stationary distribution (in particular, the distribution of the conditioned process admits a limit when time goes to infinity). Moreover, we prove that the distribution of the process converges exponentially fast in total variation norm to its quasi-stationary distribution and we provide an explicit rate of convergence. As a first application of our result, we bring a new insight o…
Before and beyond the mixing time: New approximations for additive functionals of stationary Gauss-Markov processes
本文研究了平稳高斯-马尔可夫过程加性函数在观测数量与混合时间比例不同时的近似问题，提出了一种新的定量和函数极限定理，揭示了在混合时间附近出现的相变现象及新极限过程的性质。
We study a class of stationary Markov processes with marginal distributions identifiable by moments such that every conditional moment of degree say $m$ is a polynomial of degree at most $m\;\text{.}\;$ We show that then under some additional, natural technical assumption there exists a family of orthogonal polynomial martingales. More precisely we show that such a family of processes is completely characterized by the sequence $\{(α_{n} ,p_{n})\}_{n \geq 0}$ where $α_{n}^{ \prime } s$ are some positive reals while $p_{n}^{ \prime } s$ are some monic orthogonal polynomials. Paper of Bakry\&Maz…
We consider a class of discrete time Markov chains with state space [0,1] and the following dynamics. At each time step, first the direction of the next transition is chosen at random with probability depending on the current location. Then the length of the jump is chosen independently as a random proportion of the distance to the respective end point of the unit interval, the distributions of the proportions being fixed for each of the two directions. Chains of that kind were subjects of a number of studies and are of interest for some applications. Under simple broad conditions, we establis…
This paper considers the subexponential asymptotics of the stationary distributions of GI/G/1-type Markov chains in two cases: (i) the phase transition matrix in non-boundary levels is stochastic; and (ii) it is strictly substochastic. For the case (i), we present a weaker sufficient condition for the sub exponential asymptotics than those given in the literature. As for the case (ii), the subexponential asymptotics has not been studied, as far as we know. We show that the subexponential asymptotics in the case (ii) is different from that in the case (i). We also study the locally subexponenti…
We compute the stationary distribution of a continuous-time Markov chain which is constructed by gluing together two finite, irreducible Markov chains by identifying a pair of states of one chain with a pair of states of the other and keeping all transition rates from either chain (the rates between the two shared states are summed). The result expresses the stationary distribution of the glued chain in terms of quantities of the two original chains. Some of the required terms are nonstandard but can be computed by solving systems of linear equations using the transition rate matrices of the t…
On a boundary of the central limit theorem for strictly stationary, reversible Markov chains
本文证明可逆严格平稳马尔可夫链中心极限定理成立的边界混合速率恰为指数速率，并构造反例表明混合速率任意接近指数时定理可失效。
This paper proposes a new algorithm for computing the stationary distribution vector in continuous-time upper block-Hessenberg Markov chains. To this end, we consider the last-block-column-linearly-augmented (LBCL-augmented) truncation of the (infinitesimal) generator of the upper block-Hessenberg Markov chain. The LBCL-augmented truncation is a linearly-augmented truncation such that the augmentation distribution has its probability mass only on the last block column. We first derive an upper bound for the total variation distance between the respective stationary distribution vectors of the …
We consider a simple but important class of metastable discrete time Markov chains, which we call perturbed Markov chains. Basically, we assume that the transition matrices depend on a parameter $\varepsilon$, and converge as $\varepsilon$. We further assume that the chain is irreducible for $\varepsilon$ but may have several essential communicating classes when $\varepsilon$. This leads to metastable behavior, possibly on multiple time scales. For each of the relevant time scales, we derive two effective chains. The first one describes the (possibly irreversible) metastable dynamics, while th…
This paper studies the light-tailed asymptotics of the stationary tail probability vectors of a Markov chain of M/G/1 type. Almost all related studies have focused on the typical case, where the transition block matrices in the non-boundary levels have a dominant impact on the decay rate of the stationary tail probability vectors and their decay is aperiodic. In this paper, we study not only the typical case but also atypical cases such that the stationary tail probability vectors decay periodically and/or their decay rate is determined by the tail distribution of jump sizes from the boundary …
Maxentropy completion and properties of some partially defined Stationary Markov chains
针对状态空间部分转移概率未知的平稳 Markov 链，在已知输入输出概率及约束条件下，提出满足最大熵原理的转移概率补全方法。
We consider a strictly substochastic matrix or an stochastic matrix with absorbing states. By using quasi-stationary distributions one shows there is a canonical associated stationary Markov chain. Based upon $2-$stringing representation of the resurrected chain we supply a stationary representation of the killed and the absorbed chains. The entropies of these representations has a clear meaning when one identifies the probability measure of natural factors. The balance between the entropies of these representations and the canonical chain, serves to check the suitability of the whole construc…
Asymptotically Optimal Policies for Weakly Coupled Markov Decision Processes
本文研究了通过n个弱耦合马尔可夫决策过程在无限时间范围内最大化期望平均奖励的问题，提出了一种将问题与确定性连续变量控制问题联系起来的方法，并给出了在n趋于无穷时实现最大期望平均奖励的策略构造方法。
Let $(X_n)_{n \in\mathbb{N}}$ be a $V$-geometrically ergodic Markov chain on a measurable space $\mathbb{X}$ with invariant probability distribution $π$. In this paper, we propose a discretization scheme providing a computable sequence $(\widehatπ_k)_{k\ge 1}$ of probability measures which approximates $π$ as $k$ growths to infinity. The probability measure $\widehatπ_k$ is computed from the invariant probability distribution of a finite Markov chain. The convergence rate in total variation of $(\widehatπ_k)_{k\ge 1}$ to $π$ is given. As a result, the specific case of first order autoregressiv…
Quasi-stationary distributions for subcritical branching Markov chains
本文研究了亚临界分支马尔可夫链，基于 spinal decomposition 和 many-to-few formula，通过矩方法给出了 Yaglom 极限存在性的概率证明，并提供了所有拟平稳分布的显式积分表示，无需依赖 Martin 边界理论。
The asymptotic tails of limit distributions of continuous time Markov chains
本文基于通量平衡方程建立平稳测度与QSD恒等式，用三个参数分类幂律转移速率连续时间马尔可夫链的三类尾部渐近，并推广至多种模型及离散时间情形。
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。