We study the limit behaviour of upper and lower bounds on expected time averages in imprecise Markov chains; a generalised type of Markov chain where the local dynamics, traditionally characterised by transition probabilities, are now represented by sets of 'plausible' transition probabilities. Our main result is a set of necessary and sufficient conditions under which these upper and lower bounds, called upper and lower expected time averages, will converge as time progresses towards infinity to limit values that do not depend on the process' initial state. Remarkably, our conditions are considerably weaker than those needed to establish similar results for so-called limit -- or steady state -- upper and lower expectations, which are often used to provide approximate information about the limit behaviour of time averages as well. We show that such an approximation is sub-optimal and that it can be significantly improved by directly using upper and lower expected time averages.
We consider the problem of characterising expected hitting times and hitting probabilities for imprecise Markov chains. To this end, we consider three distinct ways in which imprecise Markov chains have been defined in the literature: as sets of homogeneous Markov chains, as sets of more general stochastic processes, and as game-theoretic probability models. Our first contribution is that all these different types of imprecise Markov chains have the same lower and upper expected hitting times, and similarly the hitting probabilities are the same for these three types. Moreover, we provide a ch…
We justify and discuss expressions for joint lower and upper expectations in imprecise probability trees, in terms of the sub- and supermartingales that can be associated with such trees. These imprecise probability trees can be seen as discrete-time stochastic processes with finite state sets and transition probabilities that are imprecise, in the sense that they are only known to belong to some convex closed set of probability measures. We derive various properties for their joint lower and upper expectations, and in particular a law of iterated expectations. We then focus on the special cas…
The effect of perturbations of parameters for uniquely convergent imprecise Markov chains is studied. We provide the maximal distance between the distributions of original and perturbed chain and maximal degree of imprecision, given the imprecision of the initial distribution. The bounds on the errors and degrees of imprecision are found for the distributions at finite time steps, and for the stationary distributions as well.
We analyse the structure of imprecise Markov chains and study their convergence by means of accessibility relations. We first identify the sets of states, so-called minimal permanent classes, that are the minimal sets capable of containing and preserving the whole probability mass of the chain. These classes generalise the essential classes known from the classical theory. We then define a class of extremal imprecise invariant distributions and show that they are uniquely determined by the values of the upper probability on minimal permanent classes. Moreover, we give conditions for unique con…
We consider the problem of performing inference with imprecise continuous-time hidden Markov chains, that is, imprecise continuous-time Markov chains that are augmented with random output variables whose distribution depends on the hidden state of the chain. The prefix `imprecise' refers to the fact that we do not consider a classical continuous-time Markov chain, but replace it with a robust extension that allows us to represent various types of model uncertainty, using the theory of imprecise probabilities. The inference problem amounts to computing lower expectations of functions on the sta…
We study the limit behaviour of a generally non-linear ordinary differential equation whose solution is a superadditive generalisation of a stochastic matrix, and provide necessary and sufficient conditions for this solution to be ergodic, in the sense that it converges to an operator that, essentially, maps functions to constants. In the linear case, the solution of our differential equation is equal to the matrix exponential of an intensity matrix and can then be interpreted as the transition operator of a homogeneous continuous-time Markov chain. Similarly, in the generalised non-linear cas…
We study the problem of characterizing the expected hitting times for a robust generalization of continuous-time Markov chains. This generalization is based on the theory of imprecise probabilities, and the models with which we work essentially constitute sets of stochastic processes. Their inferences are tight lower- and upper bounds with respect to variation within these sets. We consider three distinct types of these models, corresponding to different levels of generality and structural independence assumptions on the constituent processes. Our main results are twofold; first, we demonstrat…
We present an algorithm that can efficiently compute a broad class of inferences for discrete-time imprecise Markov chains, a generalised type of Markov chains that allows one to take into account partially specified probabilities and other types of model uncertainty. The class of inferences that we consider contains, as special cases, tight lower and upper bounds on expected hitting times, on hitting probabilities and on expectations of functions that are a sum or product of simpler ones. Our algorithm exploits the specific structure that is inherent in all these inferences: they admit a gene…
Imprecise continuous-time Markov chains are a robust type of continuous-time Markov chains that allow for partially specified time-dependent parameters. Computing inferences for them requires the solution of a non-linear differential equation. As there is no general analytical expression for this solution, efficient numerical approximation methods are essential to the applicability of this model. We here improve the uniform approximation method of Krak et al. (2016) in two ways and propose a novel and more efficient adaptive approximation method. For ergodic chains, we also provide a method th…
We present a novel algorithm to solve a non-linear system of equations, whose solution can be interpreted as a tight lower bound on the vector of expected hitting times of a Markov chain whose transition probabilities are only partially specified. We also briefly sketch how this method can be modified to solve a conjugate system of equations that gives rise to the corresponding upper bound. We prove the correctness of our method, and show that it converges to the correct solution in a finite number of steps under mild conditions on the system. We compare the runtime complexity of our method to…
In recent years probabilistic model checking has become an important area of research because of the diffusion of computational systems of stochastic nature. Despite its great success, standard probabilistic model checking suffers the limitation of requiring a sharp specification of the probabilities governing the model behaviour. The theory of imprecise probabilities offers a natural approach to overcome such limitation by a sensitivity analysis with respect to the values of these parameters. However, only extensions based on discrete-time imprecise Markov chains have been considered so far f…
When the initial and transition probabilities of a finite Markov chain in discrete time are not well known, we should perform a sensitivity analysis. This can be done by considering as basic uncertainty models the so-called credal sets that these probabilities are known or believed to belong to, and by allowing the probabilities to vary over such sets. This leads to the definition of an imprecise Markov chain. We show that the time evolution of such a system can be studied very efficiently using so-called lower and upper expectations, which are equivalent mathematical representations of credal…
Continuous-time Markov chains are mathematical models that are used to describe the state-evolution of dynamical systems under stochastic uncertainty, and have found widespread applications in various fields. In order to make these models computationally tractable, they rely on a number of assumptions that may not be realistic for the domain of application; in particular, the ability to provide exact numerical parameter assessments, and the applicability of time-homogeneity and the eponymous Markov property. In this work, we extend these models to imprecise continuous-time Markov chains (ICTMC…
We study the limit behaviour of upper and lower bounds on expected time averages in imprecise Markov chains; a generalised type of Markov chain where the local dynamics, traditionally characterised by transition probabilities, are now represented by sets of `plausible' transition probabilities. Our first main result is a necessary and sufficient condition under which these upper and lower bounds, called upper and lower expected time averages, will converge as time progresses towards infinity to limit values that do not depend on the process' initial state. Our condition is considerably weaker …
We consider the problem of estimating the transition rate matrix of a continuous-time Markov chain from a finite-duration realisation of this process. We approach this problem in an imprecise probabilistic framework, using a set of prior distributions on the unknown transition rate matrix. The resulting estimator is a set of transition rate matrices that, for reasons of conjugacy, is easy to find. To determine the hyperparameters for our set of priors, we reconsider the problem in discrete time, where we can use the well-known Imprecise Dirichlet Model. In particular, we show how the limit of …
Imprecise Transition Matrices for Markov Cohort Models: Lower and Upper Expectations with a Practical Health Economic Application
Medtronic Trading Sàrl（美敦力贸易公司）；Center for Evidence Synthesis in Health, Department of Health Services, Policy, and Practice, Brown University（布朗大学健康服务、政策与实践系健康证据综合中心）
The possibility of flexibly assigning spectrum resources with channels of different sizes greatly improves the spectral efficiency of optical networks, but can also lead to unwanted spectrum fragmentation.We study this problem in a scenario where traffic demands are categorised in two types (low or high bit-rate) by assessing the performance of three allocation policies. Our first contribution consists of exact Markov chain models for these allocation policies, which allow us to numerically compute the relevant performance measures. However, these exact models do not scale to large systems, in…
We present an efficient exact algorithm for estimating state sequences from outputs (or observations) in imprecise hidden Markov models (iHMM), where both the uncertainty linking one state to the next, and that linking a state to its output, are represented using coherent lower previsions. The notion of independence we associate with the credal network representing the iHMM is that of epistemic irrelevance. We consider as best estimates for state sequences the (Walley--Sen) maximal sequences for the posterior joint state model conditioned on the observed output sequence, associated with a gain…
The theory of imprecise Markov chains has achieved significant progress in recent years. Its applicability, however, is still very much limited, due in large part to the lack of efficient computational methods for calculating higher-dimensional models. The high computational complexity shows itself especially in the calculation of the imprecise version of the Kolmogorov backward equation. The equation is represented at every point of an interval in the form of a minimization problem, solvable merely with linear programming techniques. Consequently, finding an exact solution on an entire interv…
We focus on credal nets, which are graphical models that generalise Bayesian nets to imprecise probability. We replace the notion of strong independence commonly used in credal nets with the weaker notion of epistemic irrelevance, which is arguably more suited for a behavioural theory of probability. Focusing on directed trees, we show how to combine the given local uncertainty models in the nodes of the graph into a global model, and we use this to construct and justify an exact message-passing algorithm that computes updated beliefs for a variable in the tree. The algorithm, which is linear …
We investigate how basic probability inequalities can be extended to an imprecise framework, where (precise) probabilities and expectations are replaced by imprecise probabilities and lower/upper previsions. We focus on inequalities giving information on a single bounded random variable $X$, considering either convex/concave functions of $X$ (Jensen's inequalities) or one-sided bounds such as $(X\geq c)$ or $(X\leq c)$ (Markov's and Cantelli's inequalities). As for the consistency of the relevant imprecise uncertainty measures, our analysis considers coherence as well as weaker requirements, n…
If the state space of a homogeneous continuous-time Markov chain is too large, making inferences - here limited to determining marginal or limit expectations - becomes computationally infeasible. Fortunately, the state space of such a chain is usually too detailed for the inferences we are interested in, in the sense that a less detailed - smaller - state space suffices to unambiguously formalise the inference. However, in general this so-called lumped state space inhibits computing exact inferences because the corresponding dynamics are unknown and/or intractable to obtain. We address this is…
We use the functional It{ô} calculus to prove that the solution of a BSDE with singular terminal condition is continuous at the terminal time. Hence we extend known results for a non-Markovian terminal condition.
We consider a Markov chain on $R^+$ with asymptotically zero drift and finite second moments of jumps which is positive recurrent. A power-like asymptotic behaviour of the invariant tail distribution is proven; such a heavy-tailed invariant measure happens even if the jumps of the chain are bounded. Our analysis is based on test functions technique and on construction of a harmonic function.
Nonlinear Markov chains with finite state space have been introduced in Kolokoltsov (2010). The characteristic property of these processes is that the transition probabilities do not only depend on the state, but also on the distribution of the process. In this paper we provide first results regarding their invariant distributions and long-term behaviour. We will show that under a continuity assumption an invariant distribution exists. Moreover, we provide a sufficient criterion for the uniqueness of the invariant distribution that relies on the Brouwer degree. Thereafter, we will present exam…
Limit theorems for strongly and intermediately supercritical branching processes in Markovian environment with linear fractional offspring distributions
We are interested by the rate of growth of increasing positive self-similar Markov processes (ipssMp) such that the subordinator associated to it via Lamperti's transformation has infinite mean. We prove that the logarithm of an ipssMp normalized by the logarithm of the time converges weakly, as the time tends to infinity, if and only if the Laplace exponent of the underlying subordinator is regularly varying at zero. Moreover, we prove that the regular variation at zero of the Laplace exponent is essentially nasc for the existence of a function that normalizes the logarithm of an ipssMp. We o…
This paper studies: (i) the long time behaviour of the empirical distribution of age and normalised position of an age dependent critical branching Markov process conditioned on non-extinction; and (ii) the super-process limit of a sequence of age dependent critical branching Brownian motions.
This paper demonstrates a lower and upper solution method to investigate the asymptotic behaviour of the conservative reaction-diffusion systems associated with Markovian process algebra models. In particular, we have proved the uniform convergence of the solution to its constant equilibrium for a case study as time tends to infinity, together with experimental results illustrations.
We consider a branching system consisting of particles moving according to a Markov family in $\Rd$ and undergoing subcritical branching with a constant rate $V>0$. New particles immigrate to the system according to homogeneous space-time Poisson random field. The process of the fluctuations of the rescaled occupation time is studied with very mild assumptions on the Markov family. In this general setting a functional central limit theorem is proved. The subcriticality of the branching law is crucial for the limit behaviour and in a sense overwhelms the properties of the particles' motion. It …
We present formulas for quasi-ergodic limits of finite absorbing Markov chains. Since the irreducible case has been solved in 1965 by Darroch and Seneta, we focus on the reducible case, and our results are based on a very precise asymptotic analysis of the (exponential and polynomial) growth behaviour along admissible paths
Density dependent Markov population processes with countably many types can often be well approximated over finite time intervals by the solution of the differential equations that describe their average drift, provided that the total population size is large. They also exhibit diffusive stochastic fluctuations on a smaller scale about this deterministic path. Here, it is shown that the individuals in such processes experience an almost deterministic environment. Small groups of individuals behave almost independently of one another, evolving as Markov jump processes, whose transition rates ar…
Let $(X_n)_{n\geq 0}$ be a Markov chain with values in a finite state space $\mathbb X$ starting at $X_0=x \in \mathbb X$ and let $f$ be a real function defined on $\mathbb X$. Set $S_n=\sum_{k=1}^{n} f(X_k)$, $n\geqslant 1$. For any $y \in \mathbb R$ denote by $τ_y$ the first time when $y+S_n$ becomes non-positive. We study the asymptotic behaviour of the probability $\mathbb P_x \left( y+S_{n} \in [z,z+a] \,,\, τ_y > n \right)$ as $n\to+\infty.$ We first establish for this probability a conditional version of the local limit theorem of Stone. Then we find for it an asymptotic equivalent of o…
The influence of a time-periodic forcing on stochastic processes can essentially be emphasized in the large time behaviour of their paths. The statistics of transition in a simple Markov chain model permits to quantify this influence. In particular the first Floquet multiplier of the associated generating function can be explicitly computed and related to the equilibrium probability measure of an associated process in higher dimension. An application to the stochastic resonance is presented.
A nonnegative coarse Ricci curvature for a Markov chain and the existence of an attractive point implies the concentration of the invariant probability measure around this point. The mass outside balls centered at the attractive point, as a function of the radius, decreases at least as fast as the exponential of a double integral of the coarse Ricci curvature. This is exactly the behaviour of the density of the reversible measure for diffusion processes on the real line.
Motivated by the study of a parasite infection in a cell line, we introduce a general class of Markov processes for the modelling of population dynamics. The population process evolves as a diffusion with positive jumps whose rate is a function of the population size. It also undergoes catastrophic events which kill a fraction of the population, at a rate depending on the population state. We study the long time behaviour of this class of processes.
Discrete-time discrete-state random Markov chains with a tridiagonal generator are shown to have a random attractor consisting of singleton subsets, essentially a random path, in the simplex of probability vectors. The proof uses the Hilbert projection metric and the fact that the linear cocycle generated by the Markov chain is a uniformly contractive mapping of the positive cone into itself. The proof does not involve probabilistic properties of the sample path and is thus equally valid in the nonautonomous deterministic context of Markov chains with, say, periodically varying transitions pro…
We prove the existence of a local time, the continuity of the local time about $t$, and the regular property for $a.e.$ $x\in R$ of a Ornstein-Uhlenbeck type $\{X_t,\ t\in R^+\}$ driven by a general Lévy process, under mild regularity conditions. We discuss the asymptotic behaviour of the local time when $X$ is ergodic. We also investigate the first passage problem. These results give precise information about the local properties of the sample functions.
We consider a discrete time semi-Markov process where the characteristics defining the process depend on a small perturbation parameter. It is assumed that the state space consists of one finite communicating class of states and, in addition, one absorbing state. Our main object of interest is the asymptotic behaviour of the joint probabilities of the position of the semi-Markov process and the event of non-absorption as time tends to infinity and the perturbation parameter tends to zero. The main result gives exponential expansions of these probabilities together with an recursive algorithm f…
Continuous-time discrete-state random Markov chains generated by a random linear differential equation with a random tridiagonal matrix are shown to have a random attractor consisting of singleton subsets, essentially a random path, in the simplex of probability vectors. The proof uses comparison theorems for Carathéodory random differential equations and the fact that the linear cocycle generated by the Markov chain is a uniformly contractive mapping of the positive cone into itself with respect to the the Hilbert projective metric. It does not involve probabilistic properties of the sample p…
We study the exponential growth of bifurcating processes with ancestral dependence. We suppose here that the lifetimes of the cells are dependent random variables, that the numbers of new cells are random and dependent. Lifetimes and new cells's numbers are also assumed to be dependent. We illustrate our results by examples, including some Markov models. Our approach is related to the behaviour of the Laplace transform of nonnegative additive functional of Markov chains and require weak moment assumption (no exponential moment is needed).
We study Markov chains on $\mathbb Z^m$, $m\geq 2$, that behave like a standard symmetric random walk outside of the hyperplane (membrane) $H=\{0\}\times \mathbb Z^{m-1}$. The transition probabilities on the membrane $H$ are periodic and also depend on the incoming direction to $H$, what makes the membrane $H$ two-sided. Moreover, sliding along the membrane is allowed. We show that the natural scaling limit of such Markov chains is a $m$-dimensional diffusion whose first coordinate is a skew Brownian motion and the other $m-1$ coordinates is a Brownian motion with a singular drift controlled b…
In this paper we study the long term evolution of a continuous time Markov chain formed by two interacting birth-and-death processes. The interaction between the processes is modelled by transition rates which are functions with suitable monotonicity properties. This is in line with the approach proposed by Gauss G.F. and Kolmogorov A.N. for modelling interaction between species in ecology. We obtain conditions for transience/recurrence of the Markov chain and describe in detail its asymptotic behaviour in special transient cases. In particular, we find that in some of these cases the Markov c…
The extremal behaviour of a Markov chain is typically characterized by its tail chain. For asymptotically dependent Markov chains existing formulations fail to capture the full evolution of the extreme event when the chain moves out of the extreme tail region and for asymptotically independent chains recent results fail to cover well-known asymptotically independent processes such as Markov processes with a Gaussian copula between consecutive values. We use more sophisticated limiting mechanisms that cover a broader class of asymptotically independent processes than current methods, including …
We study the asymptotic behaviour of Markov processes on large weighted Erdos-Renyi graphs where the transition rates of the vertices are only influenced by the state of their neighbours and the corresponding weight on the edges. We find the ratio of vertices being in a certain state will converge to the solution of a differential equation obtained from mean field approximation if the graph is dense enough, namely, the average degree is at least of order $N^{\frac{1}{2}+ε}$. Proof for convergence in probability in the transient regime is shown.
Consider the real Markov walk $S_n = X_1+ \dots+ X_n$ with increments $\left(X_n\right)_{n\geq 1}$ defined by a stochastic recursion starting at $X_0=x$. For a starting point $y>0$ denote by $τ_y$ the exit time of the process $\left( y+S_n \right)_{n\geq 1}$ from the positive part of the real line. We investigate the asymptotic behaviour of the probability of the event $τ_y \geq n$ and of the conditional law of $y+S_n$ given $τ_y \geq n$ as $n \to +\infty$.
In this note we consider a Markov chain formed by a finite system of interacting birth-and-death processes on a finite state space. We study an asymptotic behaviour of the Markov chain as its state space becomes large. In particular, we show that the appropriately scaled Markov chain converges to a diffusion process, and derive conditions for existence of the stationary distribution of the limit diffusion process in special cases.
In this paper we characterize the distribution of the first exit time from an arbitrary open set for a class of semi-Markov processes obtained as time-changed Markov processes. We estimate the asymptotic behaviour of the survival function (for large $t$) and of the distribution function (for small $t$) and we provide some conditions for absolute continuity. We have been inspired by a problem of neurophyshiology and our results are particularly usefull in this field, precisely for the so-called Leacky Integrate-and-Fire (LIF) models: the use of semi-Markov processes in these models appear to be…
