For discrete-time stochastic processes, there is a close connection between return/waiting times and entropy. Such a connection cannot be straightforwardly extended to the continuous-time setting. Contrarily to the discrete-time case one does need a reference measure and so the natural object is relative entropy rather than entropy. In this paper we elaborate on this in the case of continuous-time Markov processes with finite state space. A reference measure of special interest is the one associated to the time-reversed process. In that case relative entropy is interpreted as the entropy production rate. The main results of this paper are: almost-sure convergence to relative entropy of suitable waiting-times and their fluctuation properties (central limit theorem and large deviation principle).
Suppose that $\mathcal C$ is a finite collection of patterns. Observe a Markov chain until one of the patterns in $\mathcal C$ occurs as a run. This time is denoted by $τ$. In this paper, we aim to give an easy way to calculate the mean waiting time $E(τ)$ and the stopping probabilities $P(τ=τ_A)$ with $A\in\mathcal C$, where $τ_A$ is the waiting time until the pattern $A$ appears as a run.
It is our intention to provide via fractional calculus a generalization of the pure and compound Poisson processes, which are known to play a fundamental role in renewal theory, without and with reward, respectively. We first recall the basic renewal theory including its fundamental concepts like waiting time between events, the survival probability, the counting function. If the waiting time is exponentially distributed we have a Poisson process, which is Markovian. However, other waiting time distributions are also relevant in applications, in particular such ones with a fat tail caused by a…
Consider a Markov chain $\{X_n\}_{n\ge 0}$ with an ergodic probability measure $π$. Let $Ψ$ a function on the state space of the chain, with $α$-tails with respect to $π$, $α\in (0,2)$. We find sufficient conditions on the probability transition to prove convergence in law of $N^{1/α}\sum_n^N Ψ(X_n)$ to a $α$-stable law. ``Martingale approximation'' approach and ``coupling'' approach give two different sets of conditions. We extend these results to continuous time Markov jump processes $X_t$, whose skeleton chain satisfies our assumptions. If waiting time between jumps has finite expectation, …
Flowgraph models provide an alternative approach in modeling a multi-state stochastic process. One of the most widely used stochastic processes that have many real-world applications especially in actuarial models is the Markov jump process or continuous- time Markov chain. However, finding waiting time distributions between any two states in a Markov jump process can be very difficult. Flowgraph analysis for Markov jump process comprises of modeling the possible states of the process, the interstates waiting time distribution, and working on the moment generating function domain to obtain the…
In a recent work we introduced a semi-Markovian discrete-time generalization of the telegraph process. We referred this random walk to as squirrel random walk (SRW). The SRW is a discrete-time random walk on the one-dimensional infinite lattice where the step direction is reversed at arrival times of a discrete-time renewal process and remains unchanged at uneventful time instants. We first recall general notions of the SRW. The main subject of the paper is the study of the SRW where the step direction switches at the arrival times of a generalization of the Sibuya discrete-time renewal proces…
An invariant state of a quantum Markov semigroup is an equilibrium state if it satisfies a quantum detailed balance condition. In this paper, we introduce a notion of entropy production for faithful normal invariant states of a quantum Markov semigroup on B(h) as a numerical index measuring "how much far" they are from equilibrium. The entropy production is defined as the derivative of the relative entropy of the one-step forward and backward evolution in analogy with the classical probabilistic concept. We prove an explicit trace formula expressing the entropy production in terms of the compl…
Let $V(k)$ denote the waiting time, the number of trials needed to get a consecutive $k$ ones. We propose recurrence algorithms for the probability distribution function (pdf) and the probability generating function (pgf) of $V(k)$ in sequences of independent and Markov dependent Bernoulli trials using generalized Fibonacci sequences of order $k$. Maximum likelihood estimation (MLE) methods for the probability distributions are presented in both cases with simulation examples.
In the standard models for optimal multiple stopping problems it is assumed that between two exercises there is always a time period of deterministic length $δ$, the so called refraction period. This prevents the optimal exercise times from bunching up together on top of the optimal stopping time for the one-exercise case. In this article we generalize the standard model by considering random refraction times. We develop the theory and reduce the problem to a sequence of ordinary stopping problems thus extending the results for deterministic times. This requires an extension of the underlying …
We consider a class of discrete-time random walks with directed unit steps on the integer line. The direction of the steps is reversed at the time instants of events in a discrete-time renewal process and is maintained at uneventful time instants. This model represents a discrete-time semi-Markovian generalization of the telegraph process. We derive exact formulae for the propagator using generating functions. We prove that for geometrically distributed waiting times in the diffusive limit, this walk converges to the classical telegraph process. We consider the large-time asymptotics of the ex…
For any stationary $\mZ^d$-Gibbs measure that satisfies strong spatial mixing, we obtain sequences of upper and lower approximations that converge to its entropy. In the case, $d=2$, these approximations are efficient in the sense that the approximations are accurate to within $ε$ and can be computed in time polynomial in $1/ε$.
Continuous time random walks have random waiting times between particle jumps. We define the correlated continuous time random walks (CTRWs) that converge to fractional Pearson diffusions (fPDs). The jumps in these CTRWs are obtained from Markov chains through the Bernoulli urn-scheme model and Wright-Fisher model. The jumps are correlated so that the limiting processes are not Lévy but diffusion processes with non-independent increments. The waiting times are selected from the domain of attraction of a stable law
Let $(ξ_n)_{n=0}^\infty$ be a nonhomogeneous Markov chain taking values from finite state-space of $\mathbf{X}=\{1,2,\ldots,b\}$. In this paper, we will study the generalized entropy ergodic theorem with almost-everywhere and $\mathcal{L}_1$ convergence for nonhomogeneous Markov chains, which generalizes the corresponding classical results for the Markov chains.
The correspondence between the cross-entropy method and the zero-variance approximation to simulate a rare event problem in Markov chains is shown. This leads to a sufficient condition that the cross-entropy estimator is asymptotically optimal.
We consider Hidden Markov Chains obtained by passing a Markov Chain with rare transitions through a noisy memoryless channel. We obtain asymptotic estimates for the entropy of the resulting Hidden Markov Chain as the transition rate is reduced to zero.
Continuous time random walks impose a random waiting time before each particle jump. Scaling limits of heavy tailed continuous time random walks are governed by fractional evolution equations. Space-fractional derivatives describe heavy tailed jumps, and the time-fractional version codes heavy tailed waiting times. This paper develops scaling limits and governing equations in the case of correlated jumps. For long-range dependent jumps, this leads to fractional Brownian motion or linear fractional stable motion, with the time parameter replaced by an inverse stable subordinator in the case of …
We prove that under certain mild assumptions, the entropy rate of a hidden Markov chain, observed when passing a finite-state stationary Markov chain through a discrete-time continuous-output channel, is jointly analytic as a function of the input Markov chain parameters and the channel parameters. In particular, as consequences of the main theorems, we obtain analyticity for the entropy rate associated with representative channels: Cauchy and Gaussian.
Waiting time analysis in a finite-capacity multi-server systems with dynamic priorities, dynamically evolving customer types, and abandonment
School of Natural Sciences, Discipline of Mathematics, University of Tasmania（塔斯马尼亚大学自然科学学院数学学科）；School of Medicine, University of Tasmania（塔斯马尼亚大学医学院）
In this paper we calculate Kolmogorov-Sinai entropy $h_M(S)$ of the generalized Markov shift associated with a contractive Markov system (CMS) \cite{Wer1} using the coding map constructed in \cite{Wer3}. We show that \[h_M(S)=-\sum\limits_{e\in E}\int\limits_{K_{i(e)}} p_e\log p_edμ\] where $μ$ is a unique invariant Borel probability measure of the CMS. I. Werner, Contractive Markov systems, J. London Math. Soc. (2005) 236-258. I. Werner, Coding map for a contractive Markov system, Math. Proc. Camb. Phil. Soc. to appear 140 (2), March 2006.
We show that if the random walk on a graph has positive coarse Ricci curvature in the sense of Ollivier, then the stationary measure satisfies a W^1 transport-entropy inequality. Peres and Tetali have conjectured a stronger consequence, that a modified log-Sobolev inequality (MLSI) should hold, in analogy with the setting of Markov diffusions. We discuss how our entropy interpolation approach suggests a natural attack on the MLSI conjecture.
In the absence of acceleration, the velocity formula gives "distance travelled equals speed multiplied by time". For a broad class of Markov chains such as circulant Markov chains or random walk on complete graphs, we prove a probabilistic analogue of the velocity formula between entropy and hitting time, where distance is the entropy of the Markov trajectories from state $i$ to state $j$ in the sense of [L. Ekroot and T. M. Cover. The entropy of Markov trajectories. IEEE Trans. Inform. Theory 39(4): 1418-1421.], speed is the classical entropy rate of the chain, and the time variable is the ex…
Motivated by entropic optimal transport, time reversal of Markov jump processes in $\mathbb{R}^n$ is investigated. Relying on an abstract integration by parts formula for the carré du champ of a Markov process recently obtained by Cattiaux, Gentil and the authors, and using an entropic improvement strategy discovered by Föllmer in the eighties, we compute the semimartingale characteristics of the time reversed process for a wide class of jump processes with possibly unbounded variation sample paths and singular intensities of jump.
We consider a strictly substochastic matrix or an stochastic matrix with absorbing states. By using quasi-stationary distributions one shows there is a canonical associated stationary Markov chain. Based upon $2-$stringing representation of the resurrected chain we supply a stationary representation of the killed and the absorbed chains. The entropies of these representations has a clear meaning when one identifies the probability measure of natural factors. The balance between the entropies of these representations and the canonical chain, serves to check the suitability of the whole construc…
In this work we study the Hausdorff dimension of measures whose weight distribution satisfies a markov non-homogeneous property. We prove, in particular, that the Hausdorff dimensions of this kind of measures coincide with their lower Rényi dimensions (entropy). Moreover, we show that the Tricot dimensions (packing dimension) equal the upper Rényi dimensions. As an application we get a continuity property of the Hausdorff dimension of the measures, when it is seen as a function of the distributed weights under the $\ell^{\infty}$ norm.
Motivated by queueing systems with heterogeneous parallel servers, we consider a class of structured multi-dimensional Markov processes whose state space can be partitioned into two parts: a finite set of boundary states and a structured multi-dimensional set of states, exactly one dimension of which is infinite. Using separation of variables, we show that the equilibrium distribution, typically of the queue length, can be represented as a linear combination of product forms. For an important subclass of queueing systems, we characterize explicitly the waiting time distribution in terms of mix…
A lumping of a Markov chain is a coordinate-wise projection of the chain. We characterise the entropy rate preservation of a lumping of an aperiodic and irreducible Markov chain on a finite state space by the random growth rate of the cardinality of the realisable preimage of a finite-length trajectory of the lumped chain and by the information needed to reconstruct original trajectories from their lumped images. Both are purely combinatorial criteria, depending only on the transition graph of the Markov chain and the lumping function. A lumping is strongly k-lumpable, iff the lumped process i…
The paper analyzes the performance of tandem network of polling queue with setups. For a system with two-products and two-stations, we propose a new approach based on a partially-collapsible state-space characterization to reduce state-space complexity. In this approach, the size of the state-space is varied depending on the information needed to determine buffer levels and waiting times. We evaluate system performance under different system setting and comment on the numerical accuracy of the approach as well as provide managerial insights. Numerical results show that approach yields reliable…
In the paper, we introduce the maximum entropy estimator based on 2-dimensional empirical distribution of the observation sequence of hidden Markov model , when the sample size is big: in that case computing the maximum likelihood estimator is too consuming in time by the classical Baum-Welch EM algorithm. We prove the consistency and the asymptotic normality of the maximum entropy estimator in a quite general framework, where the asymptotic covariance matrix is explicitly estimated in terms of the 2-dimensional Fisher information. To complement it, the 2-dimensional relative entropy is skillf…
A hidden Markov model (HMM) is said to have path-mergeable states if for any two states i,j there exists a word w and state k such that it is possible to transition from both i and j to k while emitting w. We show that for a finite HMM with path-mergeable states the block estimates of the entropy rate converge exponentially fast. We also show that the path-mergeability property is asymptotically typical in the space of HMM topolgies and easily testable.
We prove that under mild positivity assumptions the entropy rate of a hidden Markov chain varies analytically as a function of the underlying Markov chain parameters. A general principle to determine the domain of analyticity is stated. An example is given to estimate the radius of convergence for the entropy rate. We then show that the positivity assumptions can be relaxed, and examples are given for the relaxed conditions. We study a special class of hidden Markov chains in more detail: binary hidden Markov chains with an unambiguous symbol, and we give necessary and sufficient conditions fo…
Consider a hidden Markov chain obtained as the observation process of an ordinary Markov chain corrupted by noise. Zuk, et. al. [13], [14] showed how, in principle, one can explicitly compute the derivatives of the entropy rate of at extreme values of the noise. Namely, they showed that the derivatives of standard upper approximations to the entropy rate actually stabilize at an explicit finite time. We generalize this result to a natural class of hidden Markov chains called ``Black Holes.'' We also discuss in depth special cases of binary Markov chains observed in binary symmetric noise, and …
In this paper we consider the convergence of the conditional entropy to the entropy rate for Markov chains. Convergence of certain statistics of long range dependent processes, such as the sample mean, is slow. It has been shown in Carpio and Daley \cite{carpio2007long} that the convergence of the $n$-step transition probabilities to the stationary distribution is slow, without quantifying the convergence rate. We prove that the slow convergence also applies to convergence to an information-theoretic measure, the entropy rate, by showing that the convergence rate is equivalent to the convergen…
We study a new notion of Ricci curvature that applies to Markov chains on discrete spaces. This notion relies on geodesic convexity of the entropy and is analogous to the one introduced by Lott, Sturm, and Villani for geodesic measure spaces. In order to apply to the discrete setting, the role of the Wasserstein metric is taken over by a different metric, having the property that continuous time Markov chains are gradient flows of the entropy. Using this notion of Ricci curvature we prove discrete analogues of fundamental results by Bakry--Emery and Otto--Villani. Furthermore we show that Ricc…
Let K be an irreducible and reversible Markov kernel on a finite set X. We construct a metric W on the set of probability measures on X and show that with respect to this metric, the law of the continuous time Markov chain evolves as the gradient flow of the entropy. This result is a discrete counterpart of the Wasserstein gradient flow interpretation of the heat flow in R^n by Jordan, Kinderlehrer, and Otto (1998). The metric W is similar to, but different from, the L^2-Wasserstein metric, and is defined via a discrete variant of the Benamou-Brenier formula.
